It was recently estimated that the strong isospin-symmetry breaking (ISB) corrections to the Fermi matrix element in free neutron decay could be of the order \(10^{-4}\), one order of magnitude larger than the naïve estimate based on the Behrends-Sirlin-Ademollo-Gatto theorem. To investigate this claim, we derive a general expression of the leading ISB correction to hadronic Fermi matrix elements, which takes the form of a four-point correlation function in lattice gauge theory and is straightforward to compute from first principles. Our formalism paves the way for the first determination of such correction in the neutron sector with fully-controlled theory uncertainties.