An estimator or confidence set is statistically consistent if, in a well defined sense, it converges in probability to the truth as the number of data grows. We give sufficient conditions for it to be impossible to find consistent estimators or confidence sets in some linear inverse problems. Several common approaches to statistical inference in geophysical inverse problems use the set of models that satisfy the data within a chi-squared measure of misfit to construct confidence sets and estimates. For example, the minimum-norm estimate of the unknown model is the model of smallest norm among those that map into a chi-squared ball around the data. We give weaker conditions under which the chi-square misfit approach yields inconsistent estimators and confidence sets. Both sets of conditions depend on a measure of the redundancy of the observations, with respect to an {\em a priori\/} constraint on the model. When the observations are sufficiently redundant, using a chi-square measure of misfit to selected averages of the data yields consistent confidence sets and minimum-norm estimates. Under still weaker conditions, one can find consistent estimates and confidence intervals for finite collections of linear functionals of the model. In an idealization of the problem of estimating the Gauss coefficients of the magnetic field at the core from satellite data, using a constraint on the energy stored in the field, suitable data averaging leads to consistent confidence intervals for finite collections of the Gauss coefficients.