Empirical Implications of Unbiased Estimators I am familiar with the layperson explanation of an unbiased estimator as follows: if we repeat an experiment under identical conditions many times, the average value of the estimate will be close to the true value (see the following question and answers).
I find that, when I explain this to people, the common answer I get is something along the lines of: "But I only ever have one sample and I compute the estimator based on this one sample, so what implication does unbiasedness have for my one sample?"
Intuitively, my answer would be something like the following: if we have two estimators with equal variance but the first is unbiased and the second is biased, then for a given random sample, the unbiased estimator is more likely to be closer to the true value than the biased estimator.
My question is whether (1) this is actually correct and (2) if it is correct, is there an explanation of unbiasedness that does not also involve the variance of the estimator as well?


if we repeat an experiment under identical conditions many times, the average value of the estimate will be close to the true value

was said, and the following figure from Understanding "variance" intuitively shown

and the laymen countered

But I only ever have one sample and I compute the estimator based on this one sample, so what implication does unbiasedness have for my one sample?

I would say something along the lines of : well, this estimate could be an overestimate, or it could be an underestimate, but if this sample is all we know, we are in no position to guess about either. So we can as well bet on the unbiased estimate, as it could as well be to low as to high.
Then try to imagine how our laymen could be countering:

So you are betting? But I want to know the truth!

Me: statistics is about betting. If you want the truth, certainty, go get a larger sample!

the sample ... but how do you know the sample is representative?

Me: You need to know how your sample was obtained! If it was obtained by simple random sampling (or some elaboration ...), it has a high probability of being representative.  If it wasn't obtained in some such way that we can analyze, unbiased estimation might not be the correct tool. Then we would have to look closer at how you obtained the sample first ...

OK, I used simple random sampling. But still, what if an underestimation would be a small error, but an overestimation might be huge?

Me: That would mean that your distribution is not symmetric. Indeed, unbiased estimators works best in symmetric cases. For asymmetric cases, maybe look into median unbiased estimators.
Continuing the dialogue, I would point out that betting on the unbiased estimator is reasonable provided that we have really used all available information. So, the estimator must be optimal in some sense (maybe sufficient), so that is it really using all the relevant information in the sample. Still there might be non-sample prior information, which in some cases could show us that the realized value of the unbiased estimator is unreasonable. And at this point I could show an example from What is the difference between finite and infinite variance
But then, finally, back to your first question:

Intuitively, my answer would be something like the following: if we have two estimators with equal variance but the first is unbiased and the second is biased, then for a given random sample, the unbiased estimator is more likely to be closer to the true value than the biased estimator.

Is that true? Mostly, yes, assuming that we have an optimal unbiased estimator, such as an unbiased minimum-variance estimator.  But, in many situations, there are (slightly) biased estimators that are better, for instance in having lower mean-square error. These are often known as shrinkage estimators, or found via regularization. See

*

*Why is the James-Stein estimator called a "shrinkage" estimator?


*What is shrinkage?


*Why does shrinkage really work, what's so special about 0?
