Error bars on error bars? Inspired by my recent attendance at an environmental toxicology conference, I have the following question about error bars:
Let's say that I'm drawing samples from some unknown distribution, with finite mean and variance. I want to present the sample mean, and add some error bars. Since I don't know much about the underlying distribution, I just add error bars showing +/- the standard devaition of the samples.
My question is, is there any way I could meaningfully indicate how certain I am of those error bars? Adding error bars to the error bars, so to speak.
As as example, I have drawn 5 samples from some distribution, and I have repeated this 5 times. The sample means, and error bars of +/- the sample standard deviations, are shown below.

We can see that by chance, these sample means and error bars look quite different, and not really mutually compatible. Of course 5 samples isn't very much, but if my samples are obtained via some convoluted experimental procedure (capturing a wild animal and taking a blood sample, for example), it might not be an easy option to get more samples.
Update:
Just to add some notes on how I was thinking:
Coming from a computational physics background myself, I'm used to Monte Carlo methods, and the $1/\sqrt{N}$-error which follows from the central limit theorem. So just like the error in the sample mean has an expected distribution, I thought perhaps it would make sense to ask about the expected error in the sample standard deviation. Of course, the problem is that the distribution of the error in the sample mean is expressed in terms of the (unknown) variance of the underlying distribution, and hence I am left taking the standard deviation of the sample, or something along those lines.
But still, I thought there ought to be some way of indicating that my sample standard deviation is itself quite uncertain, due to the small $N$. But perhaps the only way is simply to list $N$, and be explicit about what the error bars show.

TLDR;
Below is a simulation where we repeated an experiment of estimating the mean of a normal distribution with $\mu = 0$ and $\sigma = 1$. We did 200 repetitions with samples of size 10.

We can indeed see that the estimate of the standard deviation is different each experiment. We are not certain about the exact value of the standard deviation.
But one thing is more or less a constant, that is the probability that the true mean is inside the interval depicted by the error bar.
In this example we made 61 times (30.5%) a wrong estimate of the interval (coloured red/blue when we underestimate or overestimate the mean). For large samples it will become approximately 32% error (see https://en.m.wikipedia.org/wiki/68–95–99.7_rule)
When we interpret the error bars more in this way, as an interval that contains the parameter some amount of time, then the error on error bars is sort of baked into it and is in this expression of error of containing the parameter.
Interval estimation
Error bars can be seen as a graphical representation of interval estimation. So this discussion about uncertainty in the uncertainty estimates themselves and can be seen as similar to a more general discussion about intervals.
Standard deviation, standard error, are simple indicators of intervals
When the error bars represent the estimated standard deviation then indeed the error bars themselves have some uncertainty as well. Standard deviations are just a simple way of expressing the uncertainty.
You can have all kinds of intervals like, credible intervals or confidence intervals, in which case the uncertainty is tackled in some way or another.
Alternative example: Confidence intervals
For instance a confidence interval will contain the correct data point $\alpha\%$ of the time. This is one way to represent the certainty and precision in the data. The more precise the data is the smaller that we can make the intervals.
But note that confidence intervals represent the uncertainty in a peculiar way. See Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean? The confidence interval contains the true data point with $\alpha\%$ probability when we condition on the model parameters, and not when we condition on the observation.
For particular observations the intervals will be more often wrong than for other observations and the intervals may differ in size (like in your graph). So there is still uncertainty about the intervals. But this uncertainty is already expressed by stating that it is an interval with $\alpha\%$ probability.
The error bars based on the 'simple' standard deviation is often very close to a 68% confidence interval (see https://en.m.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule).
How the 'problem' is solved.

*

*In the case of the confidence interval the problem can be solved by computing a statistic that is a pivotal quantity.
For instance a t statistic is a ratio of the mean and the observed standard deviation, because both the numerator and denominator in this ratio depend on the variance of the original distribution, the ratio becomes independent on this variance. In this way the uncertainty about the variance of the distribution has been 'eliminated'.


*In the case of the credible interval we use a prior distribution to express the uncertainty about the entire system. In the final computation of the interval based on the posterior distribution, the uncertainty about the interval is included.
