<p>First of all, it must be noted that the solution to the problem of finding a refractivity profile from a ground-level dependence of refraction on elevation is not unique — at least if we just aim to get a good approximation. A pair of drastically different profiles can agree within the same error bounds on the astronomical refraction.</p>
<p>Now, ref. 1 in the OP gives quite an imprecise profile that approximates each (multi-km) altitude segment with a <em>linear</em> function. The OP used the data given in ref. 1, interpolating between the points on a logarithmic scale, which may have broken the approximation completely. So, the computation of astronomical refraction there is unreliable.</p>
<p>We can do better, namely, take profiles from ref. A that give more detailed (and more complete — up to 120 km) profiles of atmospheric densities and volume percentages of various constituents. Using the simplistic approach, i.e. assuming that the refractivity of air depends only on number density at each altitude (and independent of actual constitution) will give us, for <span class="math-container">$\lambda=555\,\mathrm{nm}$</span> (with <span class="math-container">$n$</span> taken from <a href="https://refractiveindex.info/?shelf=other&amp;book=air&amp;page=Ciddor" rel="nofollow noreferrer">here</a>), the following refractivity profile for subarctic summer conditions in ref. A:</p>
<p><a href="https://i.sstatic.net/8IuzO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8IuzO.png" alt="" /></a></p>
<p>We can see that the values in ref. 1 from the OP are not that far from our new profile, and both are very far from the OP's fit. At the same time, the difference from the Bennett's formula (adjusted to the same ground temperature of <span class="math-container">$287.2\,\mathrm{K}$</span> from its default <span class="math-container">$283\,\mathrm{K}$</span>) is already quite small:</p>
<p><a href="https://i.sstatic.net/qEOyw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qEOyw.png" alt="" /></a></p>
<p>Peak difference from the Bennett formula is about <span class="math-container">$0.0072°$</span>, which is comparable to the peak difference of the OP's fit that's about <span class="math-container">$0.0023°$</span>, and is much less than the difference of <span class="math-container">$0.042°$</span> in the OP's physically-based calculations. A simple hack of using the temperature less by <span class="math-container">$2.9°$</span> than actual AFGL data when generating the refractivity profile yields an error of <span class="math-container">$0.0015°$</span> that's even smaller than that of the OP's fit. This is a cheat of course, but this just shows the non-uniqueness of the solution. The correct way would be to take refractivity of each constituent and compute the appropriate value for each altitude separately.</p>
<p>As for the Bennett formula's precision, Wikipedia cites Bennett's own paper, ref. B, saying that its error is within <span class="math-container">$0.07'=0.0012°$</span> from ray tracing calculations with the algorithm from ref. C. So we can consider this to be the answer to the OP.</p>
<h3>References</h3>
<p>A. <a href="https://www.researchgate.net/publication/235054307_AFGL_Atmospheric_Constituent_Profiles_0120km" rel="nofollow noreferrer">AFGL Atmospheric Constituent Profiles AFGL-TR-86-0110</a></p>
<p>B. Bennett, G. (1982). <a href="https://www.cambridge.org/core/journals/journal-of-navigation/article/abs/calculation-of-astronomical-refraction-in-marine-navigation/C11786A8CFD408E9959C0EDCB9A64960" rel="nofollow noreferrer">The Calculation of Astronomical Refraction in Marine Navigation</a>. Journal of Navigation, 35(2), 255-259. doi:10.1017/S0373463300022037</p>
<p>C. B. Garfinkel. <a href="https://ui.adsabs.harvard.edu/abs/1967AJ.....72..235G/abstract" rel="nofollow noreferrer">Astronomical refraction in a polytropic atmosphere</a>. Astronomical Journal, Vol. 72, p. 235-254 (1967). doi:10.1086/110225</p>

CC BY-SA 4.0
2021-06-23T20:36:37.127Z
2021-06-23T20:51:13.133Z
2021-06-23T20:51:13.133Z
Ruslan
<p>I was trying to reproduce the predictions of <a href="https://en.wikipedia.org/wiki/Atmospheric_refraction#Calculating_refraction" rel="nofollow noreferrer">the Bennett formula</a>* by physically-based calculations with a model based on a real-life refractivity profile. My calculations based on Table V from ref. 1 compare as follows with the Bennett formula:</p>
<p><a href="https://i.sstatic.net/wmGbm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wmGbm.png" alt="" /></a></p>
<p>As you can see, there's some noticeable discrepancy, that's larger than between e.g. inverted-Saemundsson formula and directly-used Bennett formula (which would blend on a plot).</p>
<p>I've tried to come up with a refractivity profile that would reproduce the Bennett formula much better. I've indeed been able to make a very simple exponential profile that reproduces the values much better:</p>
<p><a href="https://i.sstatic.net/kiSMT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kiSMT.png" alt="" /></a></p>
<p>But the refractivity fitted is very different from that in ref. 1:</p>
<p><a href="https://i.sstatic.net/dgayE.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dgayE.png" alt="" /></a></p>
<p>So I think that Bennett formula must actually be quite imprecise, and I shouldn't try too hard to fit the model to it. But how much is it actually imprecise compared to <em>measured</em> astronomical refraction? Are there any freely accessible measured values of refraction itself rather than refractivity profiles, that could be used to estimate the precision of Bennett formula (and of my calculations)?</p>
<h3>References</h3>
<ol>
<li>D. Vasylyev, W. Vogel, <a href="https://www.researchgate.net/publication/330553517_Satellite-mediated_quantum_atmospheric_links" rel="nofollow noreferrer"><em>&quot;Satellite-mediated quantum atmospheric links&quot;</em></a>, Phys. Rev. A 99, 053830 (2019)</li>
</ol>
<hr />
<p>*Per Wikipedia:</p>
<blockquote>
<p><span class="math-container">$$R = \cot \left( h_a + \frac{7.31}{h_a + 4.4} \right)$$</span></p>
<p>where <span class="math-container">$R$</span> is the refraction in arcminutes and <span class="math-container">$h_a$</span> is the apparent altitude of the astronomical body in degrees.</p>
</blockquote>

CC BY-SA 4.0
2021-06-23T02:13:30.500Z
Cool question! I wonder why the reference broke the atmosphere up into only 10 discrete layers; it seems the model could easily be implemented via numerical integration with much smaller step sizes; the [U.S. Standard Atmosphere, 1976](https://ntrs.nasa.gov/citations/19770009539) model they use has high granularity tabular data. Since giant step size approximations to monotonic behavior can lead to a systematic over or under prediction, it *might be* the cause of the difference in your $(n-1) \times 10^8$ plot.
uhoh
CC BY-SA 4.0
2021-06-23T02:15:10.960Z
First three points at 0, 5 and 7 km *are hugely spaced!* Note they agree at 0 and monotonically deviate thereafter. Perhaps their values are *effective ones* for the contribution throughout the thick layer whereas the other in your plot is the index value at the layer's nominal height only?
uhoh
CC BY-SA 4.0
2021-06-23T09:04:26.753Z
@uhoh hmm, re-reading my ref. 1, I've indeed misused their table. Right above it they describe that $n_i(h)$ is linearly-changing in the $i$th segment, while I simply interpolated between the points on the log scale. So that table seems to be a poor reference. Too bad the US Standard Atmosphere doesn't explicitly specify refractivity, one has to derive it from the constituents...
Ruslan
CC BY-SA 4.0
2021-06-24T03:41:25.930Z
fyi I've just asked [Ray tracing in nonuniform media; did I write this second order differential equation as two first order differential equations correctly](https://math.stackexchange.com/q/4181503/284619) in Math SE.
uhoh
<positional-astronomy><atmospheric-effects>
How precise are Saemundsson/Bennett refraction formulas?
Ruslan
# How precise are Saemundsson/Bennett refraction formulas?

(Asked by: username_0 on 2021-06-22)

I was trying to reproduce the predictions of [the Bennett formula](https://en.wikipedia.org/wiki/Atmospheric_refraction#Calculating_refraction)\* by physically-based calculations with a model based on a real-life refractivity profile. My calculations based on Table V from ref. 1 compare as follows with the Bennett formula:

[![](https://i.sstatic.net/wmGbm.png)](https://i.sstatic.net/wmGbm.png)

As you can see, there's some noticeable discrepancy, that's larger than between e.g. inverted-Saemundsson formula and directly-used Bennett formula (which would blend on a plot).

I've tried to come up with a refractivity profile that would reproduce the Bennett formula much better. I've indeed been able to make a very simple exponential profile that reproduces the values much better:

[![](https://i.sstatic.net/kiSMT.png)](https://i.sstatic.net/kiSMT.png)

But the refractivity fitted is very different from that in ref. 1:

[![](https://i.sstatic.net/dgayE.png)](https://i.sstatic.net/dgayE.png)

So I think that Bennett formula must actually be quite imprecise, and I shouldn't try too hard to fit the model to it. But how much is it actually imprecise compared to *measured* astronomical refraction? Are there any freely accessible measured values of refraction itself rather than refractivity profiles, that could be used to estimate the precision of Bennett formula (and of my calculations)?

### References

1. D. Vasylyev, W. Vogel, [*"Satellite-mediated quantum atmospheric links"*](https://www.researchgate.net/publication/330553517_Satellite-mediated_quantum_atmospheric_links), Phys. Rev. A 99, 053830 (2019)

---

\*Per Wikipedia:

> $$R = \cot \left( h\_a + \frac{7.31}{h\_a + 4.4} \right)$$
>
> where $R$ is the refraction in arcminutes and $h\_a$ is the apparent altitude of the astronomical body in degrees.

**Tags:** positional-astronomy, atmospheric-effects
> **Comments:**
> **username_1**: Cool question! I wonder why the reference broke the atmosphere up into only 10 discrete layers; it seems the model could easily be implemented via numerical integration with much smaller step sizes; the [U.S. Standard Atmosphere, 1976](https://ntrs.nasa.gov/citations/19770009539) model they use has high granularity tabular data. Since giant step size approximations to monotonic behavior can lead to a systematic over or under prediction, it *might be* the cause of the difference in your $(n-1) \times 10^8$ plot.  
> **username_1**: First three points at 0, 5 and 7 km *are hugely spaced!* Note they agree at 0 and monotonically deviate thereafter. Perhaps their values are *effective ones* for the contribution throughout the thick layer whereas the other in your plot is the index value at the layer's nominal height only?  
> **username_0**: @uhoh hmm, re-reading my ref. 1, I've indeed misused their table. Right above it they describe that $n_i(h)$ is linearly-changing in the $i$th segment, while I simply interpolated between the points on the log scale. So that table seems to be a poor reference. Too bad the US Standard Atmosphere doesn't explicitly specify refractivity, one has to derive it from the constituents...  
> **username_1**: fyi I've just asked [Ray tracing in nonuniform media; did I write this second order differential equation as two first order differential equations correctly](https://math.stackexchange.com/q/4181503/284619) in Math SE.  

---

## 1 answers

### Answer by username_0 on 2021-06-23. Score: 1

First of all, it must be noted that the solution to the problem of finding a refractivity profile from a ground-level dependence of refraction on elevation is not unique — at least if we just aim to get a good approximation. A pair of drastically different profiles can agree within the same error bounds on the astronomical refraction.

Now, ref. 1 in the OP gives quite an imprecise profile that approximates each (multi-km) altitude segment with a *linear* function. The OP used the data given in ref. 1, interpolating between the points on a logarithmic scale, which may have broken the approximation completely. So, the computation of astronomical refraction there is unreliable.

We can do better, namely, take profiles from ref. A that give more detailed (and more complete — up to 120 km) profiles of atmospheric densities and volume percentages of various constituents. Using the simplistic approach, i.e. assuming that the refractivity of air depends only on number density at each altitude (and independent of actual constitution) will give us, for $\lambda=555\,\mathrm{nm}$ (with $n$ taken from [here](https://refractiveindex.info/?shelf=other&book=air&page=Ciddor)), the following refractivity profile for subarctic summer conditions in ref. A:

[![](https://i.sstatic.net/8IuzO.png)](https://i.sstatic.net/8IuzO.png)

We can see that the values in ref. 1 from the OP are not that far from our new profile, and both are very far from the OP's fit. At the same time, the difference from the Bennett's formula (adjusted to the same ground temperature of $287.2\,\mathrm{K}$ from its default $283\,\mathrm{K}$) is already quite small:

[![](https://i.sstatic.net/qEOyw.png)](https://i.sstatic.net/qEOyw.png)

Peak difference from the Bennett formula is about $0.0072°$, which is comparable to the peak difference of the OP's fit that's about $0.0023°$, and is much less than the difference of $0.042°$ in the OP's physically-based calculations. A simple hack of using the temperature less by $2.9°$ than actual AFGL data when generating the refractivity profile yields an error of $0.0015°$ that's even smaller than that of the OP's fit. This is a cheat of course, but this just shows the non-uniqueness of the solution. The correct way would be to take refractivity of each constituent and compute the appropriate value for each altitude separately.

As for the Bennett formula's precision, Wikipedia cites Bennett's own paper, ref. B, saying that its error is within $0.07'=0.0012°$ from ray tracing calculations with the algorithm from ref. C. So we can consider this to be the answer to the OP.

### References

A. [AFGL Atmospheric Constituent Profiles AFGL-TR-86-0110](https://www.researchgate.net/publication/235054307_AFGL_Atmospheric_Constituent_Profiles_0120km)

B. Bennett, G. (1982). [The Calculation of Astronomical Refraction in Marine Navigation](https://www.cambridge.org/core/journals/journal-of-navigation/article/abs/calculation-of-astronomical-refraction-in-marine-navigation/C11786A8CFD408E9959C0EDCB9A64960). Journal of Navigation, 35(2), 255-259. doi:10.1017/S0373463300022037

C. B. Garfinkel. [Astronomical refraction in a polytropic atmosphere](https://ui.adsabs.harvard.edu/abs/1967AJ.....72..235G/abstract). Astronomical Journal, Vol. 72, p. 235-254 (1967). doi:10.1086/110225


---

