Answers.Body: <p>The <strong>Rate of Convergence of an Estimator</strong> will depend on a number of factors including:</p>
  <ul>
  <li>the type of estimator used,</li>
  <li>the data set used for training,</li>
  <li>and the tuning parameters of the estimator.</li>
  </ul>
  <p>However, in general, one can expect that an estimator will converge at a rate that is at least as fast as the inverse of the number of training examples used.</p>
  <p><code>...I suspect one of them has a higher variance and thus a slower convergence rate...</code>
  Yes, if two estimators are sampling from the same distribution, the estimator with higher variance will converge more slowly than the estimator with lower variance &amp;, this can be important when you are choosing between different algorithms to use for training your models, but <em>There is no general rule</em>; the choice depends on the specific situation, the one with the faster convergence rate will generally be more accurate that obvious.</p>
  <blockquote>
  <p>&quot;the <a href="https://en.wikipedia.org/wiki/Rate_of_convergence" rel="nofollow noreferrer">rate of convergence</a> provide useful insights when using iterative methods for calculating numerical approximations. If the order of convergence is higher, then typically fewer iterations are necessary to yield a useful approximation. Strictly speaking, however, the asymptotic behavior of a sequence does not give conclusive information about any finite part of the sequence.&quot;</p>
  </blockquote>
  <p>There are a few different ways to approach this. One way would be to use the <a href="https://www.utstat.toronto.edu/%7Ebrunner/oldclass/2201s11/readings/LargeSampleTheory.pdf" rel="nofollow noreferrer">law of large numbers(~4)</a>. This states that, as the number of samples increases, the average of the sample values will converge to the true value of the quantity being estimated.</p>
  <p>Another way would be to use <a href="https://www.utstat.toronto.edu/%7Ebrunner/oldclass/2201s11/readings/LargeSampleTheory.pdf" rel="nofollow noreferrer">the Central Limit Theorem(~5)</a>. This states that, as the number of samples increases, the distribution of the sample values will become more and more like a normal distribution. The mean of this distribution will be the true value of the quantity being estimated.</p>
  <p>Here are some references may be relevant:</p>
  <ul>
  <li><a href="https://www.math-cs.gordon.edu/courses/ma342/handouts/rate.pdf" rel="nofollow noreferrer">Computing and Estimating the Rate of Convergence</a></li>
  <li><a href="https://math.berkeley.edu/%7Emgu/MA128AFall2017/MA128ALectureWeek4.pdf" rel="nofollow noreferrer">Order of convergence - berkeley.edu</a></li>
  <li><a href="https://www.utstat.toronto.edu/%7Ebrunner/oldclass/2201s11/readings/LargeSampleTheory.pdf" rel="nofollow noreferrer">A Course in Large Sample Theory by Thomas S. Ferguson ~ pdf</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-22T21:52:59.753Z
Answers.LastActivityDate: 2022-10-22T21:52:59.753Z
Answers.UserDisplayName: Faizy
Body: <p>I want to estimate a quantity and have two choices for estimators (they both sample from the same distribution). I suspect one of them has a higher variance and thus a slower convergence rate. I want to mathematically prove this, but I don't know where to start. Is there a standard reference in statistics that I can consult for example proofs of convergence rates in ML/statistics?</p>
  
Tags: <machine-learning><convergence><variance>
Title: Prooving the convergence rate of estimators (machine Learning)
UserDisplayName: postnubilaphoebus
ThreadText: # Prooving the convergence rate of estimators (machine Learning)
  
  (Asked by: username_0 on 2022-10-22)
  
  I want to estimate a quantity and have two choices for estimators (they both sample from the same distribution). I suspect one of them has a higher variance and thus a slower convergence rate. I want to mathematically prove this, but I don't know where to start. Is there a standard reference in statistics that I can consult for example proofs of convergence rates in ML/statistics?
  
  **Tags:** machine-learning, convergence, variance
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-22. Score: 2 | Accepted answer
  
  The **Rate of Convergence of an Estimator** will depend on a number of factors including:
  
  * the type of estimator used,
  * the data set used for training,
  * and the tuning parameters of the estimator.
  
  However, in general, one can expect that an estimator will converge at a rate that is at least as fast as the inverse of the number of training examples used.
  
  `...I suspect one of them has a higher variance and thus a slower convergence rate...`
  Yes, if two estimators are sampling from the same distribution, the estimator with higher variance will converge more slowly than the estimator with lower variance &, this can be important when you are choosing between different algorithms to use for training your models, but *There is no general rule*; the choice depends on the specific situation, the one with the faster convergence rate will generally be more accurate that obvious.
  
  > "the [rate of convergence](https://en.wikipedia.org/wiki/Rate_of_convergence) provide useful insights when using iterative methods for calculating numerical approximations. If the order of convergence is higher, then typically fewer iterations are necessary to yield a useful approximation. Strictly speaking, however, the asymptotic behavior of a sequence does not give conclusive information about any finite part of the sequence."
  
  There are a few different ways to approach this. One way would be to use the [law of large numbers(~4)](https://www.utstat.toronto.edu/%7Ebrunner/oldclass/2201s11/readings/LargeSampleTheory.pdf). This states that, as the number of samples increases, the average of the sample values will converge to the true value of the quantity being estimated.
  
  Another way would be to use [the Central Limit Theorem(~5)](https://www.utstat.toronto.edu/%7Ebrunner/oldclass/2201s11/readings/LargeSampleTheory.pdf). This states that, as the number of samples increases, the distribution of the sample values will become more and more like a normal distribution. The mean of this distribution will be the true value of the quantity being estimated.
  
  Here are some references may be relevant:
  
  * [Computing and Estimating the Rate of Convergence](https://www.math-cs.gordon.edu/courses/ma342/handouts/rate.pdf)
  * [Order of convergence - berkeley.edu](https://math.berkeley.edu/%7Emgu/MA128AFall2017/MA128ALectureWeek4.pdf)
  * [A Course in Large Sample Theory by Thomas S. Ferguson ~ pdf](https://www.utstat.toronto.edu/%7Ebrunner/oldclass/2201s11/readings/LargeSampleTheory.pdf)
  
  
  ---
  
  