Answers.Body: <p>I think a nice back-of-the envelope calculation is the intuition for exploding/vanishing gradients in RNNs:</p>
  <p><strong>Simplifications</strong></p>
  <ul>
  <li>diagonalisable weights <span class="math-container">$U$</span> and <span class="math-container">$W$</span></li>
  <li>no non-linearities</li>
  <li>1 layer</li>
  </ul>
  <p>This gives a hidden state <span class="math-container">$h_t$</span> at timestep <span class="math-container">$t$</span> for input <span class="math-container">$x_t$</span>:  <span class="math-container">$h_t = W\cdot h_{t-1} + U\cdot x_t$</span></p>
  <p>Let <span class="math-container">$L_t$</span> be the loss at timestep <span class="math-container">$t$</span> and the total loss <span class="math-container">$L = \sum_t L_t$</span>. Then (eq. 3 -&gt; 5 in the paper)</p>
  <p><span class="math-container">$$
  \frac{\partial L_t}{\partial W} \sim = 
  \sum_{k=1}^{t} \frac{\partial L_t}{\partial h_t} \frac{\partial h_t}{\partial h_k} \frac{\partial h_k}{\partial W} = \sum_{k=1}^{t}\frac{\partial h_t}{\partial h_k}\times\alpha_{t, k}
  $$</span></p>
  <p>Let's not care about terms regrouped in <span class="math-container">$\alpha_{t, k}$</span>:</p>
  <p><span class="math-container">$$
  \frac{\partial h_t}{\partial h_k} = \prod_{k&lt;i\leq t} \frac{\partial h_i}{\partial h_{i-1}} =  \prod_{k&lt;i\leq t}  W =  \prod_{k&lt;i\leq t} PDP^\top = PD^{t-k}P^{\top}
  $$</span></p>
  <p>So you can easily see<span class="math-container">$^1$</span> that if the eigen values of <span class="math-container">$W$</span> (in the diagonal matrix <span class="math-container">$D$</span>) are larger than <span class="math-container">$1$</span>, the gradient will explode with time, and if they are smaller than <span class="math-container">$1$</span>, it will vanish.</p>
  <p>More detailed derivations in <a href="http://proceedings.mlr.press/v28/pascanu13.pdf" rel="nofollow noreferrer">On the difficulty of training recurrent neural networks</a></p>
  <hr />
  <p><span class="math-container">$^1$</span> remember <span class="math-container">$\lim_{n \to +\infty}|x^n| = +\infty$</span> if <span class="math-container">$|x|&gt;1$</span> and <span class="math-container">$=0$</span> for <span class="math-container">$|x| &lt; 1$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-04T10:51:35.317Z
Answers.Text: You say "you can easily see that", but I cannot easily see it (so I guess many others can't easily see it too). Maybe you should provide more intuition behind that and, more importantly, define your matrices and other symbols.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-05T05:29:07.880Z
Answers.Text: I didn't think I had to explain that x^n goes to inf for |x| > 1 and to 0 for |x| < 1 as n goes to inf. Also notations are standard rnn notations and there is a link to the paper for the details you asked for.
Answers.UserDisplayName: ted
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T23:39:03.320Z
Answers.LastActivityDate: 2020-10-05T05:38:52.627Z
Answers.LastEditDate: 2020-10-05T05:38:52.627Z
Answers.UserDisplayName: ted
Answers.Body: <p>One neat thing is to estimate the GPU usage when training an ML model. This can be relevant for training or fine-tuning large models. If inference with one instance takes <span class="math-container">$N$</span> GB of vRAM, then one training step with batch size <span class="math-container">$B$</span> requires <span class="math-container">$2 \times B \times N $</span> GB of vRAM.</p>
  <hr />
  <p><strong>Note.</strong> The factor of 2 is there because forward pass computes activations and stores them, while the backward pass needs to store both activations and gradients for computing parameter updates through backpropagation. The exact memory usage can vary based on the network architecture and the specific operations involved, but a rule of thumb is that it's approximately 2x the memory of the forward pass.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-20T16:21:19.230Z
Answers.LastActivityDate: 2023-12-20T01:48:32.653Z
Answers.LastEditDate: 2023-12-20T01:48:32.653Z
Answers.UserDisplayName: Ron
Answers.Body: <p>I have one to share. This is no formula, but a general thing I have noticed. </p>
  
  <p>The number of neurons + neurons should be proportionate, in some way, to the complexity of the classification.</p>
  
  <p>Although this is fairly basic and widely known, it has helped me in many times to consider one thing: how many at a minimum does it need?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-04T10:49:31.450Z
Answers.Text: What does "The number of neurons + neurons" mean? Maybe you mean the number of parameters is proportional to the complexity of the neural network?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-17T15:41:06.720Z
Answers.Text: It looks like my question may be a bit vague. I’ll try to add more constraints this afternoon if I have time.
Answers.UserDisplayName: Charles
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-17T15:00:49.500Z
Answers.LastActivityDate: 2018-04-17T15:11:00.413Z
Answers.LastEditDate: 2018-04-17T15:11:00.413Z
Answers.UserDisplayName: FreezePhoenix
Body: <p>There is a popular story regarding the <a href="https://en.wikipedia.org/wiki/Back-of-the-envelope_calculation" rel="noreferrer">back-of-the-envelope calculation</a> performed by a British physicist named G. I. Taylor. He used <a href="http://www.atmosp.physics.utoronto.ca/people/codoban/PHY138/Mechanics/dimensional.pdf" rel="noreferrer">dimensional analysis</a> to estimate the power released by the explosion of a nuclear bomb, simply by analyzing a picture that was released in a magazine at the time.</p>
  
  <p>I believe many of you know some nice back-of-the-envelope calculations performed in machine learning (more specifically neural networks). Can you please share them?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-04T10:46:20.033Z
Comments.Text: Yes, this question is not fully clear. [This answer](https://ai.stackexchange.com/a/6090/2444) attempts to give you a "back-of-the-envelope" calculation (as defined by the linked Wikipedia article), i.e. a _rule of thumb to approximately calculate_ something, although I don't think that answer is clear (or even correct). Is this type of calculation that you were looking for, or were you looking for something else? Please, edit your post to clarify that, otherwise, I will close it as "needs details or clarity".
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-05T13:37:31.140Z
Comments.Text: I think the question is clear, and so do people who have provided answers. I think both submitted answers look like what I might expect. Frankly, I modeled this question based off of https://mathoverflow.net/questions/8846/proofs-without-words. So, if this question is closed, then I would expect that question also be closed. I'm open to improving the question somehow if you have suggestions.
Comments.UserDisplayName: Charles
Tags: <neural-networks><machine-learning><artificial-neuron>
Title: Back-of-the-envelope machine learning (specifically neural networks) calculations
UserDisplayName: Charles
ThreadText: # Back-of-the-envelope machine learning (specifically neural networks) calculations
  
  (Asked by: username_0 on 2018-04-16)
  
  There is a popular story regarding the [back-of-the-envelope calculation](https://en.wikipedia.org/wiki/Back-of-the-envelope_calculation) performed by a British physicist named G. I. Taylor. He used [dimensional analysis](http://www.atmosp.physics.utoronto.ca/people/codoban/PHY138/Mechanics/dimensional.pdf) to estimate the power released by the explosion of a nuclear bomb, simply by analyzing a picture that was released in a magazine at the time.
  
  I believe many of you know some nice back-of-the-envelope calculations performed in machine learning (more specifically neural networks). Can you please share them?
  
  **Tags:** neural-networks, machine-learning, artificial-neuron
  > **Comments:**
  > **username_1**: Yes, this question is not fully clear. [This answer](https://ai.stackexchange.com/a/6090/2444) attempts to give you a "back-of-the-envelope" calculation (as defined by the linked Wikipedia article), i.e. a _rule of thumb to approximately calculate_ something, although I don't think that answer is clear (or even correct). Is this type of calculation that you were looking for, or were you looking for something else? Please, edit your post to clarify that, otherwise, I will close it as "needs details or clarity".  
  > **username_0**: I think the question is clear, and so do people who have provided answers. I think both submitted answers look like what I might expect. Frankly, I modeled this question based off of https://mathoverflow.net/questions/8846/proofs-without-words. So, if this question is closed, then I would expect that question also be closed. I'm open to improving the question somehow if you have suggestions.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2020-05-06. Score: 1
  
  I think a nice back-of-the envelope calculation is the intuition for exploding/vanishing gradients in RNNs:
  
  **Simplifications**
  
  * diagonalisable weights $U$ and $W$
  * no non-linearities
  * 1 layer
  
  This gives a hidden state $h\_t$ at timestep $t$ for input $x\_t$: $h\_t = W\cdot h\_{t-1} + U\cdot x\_t$
  
  Let $L\_t$ be the loss at timestep $t$ and the total loss $L = \sum\_t L\_t$. Then (eq. 3 -> 5 in the paper)
  
  $$
  \frac{\partial L\_t}{\partial W} \sim =
  \sum\_{k=1}^{t} \frac{\partial L\_t}{\partial h\_t} \frac{\partial h\_t}{\partial h\_k} \frac{\partial h\_k}{\partial W} = \sum\_{k=1}^{t}\frac{\partial h\_t}{\partial h\_k}\times\alpha\_{t, k}
  $$
  
  Let's not care about terms regrouped in $\alpha\_{t, k}$:
  
  $$
  \frac{\partial h\_t}{\partial h\_k} = \prod\_{k
  
  So you can easily see$^1$ that if the eigen values of $W$ (in the diagonal matrix $D$) are larger than $1$, the gradient will explode with time, and if they are smaller than $1$, it will vanish.
  
  More detailed derivations in [On the difficulty of training recurrent neural networks](http://proceedings.mlr.press/v28/pascanu13.pdf)
  
  ---
  
  $^1$ remember $\lim\_{n \to +\infty}|x^n| = +\infty$ if $|x|>1$ and $=0$ for $|x| < 1$
  
  > **Comments:**
  > **username_1**: You say "you can easily see that", but I cannot easily see it (so I guess many others can't easily see it too). Maybe you should provide more intuition behind that and, more importantly, define your matrices and other symbols.  
  > **username_2**: I didn't think I had to explain that x^n goes to inf for |x| > 1 and to 0 for |x| < 1 as n goes to inf. Also notations are standard rnn notations and there is a link to the paper for the details you asked for.  
  
  ---
  
  ### Answer by username_3 on 2023-10-20. Score: 1
  
  One neat thing is to estimate the GPU usage when training an ML model. This can be relevant for training or fine-tuning large models. If inference with one instance takes $N$ GB of vRAM, then one training step with batch size $B$ requires $2 \times B \times N $ GB of vRAM.
  
  ---
  
  **Note.** The factor of 2 is there because forward pass computes activations and stores them, while the backward pass needs to store both activations and gradients for computing parameter updates through backpropagation. The exact memory usage can vary based on the network architecture and the specific operations involved, but a rule of thumb is that it's approximately 2x the memory of the forward pass.
  
  
  ---
  
  ### Answer by username_4 on 2018-04-17. Score: 1
  
  I have one to share. This is no formula, but a general thing I have noticed.
  
  The number of neurons + neurons should be proportionate, in some way, to the complexity of the classification.
  
  Although this is fairly basic and widely known, it has helped me in many times to consider one thing: how many at a minimum does it need?
  
  > **Comments:**
  > **username_0**: It looks like my question may be a bit vague. I’ll try to add more constraints this afternoon if I have time.  
  > **username_1**: What does "The number of neurons + neurons" mean? Maybe you mean the number of parameters is proportional to the complexity of the neural network?  
  
  ---
  
  