Answers.Body: <p>Notice the architectural change your reference above is only about optimizer, regularizer, and loss functions which only affect the proportionality factor as claimed, but <span class="math-container">$D$</span> here has other architectural contents like the most critical <em>type</em> of learning model employed which would affect the exponent.</p>
  <p>And indeed the transformer model including MHA+FFN scales much better than LSTM which would be reflected in the exponent of the neural scaling law, since LSTM training is <em>sequential</em> due to RNN's inherent memory states recurrent nature leading to vanishing gradients and is limited in capturing long-range dependencies. Even a decoder-only transformer at training stage processes the input sequence in parallel to leverage the efficiency of the transformer architecture which has no inherent hidden memory states.</p>
  <p>Additionally since transformers have better domain specific inductive bias, they could achieve lower training loss compared to LSTM. Therefore from the other dataset size perspective, transformers' number of parameters scale better in the sense of generalization with same training dataset size.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-14T01:45:49.533Z
Answers.Text: thanks so much for a great answer here, you are right I completely skipped over the details of the architecture specifics they tested over.
  
  One thing I am missing from my understanding on your answer is 
  
  
  "Additionally since transformers have better domain specific inductive bias, they could achieve lower training loss compared to LSTM."
  
  Can you explain this?
Answers.UserDisplayName: Jacob B
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-14T01:58:06.753Z
Answers.Text: I mean transformers capture the long-range dependency modeling capacity of language model via self-attention mechanism which is proved empirically (much) better than hidden-sequential-state mechanism of LSTM to capture the same. Transformer's assumed mechanism and model architecture are more useful and accurate for language modeling purpose. Hope this completely clarifies your lingering confusion of this specific question.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-14T01:13:04.793Z
Answers.LastActivityDate: 2024-12-14T01:13:04.793Z
Answers.UserDisplayName: cinch
Body: <p>I was reading about a study on neural scaling laws from 2017 and they noted this as a summary. From Hestness, Joel; Narang, Sharan; Ardalani, Newsha; Diamos, Gregory; Jun, Heewoo; Kianinejad, Hassan; Patwary, Md Mostofa Ali; Yang, Yang; Zhou, Yanqi (2017-12-01). &quot;Deep Learning Scaling is Predictable, Empirically&quot;. arXiv:1712.00409 [cs.LG].</p>
  <p><a href="https://i.sstatic.net/gwULRGgI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gwULRGgI.png" alt="This is an excerpt from wikipedia on neural scaling laws" /></a></p>
  <p>One of the studies was generative LM with an LSTM. Since then of course we use transformers for this which have beaten every benchmark.
  Transformers are an architecture change meaning that there should be no change in the power here, so why do they perform better?</p>
  <p>Is it a matter of them having a lower loss coefficient here? Is anyone aware of any studies done on the scaling laws between LSTM/Transformers?
  In the case of a decoder model I would assume the training time for an LSTM is the same given they are both decoding at every step.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-15T22:21:04.010Z
Comments.Text: Please, provide the actual text and not a screenshot of the paper. In this case, there are many formulas to translate, but there are programs that allow you to read a PDF and convert it to text that contains mathjax, which is supported on this site.
Comments.UserDisplayName: nbro
Tags: <comparison><large-language-models>
Title: How do neural scaling laws explain the improvements from LSTMs to Transformer based models
UserDisplayName: Jacob B
ThreadText: # How do neural scaling laws explain the improvements from LSTMs to Transformer based models
  
  (Asked by: username_0 on 2024-12-14)
  
  I was reading about a study on neural scaling laws from 2017 and they noted this as a summary. From Hestness, Joel; Narang, Sharan; Ardalani, Newsha; Diamos, Gregory; Jun, Heewoo; Kianinejad, Hassan; Patwary, Md Mostofa Ali; Yang, Yang; Zhou, Yanqi (2017-12-01). "Deep Learning Scaling is Predictable, Empirically". arXiv:1712.00409 [cs.LG].
  
  [![This is an excerpt from wikipedia on neural scaling laws](https://i.sstatic.net/gwULRGgI.png)](https://i.sstatic.net/gwULRGgI.png)
  
  One of the studies was generative LM with an LSTM. Since then of course we use transformers for this which have beaten every benchmark.
  Transformers are an architecture change meaning that there should be no change in the power here, so why do they perform better?
  
  Is it a matter of them having a lower loss coefficient here? Is anyone aware of any studies done on the scaling laws between LSTM/Transformers?
  In the case of a decoder model I would assume the training time for an LSTM is the same given they are both decoding at every step.
  
  **Tags:** comparison, large-language-models
  > **Comments:**
  > **username_1**: Please, provide the actual text and not a screenshot of the paper. In this case, there are many formulas to translate, but there are programs that allow you to read a PDF and convert it to text that contains mathjax, which is supported on this site.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-12-14. Score: 1 | Accepted answer
  
  Notice the architectural change your reference above is only about optimizer, regularizer, and loss functions which only affect the proportionality factor as claimed, but $D$ here has other architectural contents like the most critical *type* of learning model employed which would affect the exponent.
  
  And indeed the transformer model including MHA+FFN scales much better than LSTM which would be reflected in the exponent of the neural scaling law, since LSTM training is *sequential* due to RNN's inherent memory states recurrent nature leading to vanishing gradients and is limited in capturing long-range dependencies. Even a decoder-only transformer at training stage processes the input sequence in parallel to leverage the efficiency of the transformer architecture which has no inherent hidden memory states.
  
  Additionally since transformers have better domain specific inductive bias, they could achieve lower training loss compared to LSTM. Therefore from the other dataset size perspective, transformers' number of parameters scale better in the sense of generalization with same training dataset size.
  
  > **Comments:**
  > **username_0**: thanks so much for a great answer here, you are right I completely skipped over the details of the architecture specifics they tested over.  One thing I am missing from my understanding on your answer is    "Additionally since transformers have better domain specific inductive bias, they could achieve lower training loss compared to LSTM."  Can you explain this?  
  > **username_2**: I mean transformers capture the long-range dependency modeling capacity of language model via self-attention mechanism which is proved empirically (much) better than hidden-sequential-state mechanism of LSTM to capture the same. Transformer's assumed mechanism and model architecture are more useful and accurate for language modeling purpose. Hope this completely clarifies your lingering confusion of this specific question.  
  
  ---
  
  