Answers.Body: <p>The diagram is a concise representation where the Nx refers to a collapsed array. This diagram can be expanded horizontally with multiple left columns chained together. This diagram can be expanded vertically with multiple yellow/orange rows chained together. If you do an image search you'll find some zoomed out color coded maps of various topologies.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-11T11:18:55.493Z
Answers.Text: Can you please provide an example of a model based on the transformer that does what you say?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-11T10:34:35.983Z
Answers.LastActivityDate: 2023-02-11T10:34:35.983Z
Answers.UserDisplayName: bitRAKE
Body: <p>I can find a million explanations of the diagram in the original transformer paper:</p>
  <p><a href="https://i.sstatic.net/jIwvJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jIwvJ.png" alt="enter image description here" /></a></p>
  <p>But I know that modern GPT models have many millions of weights. Where are they?  Or in other words, how does this thing scale?</p>
  
Tags: <transformer><gpt>
Title: How do they make transformers bigger/deeper?
UserDisplayName: Mastiff
ThreadText: # How do they make transformers bigger/deeper?
  
  (Asked by: username_0 on 2023-02-10)
  
  I can find a million explanations of the diagram in the original transformer paper:
  
  [![enter image description here](https://i.sstatic.net/jIwvJ.png)](https://i.sstatic.net/jIwvJ.png)
  
  But I know that modern GPT models have many millions of weights. Where are they? Or in other words, how does this thing scale?
  
  **Tags:** transformer, gpt
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-11. Score: 2
  
  The diagram is a concise representation where the Nx refers to a collapsed array. This diagram can be expanded horizontally with multiple left columns chained together. This diagram can be expanded vertically with multiple yellow/orange rows chained together. If you do an image search you'll find some zoomed out color coded maps of various topologies.
  
  > **Comments:**
  > **username_2**: Can you please provide an example of a model based on the transformer that does what you say?  
  
  ---
  
  