Answers.Body: <p>Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are <em>hyper-parameters</em> which basically means they are user specified, whereas weights are what the learning algorithm will learn through training.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T09:50:31.090Z
Answers.LastActivityDate: 2020-07-28T09:50:31.090Z
Answers.UserDisplayName: David
Body: <p>When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that they might just be sentences.</p>
  <p><a href="https://i.sstatic.net/LWEEQ.png" rel="noreferrer"><img src="https://i.sstatic.net/LWEEQ.png" alt="enter image description here" /></a></p>
  <p>Even tutorial sites like <a href="https://www.analyticsvidhya.com/blog/2019/07/openai-gpt2-text-generator-python/" rel="noreferrer">this one</a> start talking about the usual parameters, but also say <code>&quot;model_name: This indicates which model we are using. In our case, we are using the GPT-2 model with 345 million parameters or weights&quot;</code>. So are the 175 billion &quot;parameters&quot; just neural weights? Why then are they called parameters? <a href="https://arxiv.org/pdf/2005.14165.pdf" rel="noreferrer">GPT3's paper</a> shows that there are only 96 layers, so I'm assuming it's not a very deep network, but extremely fat. Or does it mean that each &quot;parameter&quot; is just a representation of the encoders or decoders?</p>
  <p><a href="https://i.sstatic.net/dthvC.png" rel="noreferrer"><img src="https://i.sstatic.net/dthvC.png" alt="enter image description here" /></a></p>
  <p>An excerpt from <a href="https://minimaxir.com/2019/09/howto-gpt2/" rel="noreferrer">this website</a> shows tokens:</p>
  <blockquote>
  <p>In this case, there are two additional parameters that can be passed
  to gpt2.generate(): truncate and include_prefix. For example, if each
  short text begins with a &lt;|startoftext|&gt; token and ends with a
  &lt;|endoftext|&gt;, then setting prefix='&lt;|startoftext|&gt;',
  truncate=&lt;|endoftext|&gt;', and include_prefix=False, and length is
  sufficient, then gpt-2-simple will automatically extract the shortform
  texts, even when generating in batches.</p>
  </blockquote>
  <p>So are the parameters various kinds of tokens that are manually created by humans who try to fine-tune the models? Still, 175 billion such fine-tuning parameters is too high for humans to create, so I assume the &quot;parameters&quot; are auto-generated somehow.</p>
  <p>The <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="noreferrer">attention-based</a> paper mentions the <a href="http://jalammar.github.io/illustrated-gpt2/" rel="noreferrer">query-key-value weight</a> matrices as the &quot;parameters&quot;. <strong>Even if it is these weights, I'd just like to know what kind of a process generates these parameters, who chooses the parameters and specifies the relevance of words? If it's created automatically, how is it done?</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-26T19:39:24.717Z
Comments.Text: Parameters is a synonym for weights, which are what you learn. Batch size, learning rate etc are typically hyper parameters
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-27T02:35:10.837Z
Comments.Text: Thank you David. So now my understanding is that GPT3 has 96 layers and 175 billion nodes (weights or parameters) arranged in various ways as part of the transformer model.
Comments.UserDisplayName: Nav
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-27T08:48:17.103Z
Comments.Text: It won’t have 175million nodes, if you think of a simpler neural network then the number of parameters is how many connections there are between nodes. If there was a NN with 1 layer of 2 neurons followed by a layer with 3 neurons, then an output layer with 1 neuron there would be 9 parameters (you could draw this out to see what I mean).
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T05:40:42.540Z
Comments.Text: Understood. The weights are vectors which are called the "learned parameters". Thank you very much, David. If you post it as an answer, I'll accept it. Hopefully, this would be of help to anyone else searching for this answer.
Comments.UserDisplayName: Nav
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-20T09:05:04.310Z
Comments.Text: Note that @DavidIreland is assuming a fully connected NN for 9 parameters
Comments.UserDisplayName: Jordan Morris
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-22T01:04:17.783Z
Comments.Text: and no biases, if you’re being pedantic @JordanMorris
Comments.UserDisplayName: David
Tags: <recurrent-neural-networks><open-ai><transformer><attention><gpt>
Title: What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?
UserDisplayName: Nav
ThreadText: # What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?
  
  (Asked by: username_0 on 2020-07-26)
  
  When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that they might just be sentences.
  
  [![enter image description here](https://i.sstatic.net/LWEEQ.png)](https://i.sstatic.net/LWEEQ.png)
  
  Even tutorial sites like [this one](https://www.analyticsvidhya.com/blog/2019/07/openai-gpt2-text-generator-python/) start talking about the usual parameters, but also say `"model_name: This indicates which model we are using. In our case, we are using the GPT-2 model with 345 million parameters or weights"`. So are the 175 billion "parameters" just neural weights? Why then are they called parameters? [GPT3's paper](https://arxiv.org/pdf/2005.14165.pdf) shows that there are only 96 layers, so I'm assuming it's not a very deep network, but extremely fat. Or does it mean that each "parameter" is just a representation of the encoders or decoders?
  
  [![enter image description here](https://i.sstatic.net/dthvC.png)](https://i.sstatic.net/dthvC.png)
  
  An excerpt from [this website](https://minimaxir.com/2019/09/howto-gpt2/) shows tokens:
  
  > In this case, there are two additional parameters that can be passed
  > to gpt2.generate(): truncate and include\_prefix. For example, if each
  > short text begins with a <|startoftext|> token and ends with a
  > <|endoftext|>, then setting prefix='<|startoftext|>',
  > truncate=<|endoftext|>', and include\_prefix=False, and length is
  > sufficient, then gpt-2-simple will automatically extract the shortform
  > texts, even when generating in batches.
  
  So are the parameters various kinds of tokens that are manually created by humans who try to fine-tune the models? Still, 175 billion such fine-tuning parameters is too high for humans to create, so I assume the "parameters" are auto-generated somehow.
  
  The [attention-based](https://arxiv.org/pdf/1706.03762.pdf) paper mentions the [query-key-value weight](http://jalammar.github.io/illustrated-gpt2/) matrices as the "parameters". **Even if it is these weights, I'd just like to know what kind of a process generates these parameters, who chooses the parameters and specifies the relevance of words? If it's created automatically, how is it done?**
  
  **Tags:** recurrent-neural-networks, open-ai, transformer, attention, gpt
  
  > **Comments:**
  > **username_1**: Parameters is a synonym for weights, which are what you learn. Batch size, learning rate etc are typically hyper parameters  
  > **username_0**: Thank you username_1. So now my understanding is that GPT3 has 96 layers and 175 billion nodes (weights or parameters) arranged in various ways as part of the transformer model.  
  > **username_1**: It won’t have 175million nodes, if you think of a simpler neural network then the number of parameters is how many connections there are between nodes. If there was a NN with 1 layer of 2 neurons followed by a layer with 3 neurons, then an output layer with 1 neuron there would be 9 parameters (you could draw this out to see what I mean).  
  > **username_0**: Understood. The weights are vectors which are called the "learned parameters". Thank you very much, username_1. If you post it as an answer, I'll accept it. Hopefully, this would be of help to anyone else searching for this answer.  
  > **username_2**: Note that @DavidIreland is assuming a fully connected NN for 9 parameters  
  > **username_1**: and no biases, if you’re being pedantic @username_2  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-28. Score: 24 | Accepted answer
  
  Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are *hyper-parameters* which basically means they are user specified, whereas weights are what the learning algorithm will learn through training.
  
  
  ---
  
  