Answers.Body: <p><strong>Up-project and down-project refer to the first and second feed forward layer respectively, found in each transformer block.</strong></p>
  <p>They use this in the context of sparsity in the feedforward layers in each transformer block.</p>
  <p>Recall that each transformer block has two FF layers with ReLU activations. Each FF can be thought of as a &quot;projection&quot; to a different dimension. They noticed that with ReLU, the output of the first FF layer (the &quot;intermediate values&quot;) tends to be very sparse:</p>
  <blockquote>
  <p>The ReLU activation function naturally induces over 90% sparsity in the FFN’s intermediate outputs, which reduces the memory footprint for subsequent layers that utilize these sparse outputs. However, the preceding layer, namely the up project for OPT and Falcon, must be fully present in memory.</p>
  </blockquote>
  <p>Because of this sparsity, you don't want to load the entire FFN weights, so the authors propose efficient ways to do this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-30T01:29:58.953Z
Answers.Text: I suspect you are correct, and, in slightly more detail, that the authors want to load both the incoming weights for a particular neuron, as well as all the weights that will be multiplied by the output of that neuron in the next layer. One thing that made this hard for me to understand is that I don't think the terms "up project" or "down project" are standard for feedforward networks. At least not in the literature I have previously read.
Answers.UserDisplayName: Tyler
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-19T00:53:26.670Z
Answers.LastActivityDate: 2024-01-19T00:53:26.670Z
Answers.UserDisplayName: Alexander Wan
Answers.Body: <p>The answer could be found in the author's another paper.
  <a href="https://arxiv.org/pdf/2310.04564.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2310.04564.pdf</a></p>
  <p>For <code>FFN(x) = linerar1(relu(linear0(x)))</code>, the first <code>nn.Linear</code> is the &quot;up-project&quot; , the second <code>nn.Linear</code> is the &quot;down-project&quot;</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-15T04:06:13.450Z
Answers.LastActivityDate: 2024-03-18T03:01:50.920Z
Answers.LastEditDate: 2024-03-18T03:01:50.920Z
Answers.UserDisplayName: edimetia3d
Body: <p>In the paper, they first use the terms &quot;up projection layer,&quot; and similarly for down projection, in this paragraph in the introduction:</p>
  <blockquote>
  <p>Row-column bundling: We store a concatenated row and column of the up-projection and down-projection layers to read bigger contiguous chunks from flash memory. This increases throughput by reading larger chunks.</p>
  </blockquote>
  <p>What does this refer to in terms of the architecture of a given LLM?</p>
  <p>This paper focuses on the Falcon and OPT LLM models.</p>
  <p>To try to understand this, I've read through the paper for other explanations, and I searched online. One suggestion I found is that there are special layers in a network that (speaking informally) increase or decrease the size of the resolution, but I don't think that would apply to these specific LLM models. (For example, a max-pooling layer may be seen as decreasing resolution.)</p>
  <p>The image below is relevant, but I don't think it answers the question. <a href="https://arxiv.org/abs/2312.11514" rel="nofollow noreferrer">Here</a> is the paper in question.</p>
  <p><a href="https://i.sstatic.net/IfGEM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IfGEM.png" alt="Figure 7 from the paper" /></a></p>
  
Tags: <neural-networks><terminology><large-language-models><academia>
Title: In the paper "LLM in a flash," what is meant by an up projection or down projection layer?
UserDisplayName: Tyler
ThreadText: # In the paper "LLM in a flash," what is meant by an up projection or down projection layer?
  
  (Asked by: username_0 on 2024-01-19)
  
  In the paper, they first use the terms "up projection layer," and similarly for down projection, in this paragraph in the introduction:
  
  > Row-column bundling: We store a concatenated row and column of the up-projection and down-projection layers to read bigger contiguous chunks from flash memory. This increases throughput by reading larger chunks.
  
  What does this refer to in terms of the architecture of a given LLM?
  
  This paper focuses on the Falcon and OPT LLM models.
  
  To try to understand this, I've read through the paper for other explanations, and I searched online. One suggestion I found is that there are special layers in a network that (speaking informally) increase or decrease the size of the resolution, but I don't think that would apply to these specific LLM models. (For example, a max-pooling layer may be seen as decreasing resolution.)
  
  The image below is relevant, but I don't think it answers the question. [Here](https://arxiv.org/abs/2312.11514) is the paper in question.
  
  [![Figure 7 from the paper](https://i.sstatic.net/IfGEM.png)](https://i.sstatic.net/IfGEM.png)
  
  **Tags:** neural-networks, terminology, large-language-models, academia
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-01-19. Score: 2 | Accepted answer
  
  **Up-project and down-project refer to the first and second feed forward layer respectively, found in each transformer block.**
  
  They use this in the context of sparsity in the feedforward layers in each transformer block.
  
  Recall that each transformer block has two FF layers with ReLU activations. Each FF can be thought of as a "projection" to a different dimension. They noticed that with ReLU, the output of the first FF layer (the "intermediate values") tends to be very sparse:
  
  > The ReLU activation function naturally induces over 90% sparsity in the FFN’s intermediate outputs, which reduces the memory footprint for subsequent layers that utilize these sparse outputs. However, the preceding layer, namely the up project for OPT and Falcon, must be fully present in memory.
  
  Because of this sparsity, you don't want to load the entire FFN weights, so the authors propose efficient ways to do this.
  
  > **Comments:**
  > **username_0**: I suspect you are correct, and, in slightly more detail, that the authors want to load both the incoming weights for a particular neuron, as well as all the weights that will be multiplied by the output of that neuron in the next layer. One thing that made this hard for me to understand is that I don't think the terms "up project" or "down project" are standard for feedforward networks. At least not in the literature I have previously read.  
  
  ---
  
  ### Answer by username_2 on 2024-03-15. Score: 0
  
  The answer could be found in the author's another paper.
  <https://arxiv.org/pdf/2310.04564.pdf>
  
  For `FFN(x) = linerar1(relu(linear0(x)))`, the first `nn.Linear` is the "up-project" , the second `nn.Linear` is the "down-project"
  
  
  ---
  
  