Body: <p>I'm trying to understand these two points from an article:</p>
  <blockquote>
  <ul>
  <li>Models with <strong>large variables</strong> i.e weight matrices. As a consequence such models have correspondingly <strong>large gradients</strong> and optimizer states. The activations (intermediate outputs from the model layers) tend to be relatively small (depends on the batch size). Typically fully connected networks and RNNs fall under this category.</li>
  <li>Models with <strong>small weights</strong> but <strong>large activations</strong>. CNNs and transformers tend to fall under this category.</li>
  </ul>
  </blockquote>
  <p>This is my attempt to understand it:</p>
  <p>My current understanding is that large variables would mean large matrices, and the corresponding gradient matrices are of the same size. In fully connected neural network the activation of one layer would be the same size as the next layer. So activation should usually be 1 dimension.</p>
  <p>Now the confusing part: A model with small weights can have large activations -- I think this is because the same (small) weight matrices(e.g. 3x3 in CNN) are reused. While the activations are of dimension <span class="math-container">$(n-(k-1)) \times (n-(k-1))$</span>, where <span class="math-container">$k$</span> is the width of the kernel.</p>
  <p>Is my understanding correct?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-16T08:26:52.520Z
Comments.Text: Could you please provide the link to the article to have more context?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-16T09:25:43.737Z
Comments.Text: Sure. Thanks for your help in advance. This is the [GitHub link](https://github.com/pidajay/addons/blob/grad_checkpointing_eager/docs/tutorials/training_gradient_checkpointing.ipynb), the two quoted points can be found at the start of the ipynb file. I was trying to understand when could this package be applied. @nbro
Comments.UserDisplayName: NRain
Tags: <neural-networks><machine-learning><terminology><weights><gradient>
Title: What do "large variables" and "small weights" mean in these sentences?
UserDisplayName: NRain
ThreadText: # What do "large variables" and "small weights" mean in these sentences?
  
  (Asked by: username_0 on 2022-05-13)
  
  I'm trying to understand these two points from an article:
  
  > * Models with **large variables** i.e weight matrices. As a consequence such models have correspondingly **large gradients** and optimizer states. The activations (intermediate outputs from the model layers) tend to be relatively small (depends on the batch size). Typically fully connected networks and RNNs fall under this category.
  > * Models with **small weights** but **large activations**. CNNs and transformers tend to fall under this category.
  
  This is my attempt to understand it:
  
  My current understanding is that large variables would mean large matrices, and the corresponding gradient matrices are of the same size. In fully connected neural network the activation of one layer would be the same size as the next layer. So activation should usually be 1 dimension.
  
  Now the confusing part: A model with small weights can have large activations -- I think this is because the same (small) weight matrices(e.g. 3x3 in CNN) are reused. While the activations are of dimension $(n-(k-1)) \times (n-(k-1))$, where $k$ is the width of the kernel.
  
  Is my understanding correct?
  
  **Tags:** neural-networks, machine-learning, terminology, weights, gradient
  > **Comments:**
  > **username_1**: Could you please provide the link to the article to have more context?  
  > **username_0**: Sure. Thanks for your help in advance. This is the [GitHub link](https://github.com/pidajay/addons/blob/grad_checkpointing_eager/docs/tutorials/training_gradient_checkpointing.ipynb), the two quoted points can be found at the start of the ipynb file. I was trying to understand when could this package be applied. @nbro  
  
  ---
  
  ## 0 answers
  
  