Answers.Body: <p>GPU DRAM capacity - 7.5G
  the below link explains how nVIDIA GPU cUDNN does memory optimization.
  <a href="https://devblogs.nvidia.com/optimizing-recurrent-neural-networks-cudnn-5/" rel="nofollow noreferrer">https://devblogs.nvidia.com/optimizing-recurrent-neural-networks-cudnn-5/</a>
  the below link has detailed steps to calculate memory required by parameters and data.
  <a href="http://cs231n.github.io/convolutional-networks/#case" rel="nofollow noreferrer">http://cs231n.github.io/convolutional-networks/#case</a>
  one data point missing in question is number of output classes in softmax layer.
  The above two links will help you to calculate memory required and how software handles large matrix multiplication. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-17T20:04:29.503Z
Answers.Text: Thank you. The number of output classes is 5000, which is the number of vocabulary. Apparently, I didn't need to multiply 5000, so the memory consumption is at the order of 10MB.
Answers.UserDisplayName: Math.StackExchange
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-17T18:35:57.343Z
Answers.LastActivityDate: 2018-09-17T18:35:57.343Z
Answers.UserDisplayName: Sreedhar Veluri
Body: <p>I'm training a language model with <code>5000</code> vocabularies using a single <code>M60 GPU</code> (w/ actually usable memory about 7.5G). 
  <br>The number of tokens per batch is about <code>8000</code>, and the hidden dimension to the softmax layer is <code>512</code>. So, if I understand correctly, fully-connected (softmax) layer theoretically consumes <code>5000*8000*512*4=81.92GB</code> for a forward pass (4 is for float32).
  <br>But the GPU performed the forward and backward passes without any problem, and it says the GPU memory usage is less than <code>7GB</code> in total. </p>
  
  <p>I used PyTorch. What's causing this? </p>
  
  <p>EDIT: To be clearer, the input to the final fc layer (256x5000 matrix) is of size [256, 32, 256]. </p>
  
Tags: <neural-networks><natural-language-processing>
Title: Calculation of GPU memory consumption on softmax layer doesn't match with the empirical result
UserDisplayName: Math.StackExchange
ThreadText: # Calculation of GPU memory consumption on softmax layer doesn't match with the empirical result
  
  (Asked by: username_0 on 2018-09-17)
  
  I'm training a language model with `5000` vocabularies using a single `M60 GPU` (w/ actually usable memory about 7.5G).
    
  The number of tokens per batch is about `8000`, and the hidden dimension to the softmax layer is `512`. So, if I understand correctly, fully-connected (softmax) layer theoretically consumes `5000*8000*512*4=81.92GB` for a forward pass (4 is for float32).
    
  But the GPU performed the forward and backward passes without any problem, and it says the GPU memory usage is less than `7GB` in total.
  
  I used PyTorch. What's causing this?
  
  EDIT: To be clearer, the input to the final fc layer (256x5000 matrix) is of size [256, 32, 256].
  
  **Tags:** neural-networks, natural-language-processing
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-09-17. Score: 1 | Accepted answer
  
  GPU DRAM capacity - 7.5G
  the below link explains how nVIDIA GPU cUDNN does memory optimization.
  <https://devblogs.nvidia.com/optimizing-recurrent-neural-networks-cudnn-5/>
  the below link has detailed steps to calculate memory required by parameters and data.
  <http://cs231n.github.io/convolutional-networks/#case>
  one data point missing in question is number of output classes in softmax layer.
  The above two links will help you to calculate memory required and how software handles large matrix multiplication.
  
  > **Comments:**
  > **username_0**: Thank you. The number of output classes is 5000, which is the number of vocabulary. Apparently, I didn't need to multiply 5000, so the memory consumption is at the order of 10MB.  
  
  ---
  
  