Answers.Body: <p>It really depends on what you want to do with it, how you load it and what else you add.</p>
  <p>For inference at half-precision (16 bit per param) a 7B param models should be 7 billion * 2 bytes per param or 14GB. If you load it at full precision it will be a lot more. For training or tuning there are a lot of other states added for the optimizer and gradients. For a detailed discussion of what all is required for tuning and how to use approaches to train at scale please see the paper &quot;ZeRO: Memory Optimizations Toward Training Trillion Parameter Models&quot; (<a href="https://arxiv.org/abs/1910.02054" rel="nofollow noreferrer">https://arxiv.org/abs/1910.02054</a>). Deepspeed is an implementation of ZeRO.</p>
  <p>Additionally, for training you can use &quot;LoRA: Low-Rank Adaptation of Large Language Models&quot; (<a href="https://arxiv.org/abs/1910.02054" rel="nofollow noreferrer">https://arxiv.org/abs/1910.02054</a>) to reduce the amount of trainable parameters. As an example, on a 7B parameter model I get a reduction of 6.5 billion param values to about 540,000. PEFT is an implementation of LoRA.</p>
  <p>As a reference, when I execute:</p>
  <pre><code>import torch
  from transformers import AutoModelForCausalLM
  model = AutoModelForCausalLM.from_pretrained(&quot;decapoda-research/llama-7b-hf&quot;, torch_dtype=torch.bfloat16)
  model.cuda()
  </code></pre>
  <p>And I used <code>nvidia-smi</code> to check memory, my A100 is using 13845MiB.</p>
  <p>NOTE: If I do not use the <code>torch_dtype=torch.bfloat16</code> then the GPU utilization is 26729MiB. Most folks inference at 16 bit or even 8 bit quantization. For example: <a href="https://huggingface.co/blog/hf-bitsandbytes-integration/" rel="nofollow noreferrer">https://huggingface.co/blog/hf-bitsandbytes-integration/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-06T10:34:08.077Z
Answers.LastActivityDate: 2023-10-06T10:34:08.077Z
Answers.UserDisplayName: Andrew Mellinger
Answers.Body: <p>It varies depending on various factors such as quantization. My rough rule of thumb is memory need is 2-4x of the disk size. Just as an example, the model at <a href="https://huggingface.co/TheBloke/wizardLM-7B-HF/tree/main" rel="nofollow noreferrer">https://huggingface.co/TheBloke/wizardLM-7B-HF/tree/main</a> is about 14GB on disk and it used ~30GB (on CPU) just to load. I am sure inference will increase this further.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-01T23:51:28.663Z
Answers.LastActivityDate: 2023-07-01T23:52:02.030Z
Answers.LastEditDate: 2023-07-01T23:52:02.030Z
Answers.UserDisplayName: beejay
Answers.Body: <p>The following formula can be used to calculate GPU memory for serving LLMs:
  <a href="https://i.sstatic.net/2fEwf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2fEwf.png" alt="calculating-gpu-mem-formula" /></a></p>
  <p>Source and more examples: <a href="https://www.substratus.ai/blog/calculating-gpu-memory-for-llm" rel="nofollow noreferrer">https://www.substratus.ai/blog/calculating-gpu-memory-for-llm</a></p>
  <p>Disclaimer: I'm the author of that blog</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-14T08:44:00.247Z
Answers.LastActivityDate: 2024-01-14T08:44:00.247Z
Answers.UserDisplayName: Sam Stoelinga
Body: <p>It is easy to see the amount of disk space consumed by an LLM model (downloaded from huggingface, for instance). Just go in the relevant directory and check the file sizes.</p>
  <p>How can I estimate the amount of GPU RAM required to run the model?</p>
  <p>For example, if the Falcon 7B model takes around 14GB of storage, how much GPU RAM should suffice for it?</p>
  
Tags: <large-language-models><gpu><hardware><hardware-evaluation>
Title: For an LLM model, how can I estimate its memory requirements based on storage usage?
UserDisplayName: ahron
ThreadText: # For an LLM model, how can I estimate its memory requirements based on storage usage?
  
  (Asked by: username_0 on 2023-06-14)
  
  It is easy to see the amount of disk space consumed by an LLM model (downloaded from huggingface, for instance). Just go in the relevant directory and check the file sizes.
  
  How can I estimate the amount of GPU RAM required to run the model?
  
  For example, if the Falcon 7B model takes around 14GB of storage, how much GPU RAM should suffice for it?
  
  **Tags:** large-language-models, gpu, hardware, hardware-evaluation
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2023-10-06. Score: 2
  
  It really depends on what you want to do with it, how you load it and what else you add.
  For inference at half-precision (16 bit per param) a 7B param models should be 7 billion * 2 bytes per param or 14GB. If you load it at full precision it will be a lot more. For training or tuning there are a lot of other states added for the optimizer and gradients. For a detailed discussion of what all is required for tuning and how to use approaches to train at scale please see the paper "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" (https://arxiv.org/abs/1910.02054). Deepspeed is an implementation of ZeRO.
  Additionally, for training you can use "LoRA: Low-Rank Adaptation of Large Language Models" (https://arxiv.org/abs/1910.02054) to reduce the amount of trainable parameters. As an example, on a 7B parameter model I get a reduction of 6.5 billion param values to about 540,000. PEFT is an implementation of LoRA.
  As a reference, when I execute:
  import torch
  from transformers import AutoModelForCausalLM
  model = AutoModelForCausalLM.from_pretrained("decapoda-research/llama-7b-hf", torch_dtype=torch.bfloat16)
  model.cuda()
  
  And I used nvidia-smi to check memory, my A100 is using 13845MiB.
  NOTE: If I do not use the torch_dtype=torch.bfloat16 then the GPU utilization is 26729MiB. Most folks inference at 16 bit or even 8 bit quantization. For example: https://huggingface.co/blog/hf-bitsandbytes-integration/
  
  
  ---
  
  ### Answer by username_2 on 2023-07-01. Score: 2
  
  It varies depending on various factors such as quantization. My rough rule of thumb is memory need is 2-4x of the disk size. Just as an example, the model at <https://huggingface.co/TheBloke/wizardLM-7B-HF/tree/main> is about 14GB on disk and it used ~30GB (on CPU) just to load. I am sure inference will increase this further.
  
  
  ---
  
  ### Answer by username_3 on 2024-01-14. Score: 0
  
  The following formula can be used to calculate GPU memory for serving LLMs:
  [![calculating-gpu-mem-formula](https://i.sstatic.net/2fEwf.png)](https://i.sstatic.net/2fEwf.png)
  
  Source and more examples: <https://www.substratus.ai/blog/calculating-gpu-memory-for-llm>
  
  Disclaimer: I'm the author of that blog
  
  
  ---
  
  