Answers.Body: <p>(V)RAM requirement for inference: <a href="https://blog.eleuther.ai/transformer-math/" rel="nofollow noreferrer">Model size &lt; RAM requirements &lt; Model size * 1.2</a>.</p>
  <p>Also <a href="https://huggingface.co/spaces/hf-accelerate/model-memory-usage" rel="nofollow noreferrer">https://huggingface.co/spaces/hf-accelerate/model-memory-usage</a> is helpful, e.g.:</p>
  <p><a href="https://i.sstatic.net/ED1gj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ED1gj.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-07T05:08:07.993Z
Answers.LastActivityDate: 2024-10-07T05:08:07.993Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>I have a request to host a multi-modal LLM to process documents with charts and handwriting. I was thinking about testing Llama 3.2 90B.</p>
  <p>What is the hardware requirement for it?</p>
  <p>Where can I find some figures about the performance on a server with NVIDIA A100x?</p>
  
Tags: <large-language-models><hardware>
Title: Hardware requirement for multi-modal LLMs
UserDisplayName: MstfLux
ThreadText: # Hardware requirement for multi-modal LLMs
  
  (Asked by: username_0 on 2024-10-03)
  
  I have a request to host a multi-modal LLM to process documents with charts and handwriting. I was thinking about testing Llama 3.2 90B.
  
  What is the hardware requirement for it?
  
  Where can I find some figures about the performance on a server with NVIDIA A100x?
  
  **Tags:** large-language-models, hardware
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-10-07. Score: 2
  
  (V)RAM requirement for inference: [Model size < RAM requirements < Model size \* 1.2](https://blog.eleuther.ai/transformer-math/).
  
  Also <https://huggingface.co/spaces/hf-accelerate/model-memory-usage> is helpful, e.g.:
  
  [![enter image description here](https://i.sstatic.net/ED1gj.png)](https://i.sstatic.net/ED1gj.png)
  
  
  ---
  
  