<p>(V)RAM requirement for inference: <a href="https://blog.eleuther.ai/transformer-math/" rel="nofollow noreferrer">Model size &lt; RAM requirements &lt; Model size * 1.2</a>.</p>
<p>Also <a href="https://huggingface.co/spaces/hf-accelerate/model-memory-usage" rel="nofollow noreferrer">https://huggingface.co/spaces/hf-accelerate/model-memory-usage</a> is helpful, e.g.:</p>
<p><a href="https://i.sstatic.net/ED1gj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ED1gj.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2024-10-07T05:08:07.993Z
2024-10-07T05:08:07.993Z
Franck Dernoncourt
<p>I have a request to host a multi-modal LLM to process documents with charts and handwriting. I was thinking about testing Llama 3.2 90B.</p>
<p>What is the hardware requirement for it?</p>
<p>Where can I find some figures about the performance on a server with NVIDIA A100x?</p>

<large-language-models><hardware>
Hardware requirement for multi-modal LLMs
MstfLux
# Hardware requirement for multi-modal LLMs

(Asked by: username_0 on 2024-10-03)

I have a request to host a multi-modal LLM to process documents with charts and handwriting. I was thinking about testing Llama 3.2 90B.

What is the hardware requirement for it?

Where can I find some figures about the performance on a server with NVIDIA A100x?

**Tags:** large-language-models, hardware


---

## 1 answers

### Answer by username_1 on 2024-10-07. Score: 2

(V)RAM requirement for inference: [Model size < RAM requirements < Model size \* 1.2](https://blog.eleuther.ai/transformer-math/).

Also <https://huggingface.co/spaces/hf-accelerate/model-memory-usage> is helpful, e.g.:

[![enter image description here](https://i.sstatic.net/ED1gj.png)](https://i.sstatic.net/ED1gj.png)


---

