meta-llama

/

Llama-4-Scout-17B-16E

like 162

Follow

Meta Llama 43.2k

I've been trying to run Llama-4-Scout-17B-16E on a single H100 but I keep getting the "CUDA out of memory" error. I'm not sure if I'm getting the quantization part right. I've been keeping an eye open on nvidia-smi as the model loads but memory usage seems under control. Could someone please advise?

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

model_name = "meta-llama/Llama-4-Scout-17B-16E"

bnb_config = BitsAndBytesConfig(load_in_4bit=True)

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto"
)

prompt = "Explain the theory of relativity in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
**inputs,
max_new_tokens=200,
do_sample=True,
top_p=0.9,
temperature=0.7
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("\nResponse:\n", response)

Thanks in advance!