YAML Metadata
Warning:
empty or missing yaml metadata in repo card
(https://huggingface.co/docs/hub/model-cards#model-card-metadata)
GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer, sym=False, desc_act=True, group_size=32)
Run inference
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
import time
model_id = "meta-llama/Llama-3.2-1B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained("gptq-llama-3.2-1B-Instruct")
prompt = "write me a 100-word essay on the topic of the history of the United States"
print(tokenizer.decode(model.generate(**tokenizer(prompt, return_tensors="pt", max_length=512).to(model.device))[0]))
- Downloads last month
- 22