meta-llama

/

Meta-Llama-3.1-8B-Instruct

like 2.51k

pip install --upgrade transformers trl

worked for me

Hi @Chan-Y @thunderdagger

Thank you for the note. I was stuck with the rope_scaling issue when I executed the following code from HF. Your tip helped me to resolve the issues. Thanks again.

import transformers
import torch

model_id = "meta-llama/Meta-Llama-3.1-8B"

pipeline = transformers.pipeline(
"text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto"
)

pipeline("Hey how are you doing today?") #******

Now, the command is running at the last line (where it has *****) but it looks like it has been running for a while (more than 5 mins). Is that expected?

Thanks for your help.

Suresh

I ran your code on colab, it took 13.5 seconds to generate answer

@Chan-Y - Wow. Thanks for doing it Chan. I was running it in Google Colab. Does this cause this problem in Colab?

It was giving this msg when it was stuck at the pipeline() command above.

Setting pad_token_id to eos_token_id:128001 for open-end generation.
/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1259: UserWarning: Using the model-agnostic default max_length (=20) to control the generation length. We recommend setting max_new_tokens to control the maximum length of the generation.
warnings.warn(

Sorry, it was this msg that appeared yesterday which I failed to mention it here.

Thanks for your help.