The "force_words_ids" does not seem to be available on llama4
#63
by
nlp-g
- opened
transformers=v4.51.1
The code snippet is as follows:
output = model.generate(**input_ids,do_sample=False,num_beams=2, max_new_tokens=1,force_words_ids=[tokenizer.convert_tokens_to_ids(['A', 'B', 'C', 'D'])])
The error message is as follows:
[rank0]: File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama4/modeling_llama4.py", line 379, in forward
[rank0]: attn_output, attn_weights = attention_interface(
[rank0]: File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama4/modeling_llama4.py", line 286, in eager_attention_forward
[rank0]: attn_weights = attn_weights + causal_mask
[rank0]: RuntimeError: The size of tensor a (8192) must match the size of tensor b (18) at non-singleton dimension 3