The problem in BATCH generation of GPT model
When I tried to use GPT model (including GPT-2, GPT-NEO-2.7B, GPT-J-6B, GPT-NEOX) to generate text, I found some strange results.

When I set the batch size to 1, all results are normal. 
BUT when I set the batch_size more than 1, such as 4, 8, ..., the generated text on GPT-J-6B,  GPT-NEOX is abmoral, which contains a large number of repeated consecutive letters or words, for example "AAAAAAAAAAAAAAA" or "The The The The The The The The The".

I can not find the root of this question. Could you please give some suggestions to solve this problem? Thank you!


Hi,

See this thread for batched generation: https://github.com/huggingface/transformers/pull/7552#issue-714062850<|||||>> Hi,
> 
> See this thread for batched generation: [#7552 (comment)](https://github.com/huggingface/transformers/pull/7552#issue-714062850)

Thanks a lot! 
And could you please tell me whether the current version supports correct sampling generation with batched setting? Thanks! <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Closing this as the issue seems resolved.