CUDA memory error on increasing the number of generations
I am getting CUDA memory error when generating text with num_return_sequences set to more than 100 for a self-trained gpt2 model. This is not expected since after every generation there should be nothing left in the GPU.




By setting `num_return_sequences` you're creating bigger batches, so it is expected to have an OOM if you ask for too much returned sequences.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.