How to generate sentences in batches, instead of generating sentences one by one
After I finetune GPT-2, I want to use it to generate sentences in batches instead of one by one.

So I tried to modify the code of `examples/text-generation/run_generation.py`.


the code on line 239 in `run_generation.py` is:
`encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors="pt")`
the `prompt_text` is a `str` type data, but when I modify it to `List[str]` type data, It always return `50256`.

But looking at the source code, the type of `prompt_text` can be `str` , `List[str]` or  `List[int]`.

I tested this example separately , and for `token_ids` it always returns 50256.

![image](https://user-images.githubusercontent.com/22442305/91297615-344f1300-e7d1-11ea-99d4-b0e28cd1ab14.png)

So, the `prompt_text` must be `str` type data?

What modifications should I make to generate sentences in batches using `examples/text-generation/run_generation.py`?

Looking forward to your reply!

Hey @SuHe36,

I'm currently working on adding support for batched generation. At the moment, this is the best answer we can give you: https://github.com/huggingface/transformers/issues/3021#issuecomment-591236688<|||||>Thanks for your reply !<|||||>Hey, @patrickvonplaten is batch generation available for T5conditiongeneration?<|||||>Yes! Please take a look at this test, which does batch=4 generation for summarization using T5: https://github.com/huggingface/transformers/blob/55cb2ee62eb482787cff17585955f7193fe35dfa/tests/test_modeling_t5.py#L559<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
