How to generate texts in huggingface in a batch way?
I am new to huggingface. My task is quite simple, where I want to generate contents based on the given titles. The below codes is of low efficiency, that the GPU Util is only about 15%. It seems that it makes generation one by one. How can I improve the code to process and generate the contents in a batch way?

```
df_test = pd.read_csv("./ag_news/test.csv").sample(frac=1)
from transformers import pipeline
text_generator = pipeline("text-generation")
rows = df_test.sample(1000)
titles = rows['title'].tolist()
contents = rows['content'].tolist()
generate_texts = text_generator(titles, max_length=40, do_sample=False)
```
Looking at the [source code](https://github.com/huggingface/transformers/blob/4c32f9f26e6a84f0d9843fec8757e6ce640bb44e/src/transformers/pipelines/text_generation.py#L108) of the text-generation pipeline, it seems that the texts are indeed generated one by one, so it's not ideal for batch generation. 

In order to genere contents in a batch, you'll have to use GPT-2 (or another generation model from the hub) directly, like so (this is based on PR #7552):

```
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.padding_side = "left" 
tokenizer.pad_token = tokenizer.eos_token # to avoid an error
model = GPT2LMHeadModel.from_pretrained('gpt2')

device = 'cuda' if torch.cuda.is_available() else 'cpu'

texts = ["this is a first prompt", "this is a second prompt"]
encoding = tokenizer(texts, padding=True, return_tensors='pt').to(device)
with torch.no_grad():
    generated_ids = model.generate(**encoding)
generated_texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
```

In my case, this prints out: 
`['this is a first prompt for the user to enter the password.\n\nThe password is a string', "this is a second prompt, but it's not a full-screen one.\n\nThe first"]`<|||||>> Looking at the [source code](https://github.com/huggingface/transformers/blob/4c32f9f26e6a84f0d9843fec8757e6ce640bb44e/src/transformers/pipelines/text_generation.py#L108) of the text-generation pipeline, it seems that the texts are indeed generated one by one, so it's not ideal for batch generation.
> 
> In order to genere contents in a batch, you'll have to use GPT-2 (or another generation model from the hub) directly, like so (this is based on PR #7552):
> 
> ```
> from transformers import GPT2Tokenizer, GPT2LMHeadModel
> import torch
> 
> tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
> tokenizer.padding_side = "left" 
> tokenizer.pad_token = tokenizer.eos_token # to avoid an error
> model = GPT2LMHeadModel.from_pretrained('gpt2')
> 
> device = 'cuda' if torch.cuda.is_available() else 'cpu'
> 
> texts = ["this is a first prompt", "this is a second prompt"]
> encoding = tokenizer(texts, return_tensors='pt').to(device)
> with torch.no_grad():
>     generated_ids = model.generate(**encoding)
> generated_texts = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
> ```
> 
> In my case, this prints out:
> `['this is a first prompt for the user to enter the password.\n\nThe password is a string', "this is a second prompt, but it's not a full-screen one.\n\nThe first"]`

thanks. It seems that the standard workflow is to organize the components of `tokenizer`, `generate` and `batch_decode`
in a cascade way. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@NielsRogge  Is padding on the left still the way to go for batched generation? It seems odd to require a workaround for such a common feature