MarianMT: doubling batch_size has no effect on time taken
## Environment info
- `transformers` version: 4.15.0
- Platform: Linux-5.4.0-97-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.6.13
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): 2.6.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten 

## Information

Model I am using (Bert, XLNet ...): MarianMTModel, MarianTokenizer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [Y] my own modified scripts: (give details below)

**Problem: Doubling the batch size does not improve the time taken to do backtranslation with MarianMTModel at all.**

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [Y] my own task or dataset: (give details below)

I am trying to do back translation for a given set of tweets (my own collection) using MarianMTModel and MarianTokenizer. With `batch_size=128`, it shows 1 hour with 4 s/it on A6000 GPU, whereas with `batch_size=256`, it still shows 1 hour with 7.7-8 s/it. FYI, I do see the steps halved using `tqdm` bar. 

## To reproduce

Steps to reproduce the behavior:

I am using the below script:
```
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# English to Romance languages
target_model_name = 'Helsinki-NLP/opus-mt-en-ROMANCE'
target_tokenizer = MarianTokenizer.from_pretrained(target_model_name)
target_model = MarianMTModel.from_pretrained(target_model_name).cuda()

# Romance languages to English
en_model_name = 'Helsinki-NLP/opus-mt-ROMANCE-en'
en_tokenizer = MarianTokenizer.from_pretrained(en_model_name)
en_model = MarianMTModel.from_pretrained(en_model_name).cuda()


def translate(texts, model, tokenizer, language="fr", num_beams=1):
    template = lambda text: f"{text}" if language == "en" else f">>{language}<< {text}"
    src_texts = [template(text) for text in texts]
    encoded = tokenizer.prepare_seq2seq_batch(src_texts, return_tensors='pt').to(device)
    translated = model.generate(**encoded, do_sample=True, max_length=256, top_k=0, num_beams=1, temperature=0.7)
    translated_texts = tokenizer.batch_decode(translated, skip_special_tokens=True)
    return translated_texts


def back_translate(texts, source_lang="en", target_lang="fr", num_beams=1):
    fr_texts = translate(texts, target_model, target_tokenizer, language=target_lang, num_beams=num_beams)
    back_translated_texts = translate(fr_texts, en_model, en_tokenizer, language=source_lang, num_beams=num_beams)
    return back_translated_texts


batch_size = 256 # 128
for i in tqdm(range(0, len(df1), batch_size)):
    rows = df1[i:i+batch_size]
    aug_text = back_translate(rows['Tweet'].tolist(), source_lang="en", target_lang="fr",num_beams=1)
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

With double the batch_size, time should reduce to half.

@patil-suraj Any updates on this issue?<|||||>Hey @kgarg8,

Why should doubling the batch size reduce the time in half exactly. Time measurements strongly depend on the hardware that is being used. What hardware do you use here?<|||||>Well, my GPU can support double the batch size. So, I can process double the batch size in one go. I am using A6000 (49GB).<|||||>Doubling the batch size should indeed reduce the time but it won't be exactly half. The GPU requires more time when processing larger batches. 

It's difficult to double check your scripts since `df1` is not defined. Could you maybe make a **short**, **reproducible** code snippet that we could copy-paste into a Python shell to run it?

Thanks!<|||||>**Minimal reproducible code:**

```
import transformers, torch, pandas as pd
from tqdm import tqdm
from transformers import MarianMTModel, MarianTokenizer

batch_size        = 256 # 128
filename          = 'sample_50k.csv'
device            = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
target_model_name = 'Helsinki-NLP/opus-mt-en-ROMANCE'
target_tokenizer  = MarianTokenizer.from_pretrained(target_model_name)
target_model      = MarianMTModel.from_pretrained(target_model_name).cuda()
en_model_name     = 'Helsinki-NLP/opus-mt-ROMANCE-en'
en_tokenizer      = MarianTokenizer.from_pretrained(en_model_name)
en_model          = MarianMTModel.from_pretrained(en_model_name).cuda()

def translate(texts, model, tokenizer, language="fr"):
    template = lambda text: f"{text}" if language == "en" else f">>{language}<< {text}"
    src_texts = [template(text) for text in texts]
    encoded = tokenizer.prepare_seq2seq_batch(src_texts, return_tensors='pt').to(device)
    translated = model.generate(**encoded, do_sample=True, max_length=256, top_k=0, num_beams=1, temperature=0.7)
    translated_texts = tokenizer.batch_decode(translated, skip_special_tokens=True)
    return translated_texts

def back_translate(texts, source_lang="en", target_lang="fr"):
    fr_texts = translate(texts, target_model, target_tokenizer, language=target_lang)
    back_translated_texts = translate(fr_texts, en_model, en_tokenizer, language=source_lang)
    return back_translated_texts


df1 = pd.read_csv(filename, usecols=[0, 1], encoding='ISO-8859-1')
df1.columns = ['Tweet', 'Target']

for i in tqdm(range(0, len(df1), batch_size)):
    rows = df1[i:i+batch_size]
    aug_text = back_translate(rows['Tweet'].tolist(), source_lang="en", target_lang="fr")
```

**Data:**
Download `sample_50k.csv` from [temp link](https://www.dropbox.com/t/OFlemA3vZHhO9vJl).

**Observations:**

`batch_size=256`
 ![Pasted Graphic 3](https://user-images.githubusercontent.com/30385849/162490206-5a6a7e33-95f2-4ee4-986c-5016fe9478f5.png)

`batch_size=128`
![Pasted Graphic 4](https://user-images.githubusercontent.com/30385849/162490242-5a6bc902-5408-48b3-9035-70c8028fab59.png)

So, with double the batch size, the processing becomes very slow (compare 4.37 s/it vs. 7.18 s/it) and the overall time is almost similar (24 mins vs. 28 mins).

Whereas with smaller batch sizes, we save around one-third the time (90 secs vs. 130 secs) with double the batch size.

`batch_size=32`
![Pasted Graphic 5](https://user-images.githubusercontent.com/30385849/162491520-6ac0cf8d-4a89-41d2-bfe1-aae5afb7fec2.png)

`batch_size=16`
![Pasted Graphic 6](https://user-images.githubusercontent.com/30385849/162491537-addffe0a-6129-4963-9d0f-543bdc470f20.png)

In my usecase, I have more than a million samples. With higher batch sizes like 256-1024, I practically don't save much time compared to batch_size of 128. I guess this tradeoff has to be judged based on multiple trials. And I am wondering if this applies to all Huggingface models.<|||||>From the code-snippet, it looks like the GPU is under-utilised here. Because while the text is being encoded, the GPU is just sitting idle. Maybe try to use a `Dataset/DataLoader` here to prepare the batches in the background while the model is translating other batches, so you can take advantage of async execution.

you could also try `pipeline` batching: https://huggingface.co/docs/transformers/main_classes/pipelines#pipeline-batching<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.