How to boost the speed of one sentence Marian Translation(no batches)?
I am using a Marian Model for translating from English to Arabic. I want to use this translation per sentence (no batching). 

I am using this simple code:

```
en_ar_tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-ar") 
en_ar_model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-en-ar")

def transformer_en_ar(sen):
    tokens = en_ar_tokenizer('>>ara<<' + sen, return_tensors="pt")
    translated = en_ar_model.generate(**tokens)
    decoded = en_ar_tokenizer.decode(translated[0], skip_special_tokens=True)
    return decoded

queries = [ 'some dummy sentence1',  'some dummy sentence2']
for query in queries:
    translation = transformer_en_ar(query)
```

Is there a way to increase the speed of inferring one sentence? Is there any software or hardware solution?

For example:
Shall I get rid of the for loop of the queries? How?
Will changing either (or both) interop and intraop threads have an effect?
Using GPU did not boost the speed of one sentence at a time? Am I doing something wrong?

Thanks
Not sure why you don't want to leverage batching, because it's always faster than passing each sentence at a time through the model:

```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-ar") 
model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-en-ar")

model.to(device)

sentences = ["Hello world",
            "How are you doing?", # use different length sentences to test batching
            ]
inputs = tokenizer(sentences, padding=True, return_tensors="pt")

output = model.generate(
    input_ids=inputs['input_ids'],
    attention_mask=inputs['attention_mask'],
)
tokenizer.batch_decode(output, skip_special_tokens=True)
```

If you're interested in optimizing the inference speed on CPU, take a look at [ONNX](https://huggingface.co/docs/transformers/serialization) or [Torchscript](https://huggingface.co/docs/transformers/serialization#torchscript). We [recently added support](https://github.com/huggingface/transformers/pull/14586) for ONNX for MarianMT.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.