Cannot Replicate xlm-roberta-large-xnli Results
## Environment info
- `transformers` version: 4.9.0
- Platform: Linux-5.4.0-1055-aws-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): 2.5.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
Models:

- albert, bert, xlm: @LysandreJik
- xlm-roberta-large-xnli @joeddav 

## Information

Model I am using (Bert, XLNet ...): joeddav/xlm-roberta-large-xnli

The problem arises when using:
* [X] the official example scripts: 
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from transformers import pipeline # no difference in manually encoding sentences and passing them through the model
import torch # the use of torch or tensorflow make absolutely no difference at all

device = 0 if torch.cuda.is_available() else -1
tokenizer = AutoTokenizer.from_pretrained("joeddav/xlm-roberta-large-xnli")

model = AutoModelForSequenceClassification.from_pretrained("joeddav/xlm-roberta-large-xnli")

sequence_to_classify = "Seriously, ANY possible sentence in any language."
candidate_labels = ["tecnologia", "cibo", "bevande", "finanza", "cinema", "giochi"]

classifier = pipeline("zero-shot-classification",
                      model=model, tokenizer=tokenizer, device=device)

#hypothesis_template = "This text is about {}." # within its addition, values are even more different
classifier(sequence_to_classify, candidate_labels, hypothesis_template, multi_class=False)
```
* [ ] my own modified scripts: 

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset:
just trying to classify sentences with zero-shot classification on a not-finetuned model

## To reproduce

Steps to reproduce the behavior:

1. Use joeddav/xlm-roberta-large-xnli model for zero-shot classification (with or without pipeline, with any sentence in any language, using either torch or tensorflow)
2. Notice that results are different from **Hosted inference API** of that model, you can try them either using the model page interface or remote API calls

## Expected behavior

Shouldn't the results coming from the use of the model, correspond to the ones coming from the model web page?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.