Tokenizer batch_encode_plus unexpected behavior 
# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
bert-base-multilingual-cased

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
A script to tokenize sentences into inputs for the model

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Standard text that is used for intent and domain classification

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-multilingual-cased")
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=True)['input_ids']
>>>tensor([[  101, 61694, 10133, 15127, 11324, 10124, 14268,   102]])
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=False)['input_ids']
>>>tensor([[61694, 10133, 15127, 11324, 10124, 14268,     0,     0]])
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=True)['input_ids']
>>>tensor([[    0, 33600,    31,   759,  9351,    83,  3362,     2]])
tokenizer.batch_encode_plus(["hello my name is Sam"], return_tensors="pt", pad_to_max_length=True, add_special_tokens=False)['input_ids']
>>>tensor([[33600,    31,   759,  9351,    83,  3362,     1,     1]])
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

When running batch_encode_plus for a single example without adding special tokens and pad_to_max_length set to True, I would not expect any pad tokens. I did not see any mention in the documentation as to why this behavior is the expected norm. 
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: MacOS
- Python version: 3.7.3
- PyTorch version (GPU?): 1.5.0
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Yes, this was fixed on master today with https://github.com/huggingface/transformers/pull/5252
