Wrong `special_tokens_mask` when using `facebook/opt-350m`
### System Info

The logic behind the computation of `special_tokens_mask` seems a bit broken when using the OPT model (might also be true for other models but I have not been able to reproduce the behaviour yet). 

```python 
tokenizer = transformers.AutoTokenizer.from_pretrained("facebook/opt-125m")
result = tokenizer(
    [
        ("Two radio engineers got married.", "The reception was fantastic."),
        ("Atheism is", "a non-prophet organization.")
    ],
    padding=True,
    return_tensors='pt',
    is_split_into_words=False,
    return_special_tokens_mask=True,
    return_token_type_ids=True
)
```


### Who can help?

@LysandreJik this is related to the #allenai-colab issue mentioned by `Dirk Groeneveld`.


The part of the `tokenizer` code that is interacted with is the following : 

```python 
        # Add special tokens
        if add_special_tokens:
            sequence = self.build_inputs_with_special_tokens(ids, pair_ids)
            token_type_ids = self.create_token_type_ids_from_sequences(ids, pair_ids)
        else:
            sequence = ids + pair_ids if pair else ids
            token_type_ids = [0] * len(ids) + ([0] * len(pair_ids) if pair else [])

        # Build output dictionary
        encoded_inputs["input_ids"] = sequence
        if return_token_type_ids:
            encoded_inputs["token_type_ids"] = token_type_ids
        if return_special_tokens_mask:
            if add_special_tokens:
                encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(ids, pair_ids)
            else:
                encoded_inputs["special_tokens_mask"] = [0] * len(sequence)
```

 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The problem is that `get_special_tokens_mask` will only return ` [0] * ((len(token_ids_1) if token_ids_1 else 0) + len(token_ids_0))` at this point. 


### Expected behavior


In order to get the expected behaviour, the call should be : `encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(sequence, already_has_special_tokens = True)` which makes sense as at this point, `sequence` should contain special tokens. 