Discrepancy in the pad_token_id between the tokenizer and the model code of the T5
- `transformers` version: 2.11.0
- Platform: Darwin-19.3.0-x86_64-i386-64bit
- Python version: 3.7.6

 tokenizers: @mfuntowicz
 T5: @patrickvonplaten

It seems that there's a discrepancy between the tokenizer and the model code of the T5 regarding the pad_token_id.
looking at the following output the pad_token_id is 0:
```
from transformers import T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained('t5-large')
tokenizer.pad_token_id
Out[4]: 0
```

When calculating the loss. I would expect the calculation to ignore the paddings, but when looking at the code, it ignores token_id = -100.
modeling_t5.py:
```
        if lm_labels is not None:
            loss_fct = CrossEntropyLoss(ignore_index=-100)
            loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), lm_labels.view(-1))
            # TODO(thom): Add z_loss https://github.com/tensorflow/mesh/blob/fa19d69eafc9a482aff0b59ddd96b025c0cb207d/mesh_tensorflow/layers.py#L666
            decoder_outputs = (loss,) + decoder_outputs
```



Hi @eyal-str , `pad_token_id` is indeed 0 for T5. But it's a convention to use -100 as ignore index in `CrossEntropyLoss`, so when preparing `labels` the `pad_token_ids` are replaced by -100. <|||||>@patil-suraj Thanks for the prompt reply. I just saw what you meant in the DataCollatorForLanguageModeling.mask_tokens function.