T5: forward and generate produce different results even for greedy decoding of a single token
Tagging @sshleifer following earlier discussions. 

## Environment info
     
- `transformers` version: 3.2.0 (also 3.0.2)
- Platform: Linux-4.4.0-17763-Microsoft-x86_64-with-glibc2.10  (WSL, but also on normal Ubuntu)
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: Fails independently of this
- Using distributed or parallel set-up in script?: Fails independently of this

## Information

I am using the T5 model to train on a seq2seq task. The model.forward() and model.generate() can differ, even for greedy decoding of a single token, since model.generate() seems to add a padding token before generating. This matters for "classification-style" tasks where we usually decode a single token (positive/negative for sentiment for instance).


The problem arises when using:
* [X] my own modified scripts: (give details below)

Very similar to example: this code is enough:
```python

from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch

tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")


def run_comparison(input_sent):
    input_ids = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(input_sent))
    input_ids = torch.tensor([input_ids])

    target_ids = tokenizer.convert_tokens_to_ids(tokenizer.tokenize("chocolate </s>"))
    target_ids = torch.tensor([target_ids])

    res_fwd = torch.argmax(model(input_ids=input_ids, labels=target_ids)[1], -1)
    res_gen = model.generate(input_ids=input_ids, max_length=2)
    
    print("Running comparison for %s" % input_sent)
    print("Using model.forward(): ", tokenizer.decode(res_fwd[0]), res_fwd[0])
    print("Using model.generate(): ", tokenizer.decode(res_gen[0]), res_gen[0])

run_comparison("I love </s>")
```

Outputs:

Running comparison for I love </s>
Using model.forward():  and tensor([ 3, 11])
Using model.generate():   tensor([0, 3])


## To reproduce

Steps to reproduce the behavior:

1. Run code above.

## Expected behavior

model.forward() and model.generate() give the same output for single token greedy decoding. 

Part  discrepancy here is that during `generate` we put `decoder_start_token_id` at the front of the output, tell the model to predict the next token, then append that next token to the end of the output. 
For `forward`, at the first position, we tell the model to predict the next token conditional on the previous token in `decoder_input_ids`, which should be `pad_token=0`, but we don't have any append step since everything is done in parallel.

If you ignore the leading zero, have you found examples where `generate` and `forward` produce different outputs?
<|||||>Yes, though for outputs of length >= 1, where I guess it would be expected since forward is not autoregressive whereas generate is.

Playing around with a few sentences, it seems like behaviors/results are the same, iff:
`model.generate(input_ids=input_ids, max_length=2)`  (1 breaks because it counts padding) and discard the first padding. This sounds reasonable (albeit a bit undocumented) and I guess the leading 0 is ignored when calling decode so most users don't run into this.

<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Closing given discussion above,