T5 Decoder Inputs
# ❓ Questions & Help
Just confirming that my data preprocessing is perfect for T5. I added a print statement in `T5ConditionalGeneration` for the `decoder_input_ids` and `decoder_attention_mask` just before they're passed to the decoder. Which of these is right?

```
# pad_token prepended, eos_token is not in the sequence
decoder_input_ids: tensor([[0, 2018, 55, 0, 0, 0, 0]])
decoder_attention_mask: tensor([[1, 1, 1, 0, 0, 0, 0]])

# pad_token prepended, eos_token unmasked in attention
decoder_input_ids: tensor([[0, 2018, 55, 1, 0, 0, 0]])
decoder_attention_mask: tensor([[1, 1, 1, 1, 0, 0, 0]])

# pad_token prepended, eos_token masked in attention
decoder_input_ids: tensor([[0, 2018, 55, 1, 0, 0, 0]])
decoder_attention_mask: tensor([[1, 1, 1, 0, 0, 0, 0]])
```
your 2nd option is correct here: 

```python
# pad_token prepended, eos_token unmasked in attention
decoder_input_ids: tensor([[0, 2018, 55, 1, 0, 0, 0]])
decoder_attention_mask: tensor([[1, 1, 1, 1, 0, 0, 0]])
```

1) You have to start with `decoder_start_token_id = pad_token_id` in T5 and 
2) the last EOS token should be attended to because the model "should learn" when the sentence is finished.<|||||>@patrickvonplaten Thanks, Patrick! That makes perfect sense. You're awesome!<|||||>@patrickvonplaten I noticed that when you pass this to the model:

```
decoder_input_ids: tensor([[2018, 55, 1, 0]])
decoder_attention_mask: tensor([[1, 1, 1, 0])
```
`T5ConditionalGeneration` changes it to this before passing it to the decoder:

```
# Masks the eos_token
# Correctly prepends an extra pad_id to inputs BUT appends a pad_token to attention_mask
decoder_input_ids: tensor([[0, 2018, 55, 1, 0]])
decoder_attention_mask: tensor([[1, 1, 1, 0, 0])
```
So that you have to actually pass this initially:

```
# Pass this to model
decoder_input_ids: tensor([[2018, 55, 1, 0]])
decoder_attention_mask: tensor([[1, 1, 1, 1])

# Which turns into this
decoder_input_ids: tensor([[0, 2018, 55, 1, 0]])
decoder_attention_mask: tensor([[1, 1, 1, 1, 0])
```<|||||>Hey @alexorona - sorry I don't quite follow here...could you provide a code example that I can run to see what you mean? :-) 