Seq2Seq: decoder hidden_states shape not tested
[this line] points at encoder hidden states for Bart.

https://github.com/huggingface/transformers/blob/2ee410560e45ae3c619dc1e0b0fc4d257c48e18a/tests/test_modeling_common.py#L464
same for `T5`. I thought a bit about how to correctly sort the `encoder_hidden_states`, `decoder_hidden_states` output (if the user wants to output it). I think it's not easy at all to implement this cleanly...namedtuples would make this so much easier, so maybe just wait until we add those? <|||||>Makes sense. 
Are namedtuples on anybody's roadmap?<|||||>Not really as far as I know! It would be a big change (lots of code has to be adapted). In my opinion it would be best to start with the outer-most outputs (the returned `outputs` of the models) and see how that goes:
- How easy is it to have everything backwards compatible?
- How much cleaner does the code get in `generate()` this way?
- How much code has to be added for named tuples? <|||||>(Just assigning myself so that I can easily find our discussion again)<|||||>Hi @patrickvonplaten,
for a distillation purpose of T5, I want to return the `deocder_hidden_states`. using this:

```
from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")
# training
input_ids = tokenizer("The <extra_id_0> walks in <extra_id_1> park", return_tensors="pt").input_ids
labels = tokenizer("<extra_id_0> cute dog <extra_id_1> the <extra_id_2>", return_tensors="pt").input_ids
outputs = model(input_ids=input_ids, labels=labels)
loss = outputs.loss
logits = outputs.logits
print(outputs.decoder_hidden_states) # None!!
```
<|||||>Can you add `output_hidden_states=True` to `model(...)`? 