T5 - Finetuning of an EncoderDecoder Model
Hello,

I know that the T5 implementation is quite new, but is there already code to finetune and lateron decode from the T5 model? 

As I understand most of your models are no EncoderDecoder models, so I guess that the default pipeline / code is not working for T5, is that right?

Could you point me to a script / command / piece of code for finetuning T5?
As I know, there is **no** Python scripts for fine-tuning T5 model, **at the moment**.
Besides the source code you can see in this library, you can see the PR #1739 which implements T5 model.

> Hello,
> 
> I know that the T5 implementation is quite new, but is there already code to finetune and lateron decode from the T5 model?
> 
> As I understand most of your models are no EncoderDecoder models, so I guess that the default pipeline / code is not working for T5, is that right?
> 
> Could you point me to a script / command / piece of code for finetuning T5?<|||||>The same question.  #1739 was merged. First of all, In T5_INPUTS_DOCSTRING is said:
```
            To match pre-training, T5 input sequence should be formatted with [CLS] and [SEP] tokens as follows:
            (a) For sequence pairs:
                tokens:         [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
            (b) For single sequences:
                tokens:         [CLS] the dog is hairy . [SEP]

```

At second, it looks like T5Model can work in encoder mode only. So, it's possible to treat it as usual LM:

```
        tokenizer = T5Tokenizer.from_pretrained('t5-small')
        model = T5Model.from_pretrained('t5-small')
        input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
        outputs = model(input_ids)
        last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
```

Maybe @thomwolf can clarify how better to fine-tune T5 for classification tasks<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
