You can choose whatever format that works well for you, only thing to note is your dataset or collatorshould return input_ids, attention_mask and labels.


To add new tokens


tokenizer.add_tokens(list_of_new_tokens)

# resize the embeddings
 model.resize_token_embeddings(len(tokenizer))


Using task prefix is optional.
No, you won’t need to register the task, the original T5 repo requires that but it’s not required here.

You might find these two notebooks useful

Fine-tune T5 for Classification and Multiple Choice 85
Fine-tune T5 for Summarization 136
Train T5 on TPU 24


Note: These notebooks manually add the eos token  (</s>), but it’s not with the current version, the  tokenizer  will handle that.

Here’s a great thread on tips and tricks for T5 fine-tuning
  
    
    
    T5 Finetuning Tips Models
  
  
    Starting this for results, sharing + tips and tricks, and results. This is my first attempt at this kind of thread so it may completely fail. 
Some things I’ve found 

Apparently if you copy AdaFactor from fairseq, as recommended by t5 authors, you can fit batch size = 2 for t5-large lm finetuning
fp16 rarely works.
for most tasks, you need to manually add </s> to the end of your sequence.

Thing’s I’ve read 

task specific prefix doesn’t matter much.

cc @mrm8488 @valhalla @patrickvonplaten who…