Starting this for results, sharing + tips and tricks, and results. This is my first attempt at this kind of thread so it may completely fail.
Some things I’ve found

Apparently if you copy AdaFactor from fairseq, as recommended by t5 authors, you can fit batch size = 2 for t5-large lm finetuning
fp16 rarely works.
for most tasks, you need to manually add </s> to the end of your sequence.

Thing’s I’ve read

task specific prefix doesn’t matter much.

cc @mrm8488 @valhalla @patrickvonplaten who have all tried different experiments.