Hi @melody-ju,
T5 fine-tuning works well without fp16 and if you want to fine-tune t5-large but having memory issues then you can freeze the token embedings using the --freeze-embeds argument with finetune.py script.