From my beginner-level understanding, when it comes to BERT, sometimes people will train just the last layer, or sometimes they’ll train all layers a couple epochs and then the last layer a few more epochs.
Does T5 have any similar practices? Or is it normal to just train the whole thing when fine-tuning?
And very tangentially related: to fine-tune T5, we just do loss.backward() on the result of the forward() call of the T5 model right (the loss key in the returned dict)? So there’s no need to calculate any loss on our own?