I haven’t done so myself for this task so can’t say much, but probably start with higher LR than the default one in the Trainer (which is 5e-5) as we are training from scratch, experiment with LR schedule, haparms search will definitely help choose the right params.
@sgugger might have better tips for this.