hey @dennlinger when you say:



 dennlinger:

However, once I go and train with the larger dataset, RoBERTa models no longer show any signs of convergence and instead just predict nonsense.


do you mean that both the training and validation loss don’t decrease or something else?
the two main parameters i’ve needed to tune XLM-R (not RoBERTa exactly, but close enough) effectively for text classification have been the learning rate and number of warmup steps, with the former generally needing to be in the 10^-6-10-5 range.
in the RoBERTa paper they used 6% of the total training steps for the warmup, so perhaps you could see whether increasing this to say 100,000 steps helps in your 2M samples training set.
what task are you working on?