Thanks for responding Bram!
Exactly, but I was curious why the transformers team is not using a model like Roberta for the pipeline when it can give so much better results. I’ve trained the large-Roberta model on the sentiment analysis dataset on kaggle 3 and could get to around 71 percent test accuracy with only 3 epochs and it is giving me very much more rational and accurate results in the unlabeled dataset too.