No, it doesn’t learn anything. Tried with different LRs. Are there certain specific things involved when training from scratch with these large models ? BERT-base improves from 31 to 58, whereas BERT-large stays at 31.