Hello everyone,
I am working on BERT pre-training, and I wonder out loud if using multiple Trainer instances is the best way to go. As you may know, BERT pre-training consists of two-phases. These training phases are really distinct. Besides using different sequence lengths for each phase, they have different learning rates, batch sizes, etc.
It seems like a natural fit to have two sequential phases with two Trainer instance. Each Trainer instance takes different input training parameters, loaders, collators and so on. I would pass the model between Trainer instances. Once the first phase is done, the second is initialized with a new Trainer and the model is passed from the first Trainer instance to the second.
For the most part, this works ok. Do you envision a possibility of doing it any other way?