More generally, what sort of performance metrics should one expect to see when pre-training?
To be more specific, how long should we pre-train (days vs weeks), and what’s an acceptable loss? When should we stop? I would appreciate any references on this issue, thank you!