Hi! I’m finetuning a GPT-2 model (360M params = size M) on a huge training set (dozens of GB) with a low learning rate (15e-6, or 0.000015). The loss value dips at the start, and then gradually increases for a long time. I haven’t seen it start to fall again yet. Is this behavior normal/expected?

graph809×450 24.2 KB