Yeah, this makes sense. But I tried to repeat the training process several times and kept getting exactly similar ‘flat region’. Since dataset are randomly shuffled by DataLoader, it shouldn’t get into the saddle point every time, right?
Or is this phenomenon caused by some intrinsic nature of the model / dataset used?