well, I’d first plot training loss to ensure it is decreasing epoch-to-epoch and maybe identify failure type from it (too slow or too fast learning, bad lr schedule, tiny gradients [not evident per se, need deeper inspection] or code problems)