Switching Dataloaders made no difference, both failed during the evaluation step and training worked fine, so it must be something wrong with the evaluation code. It shouldn’t be any specific example as the test/train examples are chosen randomly at the initalisation stage. There must be some kind of funadmental difference between the inference and training of the model that is resulting in OOM error.
Also removing the LR scheduler made no difference.