If the model is working during training, I assume it’s failing during validation?
If so, your memory usage might be still too high once you start the validation loop and you should check if unnecessary tensors are kept alive (e.g. the last loss tensor, which could still be attached to the computation graph).