Sure. Can you just check one more thing - see if between the training and validation phase,  you are moving the model to and from gpu to cpu.
And, if possible try to delete the batches at the end of training and validation like del images; del labels