Based on the error message it seems you might be close to the memory limit in the first epoch, and creating some additional tensors might push you over the edge.
Did you already wrap the training and validation loops in separate methods?
This could save some memory, as Python uses function scoping and would free all unused tensors after the function scope is left.
Also, I assume you are wrapping the validation loop in a with torch.no_grad() block?
If that’s the case, then you could

reduce the batch size
decrease the number of parameters or activations in your model
use native mixed-precision 1 with the nightly binaries or master build
trade compute for memory using torch.utils.checkpoint