Solved by SimonW in post #2 
              
              
                Pay attention to this. volatile now does nothing, and you should wrap your validate loop in with torch.no_grad():. 
Your graph in validate won’t be freed until the output and loss variables are overwritten in the next iteration, and thus effective doubling memory usage.