Solved by ptrblck in post #2 
              
              
                The intermediate tensors, which are needed to calculate the gradients, the gradients themselves, as well as the CUDA context will all use memory on your device, which could explain the OOM issue. 
You could try to decrease the batch size, if possible, and check the memory usage for a smaller batch …