Solved by ptrblck in post #2 
              
              
                Your GPU doesn’t seem to have enough memory for the used batch size, so you would need to reduce the memory usage e.g. by lowering the batch size or by trading compute for memory via torch.utils.checkpoint.