Your code sample runs just fine on colab 

Yeah no the sample runs depending on your gpu capabilities/ memory. I recommend scaling up the batch_size (in line train_data_loader = Dataloader(batch_size=6, size=512) to a higher one until you encounter the problem (so that the first 50 iterations run without problems but then a Cuda OOM is thrown for iter > 50)

Also the fact that the memory reported by the OS changed is expected. We use a special allocator that does not return the memory to the OS as soon as it is freed to improve speed.
So the bump here could be due to the change of allocation pattern after the error. But no further increase after that because the pattern remains the same.

So you’re saying that some extra memory is allocated to the autograd graph gradients that is never released? And cannot be released in any way?