I am facing a very weird OOM error with my current training. The OOM error happens systematically during the forward pass of the last (or second last, not sure) batch of an epoch. It happens independent of training size. It happens before validation.
I am logging the GPU memory consumption via nvidia-smi during training. During the training epoch the memory consumption stays constant, so I doubt it’s a typical memory leak (caused e.g. by a missing .detach() call). However, when running the last batch, the memory consumption suddenly starts to increase in the forward pass. This can be seen by printing the GPU memory consumption during different steps in the forward pass.
Is there something special happening at epoch end, when the queue size of the dataloader approaches zero? Did anybody else face this phenomenon?
P.s.: While I said the error happens systematically, the training did work for at one time for a couple of epochs, so there does seem some randomness to it. However, it raised the OOM error at epoch end approximately a dozen times now.
P.P.s.: Training pipeline is a modified monodepth2 and I’m getting the error at this point:  https://github.com/nianticlabs/monodepth2/blob/master/trainer.py#L285 2 I am using only 1 GPU to train.