Hey,
My training is crashing due to a ‘CUDA out of memory’ error, except that it happens at the 8th epoch. In my understanding unless there is a memory leak or unless I am writing data to the GPU that is not deleted every epoch the CUDA memory usage should not increase as training progresses, and if the model is too large to fit on the GPU then it should not pass the first epoch of training.
Any advice on how to debug this issue will be greatly appreciated as I am not sure myself where to even begin. Could faulty garbage collection be responsible? If so, are there any settings I can pass to Python’s/Pytorch’s GC to prevent this crash from occurring at the cost of performance? Are there any known bugs that could be the cause for this and if so are there any known workarounds? Does anyone know of any tools that can help me pinpoint the cause of this issue, such as a CUDA memory monitor that can tell me by how much is memory usage increasing every epoch?