Hi @ptrblck. Thank you for your response. I am replying to your questions as inline below,



 ptrblck:

Based on the error message it seems you might be close to the memory limit in the first epoch


Apparently not. I have ~8Gb of memory left in the first iteration on both the GPUs.



 ptrblck:

Did you already wrap the training and validation loops in separate methods?


Yes, I don’t validate the model until the end of training as I’m Finetuning with pre-trained weights.



 ptrblck:

Also, I assume you are wrapping the validation loop in a with torch.no_grad() block?


Yes, it’s same as torchvision.references.detection.engine.evaluate where it’s decorated with @torch.no_grad().



 ptrblck:


reduce the batch size



My batchsize is 2. It works quite well on single GPU case but just not for the multi-GPU case.



 ptrblck:


trade compute for memory using torch.utils.checkpoint




Could you please elaborate this?
I am curious why training on 2 GPUs fail while it progresses well on single GPU?
Thanks again for your time and help @ptrblck