I am able to reproduce the same behavior using imagenet example with tiny_image_dataset. Using a batch size of 256 in two gpus I get
image1390×262 39.8 KB
While using a batch size of 512 in two gpus, it gives
image1384×246 39.8 KB
Also, I assume this could be due to dataloader memory leak. If I use my entire dataset (120GB), I see out of memory (oom) kill before any batch is trained. I looked at pytorch discussion forum and looks like it is a very open issue. Any help solving this issue will be appreciated. Thanks.