Found the problem which was causing memory leak. The code I was using also uses itertools cycle() to iterate over another dataloader which was causing the memory leak.
Using iter(dataloader) does not cause any memory leak.