Looks like it’s a known dataloader bug, so all possible ideas are already mentioned in CPU memory gradually leaks when num_workers > 0 in the DataLoader · Issue #13246 · pytorch/pytorch · GitHub 6 by authors/people who are responsible for this code
dataloader is not part of distributed module and as you mentioned you don’t have any issues with distributed mode