Hi @ptrblck , the code I am using is spread among different files so as soon as I can I will create a code snippet and try to reproduce this issue.
In the meantime I want to share with you the findings:
1 GPU, batch=64, 0 workers:

1 GPU, batch=64, 7 workers:

2 GPUs, batch=128, 0 workers:

2GPUs, batch=128, 1 worker:

As you can see the last one has different processes spawned per GPU and is the only one causing OOM (batch=128 with 0 workers instead is fine). The screenshot was taken just before the OOM.
My DataLoader uses pin_memory=True, so I exclude to have CUDATensor inside the __getitem__, otherwise an exception occurs. I also used .to(device, non_blocking=True).