Thanks for the pointer! I agree it looks relevant but having read through and tried some of the things suggested there I think it might be a different issue. For me the problem isn’t caused by having num_workers>0 in the dataloader. If I train on just 1 GPU with num_workers>0, there is no problem. If I train on multiple GPUs with num_workers=0, the problem does occur.
Having read some other documentation, I wonder if this is actually the normal behavior for DistributedDataParallel. E.g. this article says about DDP (emphasis mine):


Each worker maintains its own copy of the model weights and its own copy of the dataset.


However I’m not sure whether they mean this copy is maintained in VRAM on the corresponding GPU, or if it’s maintained in RAM through a separate subprocess for each GPU (the latter seems to be occurring in my case).