Using the model or any CUDA operations in the collate_fn is uncommon, I think (at least I haven’t seen a use case so far) and I guess not well tested.
The additional CUDA context creation might come from the usage of multiprocessing.
I also don’t know what kind of 0MB process is created as I haven’t seen it before.
In your code snippet you are also using multiprocessing with DDP. What’s the reason for not using DDP directly?