I figured out that the problem was in loader instead of DDP. I had four workers for each loader, which were created repeatedly each time a new iter was created, with 8 processes it meant that 32 workers were created each time.
Reducing the number of workers solved the problem.