Yes, that’s correct. If the split is a rough split, you’ll still have to synchronize on the actual dataset size once preprocessing has completed. Distributed data parallel expects all workers to be called with an equal number of equally sized batches.