Which data parallel API are you using? DataParallel or Distributed Data Parallel (aka DDP)? With DDP you can use torch.distributed.barrier() function to explicitly synchronize your workers. Check out our docs 13 for further info.