Hey @kaka_zhao, this is very interesting finding. We didn’t have any time-based algorithm in DDP. Is there any other user share the same cluster with you? I wonder if it is possible that some recurrent job is kicked off every day at 23:00, which will compete for resources on network/GPU with your job?