Hi @rvarm1, thanks for reply. I’m not sure if it is a bug, since the program doesn’t hang, but it affects the speed greatly. And if I reduce the sync frequency, it can help to speed up, so I feel the communication between GPUs may be a bottleneck.