After training without DataParallel, the utilization graph becomes:

1.png1217×1261 31.9 KB

Now it’s assumable that the originally posted utilization graph is a result of lacking NCCL support in Windows platform.