Thanks for your reply. I checked again. They are not running on the same machines. For the problematic job, i killed it and re-run it. The job was scheduled on those 4 machines again and the speed is still 1000. Then, i submit the same job again, which are scheduled on another 4 machines. The new job is running 4k. So, the problem might be the issue of machines or rack as you suggested.
One more question is that if the network has some issues on those machines or straggler issues, would it be possible that the GPU is still utilizing 98%~100%? The GPUs are fully utilized and i was thinking there is no network issue.