@Yanli_Zhao thank you for helping me out !
As suggested by @ptrblck I switched profiler and now use “nvidia nsight” profiler.

Original model (high latency in data transfert but low in synchronization)
As expected there is not so much variance from one epoch to another. Each epoch describes a clear pattern and each epoch pattern has a “batch” sub-partern. The fact that those are low variance makes me think that the problem is not transient (eg. network throttling, …).

Master
About a third (≈310ms per epoch) is spent in cudaMemcpyAsync . Each epoch starts with a suspiciously  longer call (≈110ms) to cudaMemcpyAsync . The picture below is a zoom on one epoch.

master3770×1575 423 KB


Worker 1
Could worker synchronization time be spent waiting for that first long call to cudaMemcpyAsync  on master to be finished ?

worker-03770×1575 255 KB


No distribution
For the reference, the suspiciously  longer call (≈110ms) to cudaMemcpyAsync  does not occur when there is no distribution.

no-dist3770×1575 564 KB

Could that suspiciously long calls to cudaMemcpyAsync have something to do with hardware issue ?