TT_YY:

By the way, I wonder why the latency in your experiment is one digit lower than mine. Are you using the same model (resnet34) and CIfar10?


As I don’t have a models package locally, I replaced that with torchvision.models, and then replaced all ResNet* with reset*. I am not sure if that’s also what you use.

Would you please specify the model and data that you used to produce the results in your former post?

I used the default model type in your code, so it should be torchvision.models.resnet34() after my replacement, and was using the same data loader.

Are the GPUs that you used utilize NVLink?

Yes, the comm should go through nvlink
        GPU0    GPU1    CPU Affinity
GPU0     X      NV4     12-23
GPU1    NV4      X      12-23

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.116.00   Driver Version: 418.116.00   CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro GP100        Off  | 00000000:81:00.0 Off |                    0 |
| 26%   31C    P0    30W / 235W |     10MiB / 16278MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Quadro GP100        Off  | 00000000:82:00.0 Off |                    0 |
| 26%   31C    P0    30W / 235W |     10MiB / 16278MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+