Suddenly I noticed that the virtual memory usage is huge during my training. The size grows when the first Tensor is passed to GPU. Also, I noticed that using more GPU is much slower than training using one GPU. I am almost certain that the training time on 2 GPUs what almost 2 times faster than using one GPU.
I am using ubuntu, PyTorch 1.0.1 and CUDA 10.
Here I can show htop output of training on one GPU

Zrzut ekranu 2019-03-04 o 18.54.35.png2172×1068 936 KB

Does anyone have the idea what’s going on?