ngimel:

There are many scenarios where speed-up from data parallel would not be that great - e.g. OpenNMT example hardly benefits from data parallel.


@ngimel @smth I’m running into a situation where using DataParallel ends up training slower than without it and on one gpu (while keeping everything else constant). If I try to increase the batch size, I run out of memory. I’m running an encoder - decoder (with attention) model with 3 million parameters. When running on one gpu, I’m able to run a batch size of 2048 sequences which takes up about 6000mb out of the 6078mb. Whereas on two gpus (using DataParallel on all layers in the encoder and decoder), running the same batch size takes up 6070mb on GPU 1 and only 1022mb on GPU 2.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 980 Ti  Off  | 0000:06:00.0     Off |                  N/A |
|  0%   51C    P2   167W / 300W |   6070MiB /  6078MiB |     73%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 980 Ti  Off  | 0000:07:00.0      On |                  N/A |
|  0%   45C    P2    93W / 300W |   1021MiB /  6075MiB |     35%      Default |
+-------------------------------+----------------------+----------------------+

I have tried putting the linear layers outside of DataParallel to no avail - machine ran out of memory.
I understand that the computations on GPU 1 require more memory than those on GPU 2, but I was expecting more memory to be used on GPU 2. Am I at the maximum capacity / performance? Is there anything I can do with this seq-to-seq model to train more batches at a time or shorten training time.