When using two GPUs, the speed is
Train Epoch: 0 [900/18745 (0.048)]; Acc: 0.929; time cost: 0.722
When using one GPU, the speed is
Train Epoch: 0 [15800/18745 (0.843)]; Acc: 0.905; time cost: 0.461
The codes are written as follows:
model = models.resnet18(pretrained=True)  
model =torch.nn.DataParallel(model).cuda()
x=x.cuda(async=True)# there is no difference no matter whether we include async=True or not
yt=yt.cuda(async=True)#
output = model(x)

When using two GPUs, the output is recorded as follows:
+------------------------------------------------------+
| NVIDIA-SMI 352.79     Driver Version: 352.79         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla M40           Off  | 0000:06:00.0     Off |                    0 |
|  0%   56C    P0    74W / 250W |   2440MiB / 11519MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla M40           Off  | 0000:87:00.0     Off |                    0 |
|  0%   37C    P0    87W / 250W |   1854MiB / 11519MiB |     97%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     16788    C   python                                        1874MiB |
|    0     56331    C   python                                         298MiB |
|    0     58531    C   python                                         207MiB |
|    1     16788    C   python                                        1797MiB |
+-----------------------------------------------------------------------------+

When using one GPU,  the output is recorded as follows:
+------------------------------------------------------+
| NVIDIA-SMI 352.79     Driver Version: 352.79         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla M40           Off  | 0000:06:00.0     Off |                    0 |
|  0%   71C    P0   233W / 250W |   3878MiB / 11519MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla M40           Off  | 0000:87:00.0     Off |                    0 |
|  0%   26C    P8    18W / 250W |     55MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     33037    C   python                                        3312MiB |
|    0     56331    C   python                                         298MiB |
|    0     58531    C   python                                         207MiB |
+-----------------------------------------------------------------------------+

How can we improve the efficiency using two GPUs?