I changed the line below
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

To this enabled me to successfully train on the second gpu.
device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')

Nvidia-smi shows the result below when I am running on both of the gpus separately with 2 separate codes. This shows that both of my GPUs are successfully training separately, so the problem is in DataParallel?
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:0A:00.0  On |                  N/A |
| 41%   71C    P2    71W / 250W |   1854MiB / 11175MiB |     78%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:0B:00.0 Off |                  N/A |
| 23%   52C    P2   324W / 250W |   1481MiB / 11178MiB |     51%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1216      G   /usr/lib/xorg/Xorg                            18MiB |
|    0      1254      G   /usr/bin/gnome-shell                          50MiB |
|    0      1456      G   /usr/lib/xorg/Xorg                           170MiB |
|    0      1587      G   /usr/bin/gnome-shell                         141MiB |
|    0      9307      C   python3                                     1469MiB |
|    1      9575      C   python3                                     1469MiB |
+-----------------------------------------------------------------------------+