PIDS for each device
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     23094      C   python                                     26309MiB |
|    1     23094      C   python                                     14341MiB |
|    2     23094      C   python                                     14341MiB |
|    3     23094      C   python                                     14341MiB |
|    4     23094      C   python                                     14341MiB |
|    5     23094      C   python                                     14341MiB |
|    6     23094      C   python                                     14341MiB |
|    7     23094      C   python                                     14341MiB |
+-----------------------------------------------------------------------------+

Using distributed training will help here?