Experimenting with a 4 GPU AWS instance setup to run batch inference on a segmentation network.
I varied the num_workers with batch size and found that I could not improve the volatile GPU memory utilization beyond 50%.
When num_workers increased beyond 400, the 4th GPU could not allocate enough memory. More details in the plot below.

image.jpg1500×1032 244 KB

Here was the output of the nvidia-smi.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.111                Driver Version: 384.111                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:00:1B.0 Off |                    0 |
| N/A   41C    P0   193W / 300W |   9116MiB / 16152MiB |     78%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  Off  | 00000000:00:1C.0 Off |                    0 |
| N/A   42C    P0   199W / 300W |   7624MiB / 16152MiB |     79%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  Off  | 00000000:00:1D.0 Off |                    0 |
| N/A   39C    P0   231W / 300W |   7624MiB / 16152MiB |     74%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   35C    P0    51W / 300W |    794MiB / 16152MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+