First thing:
I have made a couple of plots to illustrate the facts that I cannot explain. I hope that I will have a clearer picture in mind with your help.


Plots

Screenshot 2018-11-08 at 21.36.53.png1242×734 55.5 KB


Screenshot 2018-11-08 at 21.37.16.png1242×734 51.9 KB


Why is it expectable for you? Please, elaborate why you have expected the non-positive trend in memory usage as batch size increases. Only V100 GPU:0 has a positive trend as expected.
Second thing:
Earlier I encountered a rather unusual behaviour of my 3x2080 Ti compared to 3x1080Ti. I had the same code, data, and set of deep learning libraries.
With 3x1080Ti I could feed a batch of size 3x85 (255) and memory was allocated evenly (a bit more on 0th as expected but not that much). The problem occurred when I tried to fit the same batch to 3x2080Ti. Of course, a 2080Ti has less memory but only 3x50 fitted well and memory allocation was unbalanced similar to this situation. Therefore, I started to think that non-0-th-GPUs actually have a proper amount of memory and the 0th one is inflated.