I was indeed speaking of the Volatile-GPU-util. percentage! The GPU memory is hard to fill without causing some errors, even though it’s possible to go higher than that by playing with the batch size, but I’d say 9.2/12 is good usage.
Regarding the utilization, a drop to 0% every other second is not so good, and especially if it’s as low as 50% the other second…
But if increasing workers does not help, I would investigate the loading / preprocessing (preprocessing first). Try disabling data augmentation and check the time before/after. (Also you might want to print processing time for a full mini-batch as well as only the loading part (i.e. after you get into the for loop iterating on the data loader). This helps to see if most of the time is spent loading / preprocessing data or after that (network, loss computing etc.).