Sorry for the late reply. This is a known issue and the google query you’re looking for is GPU memory imbalance. The correct answer is to choose a batchsize which is a multiple of 10. For instance, 10 GPUs should mean a batchsize of 10 or 20, 30 … so on.
Also, you’ll need to increase the number of workers to the capability of your system. Note that there is very little runtime benefit to having 10 GPUs (as my network admin learned after a gazillion hang ups). Data I/O will usually be the bottleneck. The only advantage in a 10 GPU system would be the insanely large batch size, i.e, higher learning rates.