Thank you for your advice. The problem is dividing the training samples into equal batches leading to unbalance GPU utility. The default GPU whose index is 0 usually has obvious more memory occupation. So, I think we should split the batches manually to maximize the utilities of multi GPUs.