Yeah！ I think I understand your answer,another question: When I using multi GPU,is it necessary to set the batch size to be even number? I find that my dual GPU Machine with default setting, the first GPU memory usage it nearly full,but the other is only 1/2.Is there any reason why the GPU:0 uses far more than the other?Can batch size be a odd number? Because the current setting seems that I left nearly 6GB without using,while another is full-time working.