Solved by 111344 in post #4 
              
              
                I think there is no difference between gpu=2 or 3. 
In my experiment: 
batch-size=8  gpu=2  -->batch_size=4 for single gpu 
batch-size=8  gpu=3  -->batch_size=2 for single gpu(so total batch_size is 6) 
batch-size=8 or 6, under normal circumstances, it does not have much impact on performance 
For s…