Solved by chenglu in post #4 
              
              
                Tested on images. And I only have 2 GPUs, for the case of more GPUs the Batch Size should be larger than the number of the GPUs to make all GPUs work.