Hi Cao,
With low memory utilization on all GPUs and a batch of 1 per GPU, you should try and increase the batch size per GPU. I took a brief look at the underlying code and it looks like it is explicitly hard coded to 1 per GPU (see this commit 7). If this can be modified to allow for >1 then you’re likely to see some speedups.
Good luck.