Thanks.
Actually, I found the dataloader could be the potential bottle neck.
The loading time is nearly proportional to the batchsize.
I will make another thread about this to focus on the loading time issue.