@ptrblck, thanks for your suggestion! This is a great way to trade compute speed with memory. However, I think I can apply this method to both single GPU and multiple GPU calculation. The memory issue can be alleviated in the same time but the imbalanced memory usage for multiple GPU still exits: Its limit depends on single GPU memory limit.
So far, I had a better experience using single GPU training than multiple GPU training. For example, 1GPU+bsz =20 VS. 2GPU+bsz=40 VS. 4GPU+bsz=80,  only about 13% performance improvement is observed when #GPU doubled. In the same time, the convergence speed decreases, since for more GPU and larger batch size, total number of updates decreases in a fixed training time.
In my case, the data is huge. I only care about the accuracy after a certain number of epochs (1~ 5). Although, a larger batch size could end up with better accuracy, I’m far from that point. Therefore, single GPU training is more practical to me at present, unless there is an elegant solution.
I wonder how other deep learning frameworks handle this issue?