Solved by jia_lee in post #6 
              
              
                Not totally. But I warped the loss into the network to alleviate this issue to some extent, look at this link: 
 
 
Alternatively, I recommend to use distributed training.