Solved by mrshenli in post #3 
              
              
                I’m wondering whether there’s a way to make PyTorch take multiple gpu as a single one, so that we don’t have to split model manually. 

Currently, this is not available, we are working on adding a model partitioning feature. 
Manually splitting model shouldn’t be too hard with today’s PyTorch API, …