I’m unsure what exactly you’re trying to do, are you trying to run different parts of your model on different GPU? If so, @ailzhang’s answer here might be relevant.

  
    
    
    How to implement distributed model parallel autograd
  
  
    Hi @Ryobot, for distributed model parallel across different machines, it requires inter-node communication, right now we don’t have good support for it like DistributedDataParallel wrapper. 
For the single node multiple GPU model parallel( like DataParallel wrapper), yes parallel_apply should work in your case.
  


If not, can you describe what you’re trying to do in more detail?