Solved by pietern in post #4 
              
              
                On calling forward it splits the input into multiple chunks (one chunk per GPU), replicates the underlying model to multiple GPUs, runs forward on each of them, and gathers the outputs.