The point is that what it is parallelize is the data so if we have a copy of model1 and model2 in same gpu we can do a forward no matter if they share or not and output. So half of the batch goes in one gpu and half in other. That is how I have understood it works.