I implemented something like @Dhorka’s example above, and the (simplified) following works in the forward pass, but…
class Network(nn.Module): #simplified
    def __init__(self, split_gpus):
        self.layerseq1 = nn.Sequential(OrderedDict(some layers))
        self.layerseq2 = nn.Sequential(OrderedDict(some layers))
        self.split_gpus = split_gpus
        if self.split_gpus:
            self.layerseq1.cuda(device=0)
            self.layerseq2.cuda(device=1)

    def forward(self, x):
        x = self.layerseq1(x)
        if self.split_gpus:
            x = x.cuda(device=1) 
        return self.layerseq2(x)

…it runs into problems when it tries to run the nn.MSELoss, claiming:
Assertion THCTensor_(checkGPU) [...] failed. Some of the weight/gradient/input tensors 
are on different GPUs. Please move them to a single one.
 at /opt/software/pytorch/aten/src/THCUNN/generic/MSECriterion.cu:13

While running an epoch such as:
criterion = nn.MSELoss()
for batch_idx, (data, target) in enumerate(some_dataset()):
     data= Variable(data).cuda()
     output = Network(data)
     loss = criterion(output, data)   # <-- fails at runtime
     optimizer.zero_grad()
     ...

I thought that couldn’t possibly mean that all trained layers need to be on the same GPU, so I tried moving both arguments to the same gpu before loss evaluation, (below), but that seems to have no effect
output.cuda(device=1)
data.cuda(device=1)

I’m still getting up to speed on pytorch, so any guidance would be appreciated…