Yes, you can split your single model across multiple-GPUs in PyTorch with minimum fuss. Here is an example from @apaszke  :
class MyModel(nn.Module):
    def __init__(self, split_gpus):
        self.large_submodule1 = ...
        self.large_submodule2 = ...

        self.split_gpus = split_gpus
        if split_gpus:
            self.large_submodule1.cuda(0)
            self.large_submodule1.cuda(1)

    def forward(self, x):
        x = self.large_submodule1(x)
        if split_gpus:
            x = x.cuda(1) # P2P GPU transfer
        return self.large_submodule2(x)

One caveat (to min. fuss) is that you probably want to try several split points for optimal GPU memory consumption across multiple devices! [Here] (https://gist.github.com/ajdroid/420624cdd6643c397b3a62c68904791c 117) is a more fleshed out example with VGG-16