ranger:

Can every network be ported for multi gpu if the model fits on single gpu?


Yes, I believe so. If your model does not fit on a single GPU though, there’s even another method that you can use (instead of DataParallel). E.g.,
class Network(nn.Module):
    def __init__(self, split_gpus):
        self.module1 = (some layers)
        self.module2 = (some layers)

        self.split_gpus = split_gpus
        if self.split_gpus:
            self.module1.device("cuda:0")
            self.module2.device("cuda:1")

    def forward(self, x):
        x = self.module1(x)
        if self.split_gpus:
            x = x.device("cuda:1") 
        return self.module2(x)