As I understand, in model parallelism, you divide a model and train each part separately in different GPUs.
An example code I’ve found is this:

class Network(nn.Module):
def init(self, split_gpus):
self.module1 = (some layers)
self.module2 = (some layers)
    self.split_gpus = split_gpus
    if self.split_gpus:
        self.module1.device("cuda:0")
        self.module2.device("cuda:1")

def forward(self, x):
    x = self.module1(x)
    if self.split_gpus:
        x = x.device("cuda:1") 
    return self.module2(x)


My question is, is there a way to join 2 GPUs to be seen as a single GPU with double memory and not having to split the model?
Does NVIDIA NVLINK do this?
If not, what does NVLINK do?
Thanks!