I am training a domain adaptation model based on Faster R-CNN. And I already use model parallel to train it. While my single RTX 2080ti can handle the source training(ie train normally on source), I constantly get a CUDA out of memory error when forwarding two images from source and target at the same time even with the model splitted in 4 2080tis. I found that when without splitting, it always ran out of memory during extractor(vgg16 here), so I split vgg16 into three parts onto three GPUs using nn.Sequential. But I still get CUDA out of memory from the second part:
[Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
 ReLU(inplace=True),
 Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
 ReLU(inplace=True),
 MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)]

even if I freeze the parameters of this part and its previous part.
so what’s wrong with my using of model parallel or anything?