Training a MLP. [70 6000 6000 4] network clearly overfits on data. Tried adding dropout layers to prevent overfitting. Ran into out of memory error.
Line of thought as follows:


Too many parameters, hence not fitting in 12Gb Tesla K80 GPU


Tried multi-GPU training by replacing if cuda: model.cuda() with model=nn.DataParallel(model).cuda(). Two 12Gb Tesla K80 GPUs run out of memory.


Tried training very small model (below) on 1 GPU. Still out ran out of memory. This network is smaller than even a MNIST MLP.


Specs:

Windows Server
Cuda 8
Python 3.6.1 (conda install)
PyTorch v0.4.0

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.bn1 = nn.BatchNorm1d(70)
        self.fc1 = nn.Linear(70, 100)
        self.d1 = nn.Dropout(0.5)
        self.fc2 = nn.Linear(100, 100)
        self.d2 = nn.Dropout(0.5)
        self.fc3 = nn.Linear(100, 4)

    def forward(self, x):
        x = x.view(-1, 70)
        x = self.bn1(x)
        x = F.relu(self.fc1(x))
        x = self.d1(x)
        x = F.relu(self.fc2(x))
        x = self.d2(x)
        return F.log_softmax(self.fc3(x))


model = Net()
if cuda:
    model.cuda()

optimizer = optim.Adam(model.parameters(), lr=0.00001, betas=(0.9, 0.999), weight_decay=0.0)

Checked discussions of few other reported cases of memory leaks.

Added torch.backends.cudnn.enabled = False after all import statements. Didn’t help

volatile =True during validation

Any fixes?