I have a similar problem where I have a simple model which won’t overfit despite not using regularising techniques like dropout.
My model looks as follows:
Net(
  (feature_extractor): Sequential(
    (0): Conv2d(1, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (1): ReLU()
    (2): Conv2d(64, 128, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (3): ReLU()
    (4): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(128, 256, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (7): ReLU()
    (8): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Linear(in_features=65536, out_features=256, bias=True)
    (1): ReLU()
    (2): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (3): Linear(in_features=256, out_features=1, bias=True)
  )
)


and my training loop looks as follows:
lr = np.linspace(0.01,1./(1000**2),25)

torch.backends.cudnn.benchmark = True
torch.backends.cudnn.fastest = True

start = time.time()
model.train()
train_loss = []
train_accu = []
i = 0
for epoch in range(25):
    optimizer = optim.Adam(model.parameters(), lr=lr[epoch])
    for image, label in train_loader:
        data, target = image.float().cuda(), label.float().cuda()
        optimizer.zero_grad()
        output = model(data)
        loss = F.mse_loss(output, target.view(output.size()))  
        loss.backward()    # calc gradients
        train_loss.append(loss.data[0]) # Calculating the loss
        optimizer.step()   # update gradients
        prediction = torch.floor(output).int()  

        accuracy = (prediction.eq(target.view(output.size()).int()).sum().float()/batch_size)*100
        train_accu.append(accuracy)
        if i % 10 == 0:
            print('Epoch:',str(epoch),'Train Step: {}\tLoss: {:.3f}\tAccuracy: {:.3f}'.format(i, loss.data[0], accuracy))
            #print('TARGET:',target)
            #print('OUTPUT:',output.view(target.size()))
        i += 1
end = time.time()
print('TRAIN TIME:')
print('%.2gs'%(end-start))

I would have thought that after 10 iterations the model would have at least started to increase in accuracy towards overfitting but so far I’m just getting a bouncing of accuracy below ~30%.
Loss:

Accuracy:

TL,DR:
mse_loss
Adam optimizer
Learning rate decay
No dropout (but do include batch norm)
Not overfitting after 10 epochs.
Thanks in advance for any guidance on this issue!