I have tried using weighted sampling, but that doesn’t help. Assuming that two classes which are usually exclusive have a low chance, they will get assigned greater weight. But that happens for all 19 out of 20 classes, so that they are all now equiprobable, increasing the probability of being picked from 1/20 to 1/19.
Weighting the loss hasn’t helped either, even when I only weighted the false positives.
My code uses vgg16 and I do not fine-tune on Pascal Voc 2012:
model = VGG(make_layers([64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512,   512, 'M']))
for param in model.features.parameters():
    param.requires_grad = False

model.load_state_dict(torch.load('./saved_models/vggPretrained/vgg16.pth'))
new_classifier = nn.Sequential(*list(model.classifier.children())[:-1])
model.classifier = new_classifier
model.classifier.fc = nn.Linear(4096,20)

optimizer = optim.SGD(model.classifier.fc.parameters(), lr=0.0001)

multi_label_loss = nn.MultiLabelSoftMarginLoss()