Thanks, I will try that, although that would mean I cannot do transfer learning. As for loss function, optimizer and lr I’m using:
criterion = nn.BCEWithLogitsLoss(reduction='sum')
optimizer = optim.SGD(model.parameters(), lr=0.001)

Although I also tried Adam and different learning rates and l2 regularization values.