I had a similar issue. Changing BCELoss to BCEWithLogitsLoss and Adam epsilon from 10**(-8) to 10**(-4) worked for me.
I found it useful to inspect the computed gradients when debugging.
print([p.grad for p in model.parameters()])