But how come your accuracy is zero the whole time when training with lr=1e-3?
Even if your model predicts all zeros (e.g. background), there should be some background in your targets, so that the accuracy would be higher. Am I missing something in your use case?
Maybe there is another issue regarding the training / accuracy calculation?