I cannot find any obvious error by skimming through your code.
However, do you see a static output? Could you remove the sigmoid and use nn.BCEWithLogitsLoss as your criterion? Maybe your output is saturated and your gradients are vanishing.