I think I found the issue. After tinkering around, I realized the nn.Sigmoid layer was the problem. It seems like the BCELoss was the issue. If I swapped to the BCEWithLogitsLoss, it seems like the loss behaves and the weight matrices actually change now! I’m not really sure why BCELoss causes this issue though, and whether it’s the numerical instability that Pytorch indicates. Nevertheless, I think it solves the mystery? Any idea how to tell if it’s due to the numerical instability or when we need to use BCEWithLogitsLoss over BCELoss?