The usage of sigmoid on top of relu looks a bit strange, so you might want to check if removing the relu might help. Also, assuming you are using nn.BCELoss, replace it with nn.BCEWithLogitsLoss and remove the sigmoid for more numerical stability. Once this is done, try to overfit a small dataset (e.g. just 10 samples) by playing around with some hyperparameters.