You could try to remove the sigmoid, if you think it might kill the gradients.
If that doesn’t help, a good debugging step is to try to overfit a small dataset first (e.g. just take 10 samples) by playing around with some hyperparameters and make sure your model and training routine is able to do so.
Once this is done, you could scale up the use case again by using more data.
If your model isn’t able to overfit a tiny dataset, then there might be a real bug in the code (e.g. you might have forgotten to zero out the gradients) or the model is just unable to learn the distribution (which I don’t think would be the case given your explanation of the use case).