In that case all parameters get valid gradients and your overall training might be “stuck”.
Try to overfit a small dataset (e.g. just 10 samples) by playing around with the hyperparameters.
Once this is done you can try to scale up the problem.