Probably training is not complete or the model does not have enough capacity to train. Get a batch and try to overfit the model on that batch. If it does, then it means the model has the power to atleast learn on the data you provided.