Basically everything or nothing could be wrong. 
It’s hard to tell the reason your model isn’t working without having any information.
I think a generally good approach would be to try to overfit a small data sample and make sure your model is able to overfit it properly.