For anyone having the similar issue: try to overfit the model with one example as a sanity check on the model implementation. Obvious problems tend to pop up with this simple yet classic debugging method.