Have you tried overfitting your model on a tiny sample (like 2 or 3 datapoints) ?  That should let you at least know if your model is capable of learning or if there are some underlying issues with the data getting passed.