To check if it is indeed a distribution or issue of problem difficulty, I would try to first debug the system by replacing the validation set with the training set and seeing if the results match the training set.
If this works I would try splitting the training set in half and seeing if training on the first half of the training set generalizes well to the second.