Model converges to the almost (I think it is due to dropouts) the same prediction for most inputs. Loss slowly decreases to some stable high point. LSTM based models achieve N times better score.
Do you think the problem might be in the model definition or training process?