Did you find a solution to this? I’m working on an audio to text(sentence) model (LSTMs, attention…)
The model always predicts the stop token. For every word of the sentence. The model learns to do this by the middle of the first epoch, after a few iterations having seen only a part of my data.
The loss is Cross Entropy Loss. The optimizer is Adam but the problem persists with SGD albeit converging to predicting stop token later.
The model is quite large and the data is as well. I imagine this happens because the stop token class is very skewed because the stop token is present in every training example output sentence? I’ve not been able to find a workaround for this and am new to training LSTMs for this problem. ANY help is appreciated.