dr.lauren:

I have also tried using just an lstm, with pretty much identical results to the above. The BCE with logits loss stalls out pretty quick, the outputs are all very close to 0.76. But when you take the softmax of the output, it’s something like 10 or 20 percent for the “on” notes and about 5 percent for the off notes. Increasing the layers of the lstm changes things a little, but the overall behavior is the same.


This might be indeed due to a high learning rate.
nn.BCEWithLogitsLoss will internally apply a log_sigmoid and thus expects raw logits as the model output. If your model trains better with an additional softmax this could mean that smaller gradients might be beneficial for your model.