I have tried increasing the lr, but it dosen’t seem to change the basic behavior of what’s going on.
I’ve found that with the lstm, the BCE with logits error gets stuck at about 0.38, and the predictions are very close to each other on the 12 classes (as it was above.) However, if I take the softmax of the output, the three highest values are quite reliably the correct three classes! It’s still not useful as a classifier though because the values are so close, and I think that would lead to a lot of false positives.
I tried removing the lstm entirely, and now the error rate does not get stuck, and goes down to about .006 after 100 epochs. Furthermore, the outputs are varied in the way you would want. The softmax of the output shows very small probabilities for the “off” notes, and close to 1/3 for the “on” notes.
Is the moral of the story that I don’t need an lstm for this problem? I doubt that, because it’s a time series, and context in time is surely meaningful.
BTW I did change the lstm code using unsqueeze; that section of the code is currently as follows:
x = x.unsqueeze(0)
x, _ = self.lstm(x)
x = x.squeeze(0)