I’m trying to train an image captioning model using 2 LSTMs, however, I’m confused with how to properly calculate the loss and make the network learn. The tutorials I see for LSTMs (not necessarily for image captioning) seem to pass in a whole sentence and instantly get back the loss for the whole sentence (as in https://pytorch.org/tutorials/beginner/nlp/sequence_models_tutorial.html 10 POS tagging).
For clarification #1:
I was under the impression that you have to pass each word to the LSTM timestep-by-timestep (which means there’s an inner for loop that iterates through each word), and each time, you have to pass in the previous word and the previous hidden state to the LSTM. Is this correct?
For clarification #2:
If #1 is correct, then, at each timestep, you get an output, and you incur a loss (for example, cross-entropy). To make the LSTM learn, do I (1) immediately call loss.backward() but run optimizer.step() after the whole sequence, or (2) accumulate the loss for all the timesteps, and only call accumulated_loss.backward() after the sequence (followed of course by optimizer.step())?
Would really appreciate the help. I’ve trained my network using method (2) but it doesn’t go lower than a certain point. I also tried running it on a small data set to make it overfit and preferrably memorize the training data just to see if it learns sequences, but the outputs don’t show any sentence structure. The highly probable words are almost always the same for all of the timesteps.
Thanks in advance!