Re #1: LSTM takes the whole sequence and performs each time step in the background. However, nothing is stopping you give LSTM just one word at a time. It depends on your task and how you want to implement it.
Re #2: I think (1) is not correct since you backpropagate multiple times over the same past time steps. (2) is the common approach for encoder-decoder models (e.g., autoencoder, machine translation) where the decoder generate words step by step. Sequence tagging is actually a bit more straightforward. If you look at your linked POS tagging tutorial, there is not loop needed. tag_scores = model(sentence_in) contains the output of all time steps. So you can calculate the loss in one go with loss = loss_function(tag_scores, targets)
Given that you want to do image captioning (i.e., generating text), you need a loop and do approach (2): accumulating the loss and call loss.backward() only once. Have a look at the Sequence-to-Sequence tutorial 20, particularly at the decoder part. Your decoder will roughly look the same. The only difference is that the hidden state won’t come from a text encoder but from some image encoder.
I hope that helps.