I have an LSTM which shall predict a value. However, this prediction shall only be done after several calls. I.e. something like this:
Untitled Diagram.png721×236 782 Bytes

I am providing my LSTM input of the shape [sequence length, batch size, features]. As far as I understand, the returned output will than have a prediction for each element in the sequence (output (seq_len, batch, hidden_size * num_directions)). While I’m happy to just ignore the values I dont require, I’m not sure how to do the backpropagation. More specifically, what to provide my loss function as target.