PyTorch should make it really easy for you to implement either alternative, but the first technique seems easier. You can simply have a termination condition, and keep generating newer outputs in a sequence until that termination condition is met. You could then compute the loss and backprop.


This really depends on the kind of data you’re using. To me it seems like you could sample subsequences of varying lengths and train on those. (This isn’t really a PyTorch question.)


LSTM hidden states change due to updates from whatever gradient-based learning scheme you end up using. I see no reason why these first 20 dimensions should be part of a hidden state if these are not learnable. It’s not possible to keep these fixed in standard RNN/LSTM implementations.