You can change a to a list, append Lx1 tensors to it, and use torch.stack after the loop. Also, if self.w is scalar, your loop can be replaced with nn.RNNCell applied to rescaled x. (if it is a trainable vector, it is too tied to W_ih to be useful, I think).