This was my approach before incorporating attention - but the issue is that computing attention autoregressively requires that you use the output from all previous timesteps in the next timestep.
I could perhaps live with the .cat bottleneck, but i still cant help but wonder, what makes inplace assignment such a no-no? not just tensor[:, 1] = something, but tensor[:, 1] += something. It seems like as long as this operation is kept track of, this operation could be differentiable, and there would be no need to make duplicate tensors.