Update:
I now tried appending each new hidden state h_t along one column of the image with
states = torch.cat((states, h_t.unsqueeze(0)))

and then append this tensor to the other columns with
context_buffer = torch.cat((context_buffer, states.unsqueeze(0)))

This also works, but is even slower.
For reference, when passing a batch of 32 images (84x64) the average times are:

Method: 0.5s Forward, 11s Backward
Method: 0.67s Forward, 13s Backward

I really hope someone knows a better solution to this…