Just an idea…
You could keep the hidden Variables in a python list and do something like this…
new_hidden = some_calculation(..)
hiddens.append(new_hidden)
while len(hiddens) > max_history:
    hiddens.pop(0)
hiddens[0].detach_() # stops backpropagation from going any further back
# do stuff with hiddens

As far as I can tell from its source code .detach() will remove any prior history a Variable has, but still keep the graph connections to any Variables calculated using its value.
Can someone else confirm whether this would work?