Can you perhaps post a more complete sample. This one doesn’t have backward(...) which actually removes the graph and frees up memory. I’m assuming you omitted it to post pseudocode. From the example:
    window = torch.cat((new_element.unsqueeze(-1), window), dim=0)

The above particular line possibly keeps things in memory. If it’s okay for you, try:
    window = torch.cat((new_element.detach().unsqueeze(-1), window), dim=0)