Here is a minimal working example that produces the error:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import matplotlib.pyplot as plt

class DNN(nn.Module):
    def __init__(self,dim):
        super(DNN,self).__init__()
        self.fc=nn.Sequential(
            nn.Linear(dim,dim),
            nn.Tanh(),
            nn.Linear(dim,dim)
        )
    def forward(self,data):
        return self.fc(data)

def main():
    E=20
    T=20
    D=5
    dim=3
    model=DNN(dim)
    optimizer=torch.optim.Adam(model.parameters())
    loss_fn=nn.MSELoss()
    data_in=torch.zeros(E,T,dim)
    data_out=torch.zeros(E,T,dim)
    data_target=torch.zeros(E,T,dim)
    for e in range(E):
        for t in range(T):
            data_in[e,t]=torch.from_numpy(np.random.rand(dim))
            data_out[e,t]=model(data_in[e,t])
            data_target[e,t]=torch.from_numpy(np.random.rand(dim))
        if e>=D:
            loss=0.0
            for t in range(T):
                loss += loss_fn(data_target[e-D,t],data_out[e-D,t])
            optimizer.zero_grad()
            loss.backward(retain_graph=True)
            optimizer.step()
    
if __name__=='__main__':
    main()

Error message:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 3]] is at version 120; expected version 119 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Basically the idea is, I want to train the model one by one with the old samples while testing/generating the new samples, i.e. in an online manner. In particular, at episode e, I want to train the model with the sample (output and target) from episode (e-D) while generating the new output of episode e. I have no idea how to do this other than slicing, but slicing (I believe) causes the in-place operation error.