Hi @tom, thank you so much for your reply. Yes, taking the gradient first is totally fine for my application. I have revised my (actual) code based on your suggestions, and it is now working great.
This is how the minimal working example should look like based on the suggestions:
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

class DNN(nn.Module):
    def __init__(self,dim):
        super(DNN,self).__init__()
        self.fc=nn.Sequential(
            nn.Linear(dim,dim),
            nn.Tanh(),
            nn.Linear(dim,dim)
        )
    def forward(self,data):
        return self.fc(data)

def main():
    E=50
    T=20
    D=5
    dim=4
    model=DNN(dim)
    optimizer=torch.optim.Adam(model.parameters())
    loss_fn=nn.MSELoss()
    data_in=[[torch.zeros(dim) for t in range(T)] for e in range(E)]
    data_out=[[torch.zeros(dim) for t in range(T)] for e in range(E)]
    data_target=[[torch.zeros(dim) for t in range(T)] for e in range(E)]
    optimizer_list=[None for e in range(E)]
    for e in range(E):
        for t in range(T):
            data_in[e][t]=torch.from_numpy(np.random.rand(dim)).float()
            data_out[e][t]=model(data_in[e][t])
            data_target[e][t]=torch.from_numpy(np.random.rand(dim)).float()
        loss=0.0
        for t in range(T):
            loss=loss_fn(data_target[e][t],data_out[e][t])
        optimizer_list[e]=torch.optim.Adam(model.parameters())
        optimizer_list[e].zero_grad()
        loss.backward()
        if e>=D:
            optimizer_list[e-D].step()
    
if __name__=='__main__':
    main()

I have been struggling on this for several weeks, you really save my life @tom. Thanks again!