I’m working on a delayed online learning scheme where I want to do the optimization step after certain delay, and I’m encountering the in-place operation when doing slicing. Here is a rough pseudo code of what I’m doing.
Let E be the number of episodes, T be the time horizon of each episode, and D be the delay.
input=torch.zeros(E,T)
output=torch.zeros(E,T)
output_target=torch.zeros(E,T)
for e in range(E):
   for t in range(T):
      input[e,t] = some outer environment source
      output[e,t] = model( input[e,t] )  # model is my defined neural net from nn.Module
      Act in the outer environment with output[e,t]
      output_target[e,t] = from outer environment source
   if e>=D:
      for t in range(T):
         loss += loss_function( output_target[e-D,t] , output[e-D,t] )
      optimizer.zero_grad()
      loss.backward(retain_graph=True)
      optimizer.step()

And I got the in-place operation error: “RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation:”
It seems that it’s because I keep filling the slice of output with new results, so the version of output also changes. What is the efficient way to program the thing I want to do then?
Thanks!