I am trying to incorporate the difference between model outputs for different iterations, into the loss for training the model.
Is it possible to have a computation graph that results from the application of the same model to two different inputs?
Something like:
model = nn.Sequential(nn.Linear(input_size, 100),
            nn.ReLU(),
            nn.Linear(100, out_size),
            nn.ReLU()
        )
outputs = []
for idx, (x,_) in enumerate(train_loader):
  output = model(x)
  outputs.append(outputs)
  if idx%2 == 0:
    loss = criterion(outputs[0], outputs[1])
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    outputs = []

In this case, we would be saving the output of two sequential iterations, and minimize the loss between those two outputs. My main concern is that the same tensors are included twice in the computation graph. Would this be a problem?
Thanks for the help!