You are creating the loss in each inner loop, but you are also using a single forward pass in the outer loop in outputs_batch = model(inputs_batch).
The forward pass will create the computation graph with all intermediate activations, which are needed for the gradient calculation in the backward pass.
The first call to loss.backward() would compute these gradients are delete all intermediate activations, which explains the error in the next iteration of the inner loop.
Given that you are able to perform the forward pass with the complete batch I’m unsure which part of your code would use too much memory so that you need to accumulate the gradients (is it the interpolate method?).
Anyway, you would have to use loss.backward(retain_graph=True) or use multiple forward passes (in the inner loop).