You have a single autograd graph here, with pending gradient computations for all iterations, hence nothing gets released. loss.backward() inside the loop may help, but you’d have to split graph by .detach() -ing model outputs passed to a next model.