No, that wouldn’t work, because you’d have to backprop through the parametrization at each iteration, but it doesn’t make sense to specify retain_variables and loose all the memory savings in other parts of the graph. I’d recommend recreating the parametrized Variable from scratch at every iteration.