What’s wrong with initilizing the optimizer inside training loop?
In the end if you want to optimize the input you will need to initilize an optimizer per input I would say.
Once you have optimized the input, old optimizer is no longer useful.
If you want to optimize both, model and init tensor, you may think about having 2 optimizers, one for the model and another one for each tensor.