When you do the forward pass, all the gradients are computed. Not only the gradients for the model weights (which you will not need if you freeze de model), but all the gradients of the tensors that have been involved in computing the loss function (including the input image). So what you have to do is to declare an optimizer (Such as adam, SDG,…) , passing it the input image instead of the model’s weights as parameters to optimize.
Then when you do ‘optimizer.step()’, you are optimizing the input image to a version that minimizes the loss function you defined.
However, I still didn’t understand what you meant by “How do I train the model to do this task such that during training time, the image is not altered?”. The way I explained it, the input image is going to be altered. Isn’t that the whole point of this kind of training?
Sorry for the delay in the answer.