vainaijr:

and then compute loss using crossentropy, and I only update representaion of pixels associate with the image that has been predicted incorrectly.


That is basically the training procedure of a neural network.
Since the parameters are somehow connected to each other (e.g. via sequential conv layers), you all parameters would be updated.
How would you isolate the weights, which are only associated with the current class?