I agree with you that it is a bit critical to do this by manipulating the weights directly using attention, and the network thus might not converge. But, I intend to do this using simulated annealing, hopefully it is gonna work; I will not know unless I try it.
I still do not know how to change the parameters of the convolution layers.