I don’t know if there any “not wanted secondary effect” of this, but maybe you can multiply the grad variable by zero in those part of the variable that you don’t want to modify. Just after the backward part, and before the optimizer step