Solved by tom in post #2 
              
              
                The conceptually clean way to fix some part of weights is to have buffers (with self.register_buffer('weight_update_mask', the_mask) in the module initialization for the mask of what should be updated and the fixed weights and then in the forward use weight = torch.where(self.weight_update_mask, sel…