I think if you register hook and zero out gradients for mask, then it will work, for example,
‘’’
def hook_fn(grad):
out = grad.clone()
out[mask] = 0
return out
layer.weight.register_hook(hook_fn)
‘’’