It looks like you are just creating the mask using numpy and apply it on your tensor fm.
If you don’t need gradients for the mask, this approach should generally work.
You might have to transform masks to a valid (non-differentiable) tensor before multiplying it with fm.
Have you tried your approach and noticed that it’s not working?