I tried something similar already, except I applied the mask to the candidate output before putting it through the loss calculation. (The target already had zero at those locations.)
It failed to improve with training, perhaps due to the non-differentiable nature of the operation. I’ll try doing it to the loss result like you are suggesting and see if that works better.
Thanks,