torch.ne has gradient zero almost everywhere and gradient undefined when x == 0. In practice, if x == 0 pytorch returns 0 as gradient of torch.ne. Therefore detaching x_mask is not useful.
x * x_mask is basically an identity mapping for some elements of x in which case the gradients flow through unmodified, or a zero mapping in which case the gradients are blocked.
I can’t see why you might get NaN with the mask and not without. Could you post some more code?