It’s not a matter of using that. I don’t know the content of those matrices but I expect you understand that masking is not backpropagable, thus, learnable.
Problem there is that when you compute labels_cpu you are moving labels to gpu but it still refers to labels in memory.
You have to use
Labels.cpu().clone().detach()
So depending on what labels[picks>thresh]= self.ignore does you may not be able to fix it.
If you only want to compute loss over those features try to mask both, gt and labels, not to modify them