How is binary mask matrix not efficient? That should only be the case with 1)no gpu 2)huge weight matrix (megabytes) 3)big sparsity, like 90%. If your scenario is similar to that, you can try torch.sparse.mm.