If you have your mask 1 locations as a rows and columns LongTensors, so for example (your matrices are larger, but for demonstration purposes)
A = torch.randn(5,10)
B = torch.randn(8,10)
rows = torch.arange(0, 16, out=torch.LongTensor())%5
cols = (torch.arange(0, 16, out=torch.LongTensor())/2)%8
idxes = torch.stack([rows,cols], dim=0)
mask = torch.sparse.FloatTensor(idxes, torch.ones(16)).to_dense()

how about doing
vals = (A[rows]*B[cols]).sum(1)
res = torch.sparse.FloatTensor(idxes, vals)

Then you see that the difference to the masked product is 0 up to machine precision:
print((res.to_dense()-(A@B.t()) * mask).abs().max())

Now this will have quite a bit of memory overhead, but you could split it into chunks of nonzero entries and cat them together afterwards if needed.
Best regards
Thomas