Solved by ptrblck in post #4 
              
              
                The gradients will be zero at the masked points: 
x = torch.randn(1, 10)
weight = nn.Parameter(torch.randn(5, 10))
adj = torch.randint(0, 2, (5, 10)).float()

out = torch.matmul(x, (adj * weight).t())
out.mean().backward()

print(weight.grad)
> tensor([[ 0.1776, -0.1401,  0.0000, -0.0000,  0.0000,  …