I ended up using a scalar tensor with requires_grad=True and then adding it to the overall constant tensor element like this:
x = torch.tensor(1., requires_grad=True)
y = torch.zeros((2, 2))
y[0, 1] += x

Now this causes all elements of y to be constant except for the (0, 1) element. If you want it the other way around, i.e. all but one element require grad, then you just need more tensors to add. For example let’s say the (0, 1) should be constant, so we can do:
x = torch.ones((2, 1), requires_grad=True)
y = torch.tensor(1., requires_grad=True)
z = torch.zeros((2, 2))
z[:, 0] += x
z[1, 1] += y

For a 2D tensor (matrix) at most four distinct tensors with requires_grad=True should suffice to make all but a single element require grad as well.
Nevertheless I think that the method of setting the .grad attribute partially to zero works as well.