I can only respond from the PyTorch perspective, but here you would make the original tensors (the ones with requires_grad=True) to be the parameters of the optimization. In the end, operations like y[0, 1] += x create a new node in the computation graph, with inputs x and y, where x is variable and y is constant. Only after that node, the resulting tensor y requires grad as well, but there is no such notion for the autograd engine, especially when it comes to the fact that only one element of that tensor should be variable. I suspect the situation is the same for tensorflow.