Gradients should be tracked for the parameters of the model such as weights and biases. The training step is supposed to learn these gradients so that it can adjust these parameters in the general direction (which is the negation of the gradient) that will cause the loss to decrease.
The input is not something that the training is supposed to improve or change in any manner, at least for the models that I have come across so far. So why should the training keep track of gradients for the inputs?