Hi,

Has requires_grad=True , then gradients will be tracked, correct?

Basically when you set requires_grad=True on a Tensor, you ask for gradients for that Tensor (assuming a leaf Tensor here).
So if you set requires_grad on the input, it will compute the gradient wrt to the input.
If you set it on the parameters, it will compute the gradients for the parameters.
And if you set both, it will compute both.
So you should only set it for the Tensor for which you actually want gradients. Otherwise you might increase memory/runtime for no reason.

Also, does model.train() set requires_grad=True for all the parameters?

.train() changes the behavior of the nn.Module. requires_grad informs the autograd whether gradient should be computed for this Tensor or not.
Same for eval.
Train and eval change the behavior of nn.Modules for training or evaluation. For example dropout is deactivated during evaluation. And batchnorm uses saved statistics during training instead of the ones computed on the current batch.

when exactly did gradients start to be tracked for the parameters?

nn.Parameter have require_grad=True by default. So when you create an nn.Parameter, it is always a leaf Tensor that requires grad (unless you pass requires_grad=False to the constructor).