A leaf tensor is a tensor with no parent in the graph. If it does not require grad, then it never has parents and is always a leaf, if it requires grad, it is a leaf only if it was created by the user (not the result of an op). When running the backward pass, the gradients are accumulated in all the leaf tensors that require gradients.