It’s everything autograd machinery.
First of all, stop thinking about optimal directions, optimizer or any DL algorithm is agnostic to optimality.
When you call loss.backward(), each tensor in the model computes gradients following chain rule and carry gradients back to leaf tensors. For almost each torch function there exist a backward function which computes gradients when you call backward.
Each tensor points to tensors from which they are composed in order to follow chain rule. Have a look at this video in which autograd is explained.
PyTorch Autograd Explained - In-depth Tutorial