Note that there are several subtleties here.
In the first layer we have the following:

There are directly differentiable functions (per tools/autograd/derivatives.yaml 7), these are the easy ones. For those, there is a backward (somewhere).
Then there are functions that reduce to directly differentiable functions (e.g. einsum). This means that they are calling these directly differentiable functions and autograd will do its autograd thing to do the differentiation.
Some of these compositions will go through unexposed/internal functions (e.g. convolutions, ctc loss etc.).
The non-differentiable functions are the remainder…

But then there are second derivatives, where the same scheme applies, and there is the question of “differentiable w.r.t. which arguments?”.
Best regards
Thomas