There’s a tensor-level rule for each core op. If you did per-element differentiation, it would be hard to automatically “reroll” the result back into GPU-efficient operations. Hence when you add a core-level operation working on tensors, you also add a hand-coded derivative for that op that works on tensors (more specifically, vector-jacobian-product).
You can see the issue of doing per-element differentiation by looking at standard derivative results for matrix operations here. Derivatives of SVD, matrix inverse, and determinant have simple expressions in matrix form, but you can’t easily get this form by doing per-element differentiation.