If you look at what matmul does (it’s in C++ but you can directly transpose it into Python) 53 you see that there are a number of reshaping / broadcasting ops involved. As matmul does not have a custom derivative (you can see this in tools/autograd/derivatives.yaml), the backward is done by keeping track of the operations performed and the inputs required for the backward. Quite likely, some of the intermediate results are cached for the backward.
If it’s excessive, you could file a bug to implement an explicit backward for matmul.
An alternative could be to try einsum and see if it is better about it (but I’m not sure it is).
Best regards
Thomas