Thank you for sharing your use case!
Why do you want to implement the chain rule manually. Maybe its better to do dL3/dA directly? For example A_grad = torch.autograd.grad(L3, A).
This is possible if you just do 1f and 2g update steps manually.