Good news I found a mistake! I have to zero the gradient of z before doing backward on x. Then the grad is the one I computed by hand and the one in jax. Does anyone have an idea how to make the grad computation  more efficient?
Best
Stefan