Hi Frank,
We actually had a very long discussion about this exact question and what to do.
It is also quite fun to see that jax and Tensorflow currently define two different gradients for these non-analytical functions!
You can see the whole discussion here: Follow the JAX or Tensorflow convention for meaning of grad() with complex inputs · Issue #41857 · pytorch/pytorch · GitHub 6
Also you can find in our doc a fairly extensive summary of the result of that discussion: Autograd mechanics — PyTorch master documentation 4
Happy to discuss it more if you’re interested!