Do you know if that is the common way to do it (i.e. do you think Tensorflow, MxNet etc have the same approach or is it PyTorch specific.

This paper 11 may provide further information for you.

Or is it more like Automatic differentiation using dual numbers (https://en.wikipedia.org/wiki/Automatic_differentiation  10)?

Yes, AD (Automatic differentiation) first appeared in Fortran project 4. Fact from the paper I sent you the link.
Fairly the most outstanding articles explaining AD I found is this one 5 from 2013.