Hi,
The autograd engine has been designed assuming that each task to be done in the backward is more expensive than running the engine itself. Note that this is true for most current cnn applications.
With that in mind, only one thread is needed to drive the computations. Heavy operations like mm or convs can still use multithreading for each op if needed for best performance.