Hi,
It’s the same for the backward method, it uses cuda calls that are asynchronous, so it will run in parallel if you GPU allows it.
On a single gpu, whenever you ask for something to be done on the gpu, it is added to a stack and returns. So if you do a lot of operations, the only thing that you do is add all of them to the stack. What the gpu is doing, is as soon as it has free compute, it takes the next item in the stack and execute it. So if your gpu has spare compute, it will always run as many stuff at the same time as possible.