jtencer:

I’m particularly interested in how to make sure that the backward pass executes in parallel.


If the two branches are on different GPU devices, then the backward pass would run then in parallel on the two different GPUs. If everything is on CPU, currently the backward pass only has single threaded execution and everything would run on a single thread on the CPU.