I know we can write loss.backward() to finish grad calculation.But when I read source code about backward,I am confused.
I add some print information in pytorch python source code to trace the backward process.
Basically loss.backward() will call torch.autograd.init.py backward function and get into Variable._execution_engine.run_backward function.This is a cpp function.When we use DataParallel,the source code finally will call torch.nn.parallel.Gather.backward function and torch.nn.parallel.Broadcast.backward function.How are these two class called?
In addition,Gather.backward funtion will call Scatter.apply function, and Scatter.apply function will call Scatter.forward function.How does it woks?
The Scatter class inherits from torch.autograd.Function class.Function class inherits from a python metaclass.The metaclass bulid a class that the class inherits from BackwardCFunction class.
So in my opinion,if we call Scatter.apply function,the Scatter.backward function will be called.But why not?

屏幕快照 2019-08-15 下午2.28.41.png1562×284 35.4 KB