Hi,I am sorry I can’t understand.How to explain that BackwardCFunction implements apply function like this.

屏幕快照 2019-08-15 下午5.46.46.png1532×264 35 KB

It call backward function?I really want to know about it.Does the apply function be called in Variable._execution_engine.run_backward function?
And I have another question.I know that pytorch use DistributedDataParallel to get distributed train.I read the source code about Recucer.It has pending variable to synchronize multiple GPU cards in one node.But how to synchronize differnent nodes?For example,There are three node. The first node finish one step gradients calculation.How does the first node know whether the other nodes finish gradients calculation or not?
Thank you very much!These questions has been bothering me for days.