Thanks for your reply Simon!
I am curious how did PyTorch finish back propagation for each layer if it has no ideas of what layers are following itself, because to update weights of that layer we need to sum all gradients coming from all of its descendants?