turns out “it’s because ReLU used in torchvision’s resnet50 is an inplace operation, given that the operation += just before the ReLU (exists in both BatchNorm and Linear) is also inplace, autograd engine does some extra checks and operation reordering during backward prop.”
I still don’t have a clue how the reordering is done, but I tried to change the ReLU in my toy model above to inplace version, and found that the order of ReLU and BatchNorm was indeed reversed during backward prop. So I guess it confirms the explanation.