Hi, I was playing with resnet50 model pre-defined in torchvision.models, I use following code to register hooks
import torchvison.models as tmodels

def is_supported_instance(module):
    for typed_modules in get_supported_modules().values():
        if isinstance(module, tuple(typed_modules)):
            return True
    return False

def fw_hook(module, input, output):
    print(f'forward hook in module {module.__full_name__}')
    
    def bw_hook(grad):
        print(f'backward hook in module {module.__full_name__}')
        
    handle = output.register_hook(bw_hook)
    module.__bw_hook_handle__ = handle

def register_hook(module, hook_fn):
    if is_supported_instance(module):
        handle = module.register_forward_hook(hook_fn)
        module.__fw_hook_handle__ = handle
    
# tm = test_model()
tm = tmodels.resnet50()

for n, m in tm.named_modules():
    m.__full_name__ = n.replace('.', '/')
    
tm.apply(lambda m: register_hook(m, fw_hook))

input = torch.ones([2, 3, 224, 224], dtype = torch.float, requires_grad = True)
out = tm(input)
out = out.mean()
out.backward()

The result seems good in forward propagation part, however, in backward prop, I noticed that the order of ReLU and BatchNorm2d was reversed, as shown below

…
forward hook in module layer4/2/conv2
forward hook in module layer4/2/bn2
forward hook in module layer4/2/relu
forward hook in module layer4/2/conv3
forward hook in module layer4/2/bn3
forward hook in module layer4/2/relu
forward hook in module avgpool
forward hook in module fc
backward hook in module fc
backward hook in module avgpool
backward hook in module layer4/2/bn3
backward hook in module layer4/2/relu
backward hook in module layer4/2/conv3
backward hook in module layer4/2/bn2
backward hook in module layer4/2/relu
backward hook in module layer4/2/conv2
…

besides, I’ve checked the code for resnet50 in torchvision.models, it seems that ReLU always follows a BatchNorm2d layer. So why did this happen?
Thanks in advance.