Hello @leopardyao , @ptrblck, if this is the case how can we return the output without the Relu operation applied to a conv layer after batch norm?
Consider this piece of architecture in Resnet50 :
(2): Bottleneck(
        (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )


in here, I need to get the output after normalizing the input using batch norm but before the relu operation, so I thought I could do something like this :
model = Resnet50() 
model.load_state_dict(torch.load('path_to_model.bin'))
model.to(device) 

# now using function hook, I was able to get the output after the conv3 layer like this :
model.model.layer4[1].conv3.register_forward_hook(get_activation("some_key_name") 

now using the output vector which is stored in the activation dict, I applied the batch norm operation on it like :
model.model.layer4[1].bn3(activation['some_key_name'])

But I see that some of the outputs compared with the relu outputs excluding 0 due to negatives from the batch norm layer vary.
ex :
This is when I do BN(CN)
A = 
[-0.2502, -0.4636, -0.4095, -0.3121],
          [-0.5342, -0.4120, -0.0899, -0.0620],
          [-0.1114, -0.1497, -0.1434,  0.0654],
          [-0.1596,  0.2669,  0.4669,  0.1378]

This is gotten from directly hooking the bn3 layer :
i.e :
model.model.layer4[1].bn3.register_forward_hook(get_activation("some_key_name_2") 

B = 
[0.        , 0.        , 0.        , 0.        ],
[0.        , 0.        , 0.        , 0.89255244],
[0.        , 0.        , 0.03623176, 0.34344062],
[0.        , 0.2668974 , 0.5831858 , 0.13784815]]

it will be usually B = F.relu(A), However some of the items does not match, how can I rectify this ?
JFYI,

I have just considered a smaller section in A and B and does not represent the whole matrix

model.model as I am encapsulating the resnet torchvision model in a class.