Sorry, maybe I wasn’t clear enough. I didn’t mean to mislead you.  And thanks for your patience!



 ptrblck:

Here is a small dummy example using vgg16 :


grads = []
def save_grad(grad):
    grads.append(grad)

# Create model
model = models.vgg16()
model.eval()

# First approach
x = torch.randn(1, 3, 224, 224)
output = model.features(x)
output.register_hook(lambda x: save_grad(x))
output = model.avgpool(output)
output = output.view(output.size(0), -1)
output = model.classifier(output)
output.mean().backward()

# Reset
model.zero_grad()

# Second approach
output = x.clone()
for name, module in model.features._modules.items():
    output = module(output)
    if '30' in name:
        output.register_hook(lambda x: save_grad(x))

output = model.avgpool(output)
output = output.view(output.size(0), -1)
output = model.classifier(output)
output.mean().backward()

# Compare gradients
print((grads[0] == grads[1]).all())
> tensor(1, dtype=torch.uint8)

The code you provide above yield the same gradients. That is, grads[0] and grads[1] are the same. But  both grads[0] and grads[1] are different from model.features[30].weight.grad in shape. Because the definition of ConvNd’s weight is:



 AndrewSoul:

self.weight = Parameter(torch.Tensor(
out_channels, in_channels // groups, *kernel_size))


So the shape of model.features[30].weight.grad and shape of  grads[0] /grads[1] are different, I think.