Hello. I am trying to do a bit of model surgery to add a GAP layer in a VGG16 net, just before the classifier, after the conv layers. I am replacing the AdaptiveAvgPool2d((7, 7)) normally saved in network.avgpool.
Basically I am doing this:
network = torchvision.models.vgg16(pretrained=False, progress=True)
network.avgpool = torch.nn.AdaptiveAvgPool2d((1, 1))
network.classifier[0] = torch.nn.Linear(512, 4096)

the problem is that this network takes ages to learn, and the testing accuracy is not even that great. I am comparing it with supposedly the same implementation in keras, training it on the same dataset. In keras it trains withtout problem and reaches an accuracy of ~100%, in PyTorch it takes ages, and the accuracy is ~70%, so something is obviously not the same. After having spent long time checking that everything is exactly the same, I am now wondering if my way of adding a GAP layer is wrong. Maybe doing it in that way is breaking some connection in the autograd?
Any help is deeply appreciated, I am totally stuck with this.