Could you explain, what a GAP layer is?
I ran a quick profile on the original VGG16 model and using your modifications and it seems your modifications yield a small speedup (which is expected):
import torch
import torchvision.models as models
import time


model = models.vgg16().cuda()
x = torch.randn(32, 3, 224, 224, device='cuda')

nb_iters = 100

# warmup
for _ in range(10):
    out = model(x)


torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    out = model(x)
    out.backward(torch.ones_like(out))
    model.zero_grad()
torch.cuda.synchronize()
t1 = time.time()
print((t1 - t0)/nb_iters)
> 0.1372007703781128

model.avgpool = torch.nn.AdaptiveAvgPool2d((1, 1)).cuda()
model.classifier[0] = torch.nn.Linear(512, 4096).cuda()

torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    out = model(x)
    out.backward(torch.ones_like(out))
    model.zero_grad()
torch.cuda.synchronize()
t1 = time.time()
print((t1 - t0)/nb_iters)
> 0.1326659917831421

You can set torch.backends.cudnn.benchmak = True to profile the kernels and select the fastest one, but I didn’t get a major speedup for this use case.