It seems that gradient computation in case of AdaptiveAvgPool2d is also broken
My gradients were different even after disabling all flags for randomization:
np.random.seed(123)
torch.manual_seed(123)
torch.cuda.manual_seed_all(123)
torch.cuda.manual_seed(123)
torch.backends.cudnn.deterministic=True
torch.backends.cudnn.benchmark=False

Hope this helps anyone struggling with the same bug  .