I tried to reproduce it with the following script but can’t get it work:
import torch
from torch.optim import Adam
import torch.nn as nn
from torch.autograd import Variable

net = nn.Sequential(
        nn.Linear(5, 10),
        nn.ReLU(),
        nn.Linear(10, 5),
        nn.ReLU(),
        nn.Softmax(1)
        ).cuda()

optim = Adam(net.parameters(), lr = 3)
net[2].weight.register_hook(lambda grad: grad * 0)
x = Variable(torch.cuda.FloatTensor(3, 5).normal_())
net(x).mean().backward()
old_weight = net[2].weight.data.clone()
optim.step()
new_weight = net[2].weight.data.clone()
print((old_weight - new_weight).abs_().mean())  # get 0

Do you mind sharing your script and torch version? I want to make sure that our Adam implementation doesn’t have bugs