Thanks for the reply this is my code. I use the SGD optimizer

‘’’
optimizer = optim.SGD(im_net.parameters(), lr=0.1, momentum=0.9,
weight_decay=5e-4)
weight_reference = im_net.features[25].weight[temp_4].clone()
for batch_idx, (inputs, targets) in enumerate(testloader):

    inputs = inputs.float().cuda()
    inputs, targets = inputs.to(device), targets.to(device)
        
        
    outputs = im_net(inputs)
                       
    ce_loss = criterion(outputs, targets)
    
    optimizer.zero_grad()
    loss = criterion(outputs, targets)
        
    
    #im_net.features[25].weight[temp_4].register_hook(lambda grad : grad * 0)
    loss.backward()
    im_net.features[25].weight.grad[temp_4] = 0.
    
    
    optimizer.step()
    print(weight_reference == im_net.features[25].weight[temp_4])


‘’’

By the way, could you explain about ‘optimizer with running estimates’? I don’t know what it means.