Did that, but the value of bias remains the same. On a different note, this is the way in which I updated the parameters. Please tell me if there is an error.
model = Net().cuda()
criterion = nn.MSELoss().cuda()
optimizer = optim.SGD([{'params':[model.bias], 'lr':0.001}], lr=0.001)
input = Variable(torch.ones((1,1,2,2))).cuda()
target = Variable(torch.zeros((1,1,2,2))).cuda()
loss = criterion(model(input),target)
print loss
loss.backward()