I am trying to train a LSTM model in a NLP problem.
I want to use learning rate decay with the  torch.optim.lr_scheduler.ExponentialLR class, yet I seem to fail to use it correctly.
My code:
optimizer = torch.optim.Adam(dual_encoder.parameters(), lr = 0.001)
scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma = 0.95)
for epoch in range(200):
context_id_list, response_id_list, label_array = load_ids_and_labels(dataframe, word_to_id)

loss_acc = 0

for i in range(len(label_array)):

    context = autograd.Variable(torch.LongTensor(context_id_list[i]).view(len(context_id_list[i]),1), requires_grad = False)
    
    response = autograd.Variable(torch.LongTensor(response_id_list[i]).view(len(response_id_list[i]), 1), requires_grad = False)
    
    label = autograd.Variable(torch.FloatTensor(torch.from_numpy(np.array(label_array[i]).reshape(1,1))), requires_grad = False)
    
    score = dual_encoder(context, response)

    loss = torch.nn.functional.binary_cross_entropy_with_logits(score, label) 
    
    loss_acc += loss.data[0]
   
    loss.backward()

    scheduler.step() #?
    
    optimizer.zero_grad()
    
print("Epoch: ", epoch, ", Loss: ", (loss_acc/len(label_array)))

If I do it like this, according to one PyTorch example, the parameters do not get updated. If I use optimizer.step() instead, the scheduler will not even be applied, as far as I understand.
I would be happy about some code example!
Thanks!