First I thought, there is no update after the loss calculation and update step but the follwing code gives me false which prove there is an update:
before = list(model.parameters())[0].clone()
loss.backward()
optimizer.step()
after = list(model.parameters())[0].clone()
logging.info(torch.equal(before.data, after.data))

Then I wanted to see the magnitude of chage by the following code:
list(model.parameters())[0].grad
Which give me very small amaount of change:
tensor([[ 3.2294e-07,  7.3983e-06,  6.5637e-06,  ..., -1.3529e-06,
2019-11-16T17:21:03.747068480Z          -4.9979e-06,  4.9799e-06],
2019-11-16T17:21:03.747074966Z         [ 5.5463e-08,  3.0771e-06, -9.5087e-07,  ...,  1.9265e-06,
2019-11-16T17:21:03.747080687Z          -4.5251e-06, -7.1564e-07].....]

This is part of the output which shows how small they are.
Here is my model :
self.lstm_hidden_dim = lstm_hidden_dim
self.lstm = nn.LSTM(word_embedding_dim,lstm_hidden_dim, num_layers=1,batch_first=False,bidirectional=True)
self.firstHidden = nn.Linear(lstm_hidden_dim*2, 300)
self.relu=nn.ReLU()
self.secondlinear=nn.Linear(300, score_space_size)
self.softmax= nn.LogSoftmax(dim=1)

Loss function and optimizer:
loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

Any idea, why the grad is very small?