Do optimizer.zero_grad() before scores = model(data, feat). I mean something like this.
...
optimizer.zero_grad()
# forward
scores = model(data, feat)
loss = criterion(scores, targets)
# backward
loss.backward()
...

And tell me if your problem has been solved.