The loss depends on the criterion you are using and a certain discrepancy between the loss value and accuracy might come from the difference in the logits.
Have a look at this dummy example:
x1 = torch.tensor([[-0.1, 0.1, -0.1, -0.1]])
x2 = torch.tensor([[-1., 1., -1., -1.]])
target = torch.tensor([1])
criterion = nn.CrossEntropyLoss()
loss1 = criterion(x1, target)
loss2 = criterion(x2, target)

While both tensors predict the target and achieve a 100% accuracy, the loss is different, since x2 passes a higher probability to class1 than x1.
In your case, however, I guess the effect is too strong and constant. Are you using any regularization, as it looks like you are adding e.g. weight decay to the loss value.