I take the loss values and add them together before running: backward(retain_graph=True).
For example:
for value in value_list:
     loss +=  value.lossA
for value in value_list:
     loss += value.lossB

loss.backward(retain_graph=True)

The loss variable after running backward(), will then be a float value. For the example, let’s say that value is: 8088996.0.
Now because I used retain_graph=True, I can collect the individual loss values and add them together like I did above, only this time they have already had .backward() used on them (because of the single .backward() above):
for value in value_list:
     totalLoss +=  value.lossA
for value in value_list:
     totalLoss += value.lossB

The result of adding these loss variables together, would then give me a result like: 8088995.81122.
Now I would expect for loss to be equal to totalLoss, but they are slightly different. I graphed the difference between these two total loss values over time, here: https://i.imgur.com/kiFrWVY.png 2
Here are some more examples:
  totalloss: 3512584.05008
  loss: 3512584.25

  totalloss: 1081956.37198
  loss: 1081956.375

  totalloss: 1028641.08926
  loss: 1028641.0625

Is this supposed to be expected behavior for PyTorch? Or is there some sort of rounding going on somewhere?
This occurred with both the Adam and L-BFGS optimizers. I have tested my code on PyTorch v0.3.1, and v0.4, but the issue remains.