Another update. On stackoverflow I found very similar (seems to be the same) about retain_graph=True problem in DDPG: https://stackoverflow.com/questions/51349272/pytorch-backward-runtimeerror-trying-to-backward-through-the-graph-a-second. There was said that error might be caused by input tensors. But I checked (in debug mode) and none of (obs, actions, rewards, etc) has any gradients…
[Update]
This is exactly what albanD comment above is saying.
But I checked that each of the Tensors observation/desired_goal/etc has requires_grad=False.