If you don’t want gradients to flow back as before, you can use .detach().
Otherwise you want to make sure that your observation/desired_goal/achieved_goal are Tensors that do not depend on the previous iteration (in a differentiable way) and use .detach() where needed to break these links if they currently exist.