cijerezg:

But when I train, the weights are not changing even though the loss is not 0, that is, there is some issue when the backpropagatiion is carried out.


Could you print out some .grad attributes of some parameters of both models after the loss.backward() call?
If they are None, you might be breaking the computation graph somewhere in your forward passes.



 cijerezg:

Because I am not sure, when I add them up, as in the first block, the optimizer modifies weights independently (I assume it does). I emphasize this because the output of the critic depends on the actor.


I don’t understand this section.
Both approaches (loss.backward() and calling backward of both outputs) should create the same gradients. You could check it again using some dummy data and comparing the .grad attributes.
The difference would be that optimizer.step() is called twice, which would make a difference, e.g. if you are using momentum.