shogan50:

The distribution is driven by the line self.std = nn.Parameter(torch.zeros(action_dim)) where the softplus function is applied to this to get a distribution. Isn’t this just a bunch of zeros that never change?


In this code, self.std should be named self.logit_std. That’s why it is given to be eaten by a softplus thing.
Only the output of F.softplus(std) is the standard deviation.
The fact it’s a parameter (nn.Parameter) says to pytorch that it must be learned (so this value is not fixed and will adapt in order to maximize the reward, minus a lagrangian penalty for entropy).
So the entropy (that as you said, closely depends on this variable, but also on the means) will be learned, and using such a trick is quite efficient (rather than the second code where the standard deviation depends on the input). The standard deviation is, if you want, the tradeoff between exploration and exploitation: the smaller the std, the less exploration.
And the entropy penalty in the loss is used to regulate the learning rate : the main role is to avoid fast convergence to local minima with deterministic solutions.