Hi,
I wonder if anyone has got A3C working with continuous actions? I guessed it would be a good idea to ask first before trying to do it, as there’s probably a good reason why no one’s got it to work yet?
I’m working on modifying, @Ilya_Kostrikov, implementation,

  
      
      GitHub
  
  
    

ikostrikov/pytorch-a3c 204

PyTorch implementation of Asynchronous Advantage Actor Critic (A3C) from "Asynchronous Methods for Deep Reinforcement Learning". - ikostrikov/pytorch-a3c


  
  
    
    
  
  


So for example Open AIs pendulum, as only got a state/observation vector of 3, so there’s no need for any conv’s in the Actor-Critic module, basically I’m trying,
lstm_out = 256
enc_in = 3 # for pendulum
enc_hidden = 200
enc_out = lstm_out

class ActorCritic(nn.Module):

    def __init__(self , lstm_in ):
        super(ActorCritic, self).__init__( )  
        self.fc_enc_in  = nn.Linear(enc_in,enc_hidden) # enc_input_layer
        self.fc_enc_out  = nn.Linear(enc_hidden,enc_out) # enc_output_layer             
        self.lstm = nn.LSTMCell(lstm_in, lstm_out)
        self.actor_mu = nn.Linear(lstm_out, 1)
        self.actor_sigma = nn.Linear(lstm_out, 1)
        self.critic_linear = nn.Linear(lstm_out, 1)
        self.train()

    def forward(self, inputs):
        
        x, (hx, cx) = inputs

        x = F.relu(self.fc_enc_in(x))
        x = self.fc_enc_out(x)

        hx, cx = self.lstm(x, (hx, cx))
        x = hx

        return self.critic_linear(x), self.actor_mu(x), self.actor_sigma(x), (hx, cx)

The initialisation code in main.py, then looks like,
env = gym.envs.make("Pendulum-v0")
lstm_in = 3    
global_model = ActorCritic( lstm_in )
global_model.share_memory()
local_model = ActorCritic( lstm_in )

And the training code is where I get confused (as usual) ???,
env = gym.envs.make("Pendulum-v0")
s0 = env.reset()
done = True
state = torch.from_numpy(s0).float().unsqueeze(0) 
value, mu, sigma, (hx, cx) = local_model((Variable(state), (hx, cx)))

#mu = mu.clamp(-1, 1) # constain to sensible values 
Softplus=nn.Softplus()     
sigma = Softplus(sigma + 1e-5) # constrain to sensible values
normal_dist = torch.normal(mu, sigma) 

prob = normal_dist
log_prob = torch.log(prob)
entropy = 0.5 * (torch.log(2. * np.pi * sigma ) + 1.)

##--------------------------------------------------------------
# TODO Calculate the Gaussian neg log-likelihood, log(1/sqrt(2sigma^2pi)) - (x - mu)^2/(2*sigma^2)
# See - https://www.statlect.com/fundamentals-of-statistics/normal-distribution-maximum-likelihood
#
log_prob = torch.log(torch.pow( torch.sqrt(2. * sigma * np.pi) , -1)) - (normal_dist - mu)*(normal_dist - mu)*torch.pow((2. * sigma), -1)
##--------------------------------------------------------------

action = Variable( prob.data )

#action=[0,]
state, reward, done, _ = env.step([action.data[0][0]])


References,
Reference - Deepmind A3C’s paper, https://arxiv.org/pdf/1602.01783.pdf 94
Section 9 - Continuous Action Control Using the MuJoCo Physics Simulator
Here’s a diagram of the algorithm, from  https://github.com/deeplearninc/relaax#distributed-a3c-architecture-with-continuous-actions 150
image.png1547×1120 89.1 KB