<p>I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow.</p>
<p>I'm following this pseudo-code taken from the book <em>Reinforcement Learning An Introduction by Richard S. Sutton and Andrew G. Barto.</em></p>
<p><a href="https://i.sstatic.net/Ysx0Y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ysx0Y.png" alt="enter image description here" /></a></p>
<p>in summary, my question can be reduced to the following:</p>
<ul>
<li>Is it a good idea to implement a one-step actor-critic algorithm to solve the OpenAI BipedalWalker-v2 problem? If not what would be a good approach? If yes; how long would it take to converge?</li>
<li>I run the algorithm for 20000 episodes, each episode has an avg of 400 steps, for each step, I immediately update the weights. The results are not better than random. I have tried different standard deviations (for my normal distribution that represents pi), different NN sizes for the Critic and Actor, and different learning-steps for the optimizer algorithm. The results never improve. I don't know what I'm doing wrong.</li>
</ul>
<h1>My Agent Class</h1>
<pre><code>import tensorflow as tf
import numpy as np
import gym
import matplotlib.pyplot as plt

class agent_episodic_continuous_action():
    def __init__(self, lr,gamma,sample_variance, s_size,a_size,dist_type):
       ... #agent parameters

    def save_model(self,path,sess):    
    def load_model(self,path,sess):       
    def weights_init_actor(self,hidd_layer,mean,stddev): #to have control over the weights initialization      
    def weights_init_critic(self,hidd_layer,mean,stddev):  #to have control over the weights initialization            
    def create_actor_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev):  #actor is represented by a fully connected NN      
    def create_critic_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev): #critic is represented by a fully connected NN      
    def critic(self):            
    def get_delta(self,sess):                 
    def normal_dist_prob(self): #Actor pi distribution is a normal distribution whose mean comes from the NN 
    def create_actor_loss(self): 
    def create_critic_loss(self):
    def sample_action(self,sess,state): #Sample actions from the normal dist. Whose mean was aprox. By the NN
    def calculate_actor_loss_gradient(self):
    def calculate_critic_loss_gradient(self):   
    def update_actor_weights(self):
    def update_critic_weights(self):
    def update_I(self):  
    def reset_I(self):      
    def update_time_step_info(self,s,a,r,s1,d):  
    def create_graph_connections(self):
    def bound_actions(self,sess,state,lower_limit,uper_limit):  
</code></pre>
<h1>Agent instantiation</h1>
<pre><code>tf.reset_default_graph()
agent= agent_episodic_continuous_action(learning-step=1e-3,gamma=0.99,pi_stddev=0.02,s_size=24,a_size=4,dist_type=&quot;normal&quot;)
agent.create_actor_brain(hidden_layers=[12,5],hidden_layers_fct=&quot;relu&quot;,output_layer=&quot;linear&quot;,mean=0.0,stddev=0.14)
agent.create_critic_brain(hidden_layers=[12,5],hidden_layers_fct=&quot;relu&quot;,output_layer=&quot;linear&quot;,mean=0.0,stddev=0.14)
agent.create_graph_connections()

path = &quot;/home/diego/Desktop/Study/RL/projects/models/biped/model.ckt&quot;   
env = gym.make('BipedalWalker-v2')
uper_action_limit = env.action_space.high
lower_action_limit = env.action_space.low   
total_returns=[]
</code></pre>
<h1>Training loops</h1>
<pre><code>with tf.Session() as sess:
    try:
        sess.run(agent.init)
        sess.graph.finalize()
        #agent.load_model(path,sess)        
        for i in range(1000): 
            agent.reset_I()
            s = env.reset()    
            d = False
            while (not d):
                a=agent.bound_actions(sess,s,lower_action_limit,uper_action_limit)  
                s1,r,d,_ = env.step(a)
                #env.render()
                agent.update_time_step_info([s],[a],[r],[s1],d)                 
                agent.get_delta(sess)
                sess.run([agent.update_critic_weights,agent.update_actor_weights],feed_dict={agent.state_in:agent.time_step_info['s']})
                agent.update_I()  
                s = s1
        agent.save_model(path,sess)    
    except Exception as e:
        print(e)
</code></pre>

CC BY-SA 4.0
2018-09-10T15:04:46.270Z
I think this could be a good, on-topic question, but the inclusion of **all** the code is a distraction. I suggest give the high level hyperparameters, and just link the full code it in case someone feels like replicating your experiment. There *might* be a bug or misunderstanding in your implementation, but realistically no-one here is going to read through 100s of lines of your project code to help find it. However, you could still get an answer to the core question: How long should this experiment take to succeed, and is the specific agent type you have chosen suitable?
Neil Slater
CC BY-SA 4.0
2018-09-10T15:06:20.330Z
@NeilSlater Thank you, I will Edit the question following your suggestion.
Diego Orellana
CC BY-SA 4.0
2018-09-10T19:59:08.927Z
I would expect the actor to have 8 outputs representing $(\mu_0, \sigma_0, \mu_1, \sigma_1, \mu_2, \sigma_2, \mu_3, \sigma_3)$ for the 4 action dimensions, and the critic to have 1 output representing state value. Not clear if that is what you have implemented here (it would also be valid to have 5 outputs in the actor, sharing the std dev, or some other arrangement - looking at part of the method it seems you might be using a fixed std dev throughout, which may be valid theoretically but would need careful tuning). Can you explain the policy representation that your actor network is using?
Neil Slater
CC BY-SA 4.0
2018-09-10T20:12:30.273Z
It's very hard to find a basic "vanilla" Actor-Critic implementation, almost all the available sources are A3C implementations, which would not be a fair comparison. That makes this question hard to answer simply (by loading up Gym and trying some hyperparams with someone else's tested agent). I am looking into it though . . .
Neil Slater
CC BY-SA 4.0
2018-09-10T21:34:03.723Z
@NeilSlater Thank you for your interest. I think I'm close to solve it. What I'm doing is solving the cart-pole problem using the same code. I had a bug in the gradient calculation. I will post the code once is correct maybe somebody finds it useful.
Diego Orellana
CC BY-SA 4.0
2018-09-11T00:04:09.497Z
@NeilSlater Yes I was using a fixed std. The policy I was using is represented by a multivariate normal distribution in which the mean is calculated by a fully connected NN and the stddev is fixed (diagonal identity matrix multiplied by a constant) I will implement   (μ0,σ0,μ1,σ1,μ2,σ2,μ3,σ3) as you suggested.
Diego Orellana
<reinforcement-learning><tensorflow><python><open-ai>
How many episodes does it take for a vanilla one-step actor-critic agent to master the OpenAI BipedalWalker-v2 problem?
Diego Orellana
# How many episodes does it take for a vanilla one-step actor-critic agent to master the OpenAI BipedalWalker-v2 problem?

(Asked by: username_0 on 2018-09-05)

I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow.
I'm following this pseudo-code taken from the book Reinforcement Learning An Introduction by Richard S. Sutton and Andrew G. Barto.

in summary, my question can be reduced to the following:

Is it a good idea to implement a one-step actor-critic algorithm to solve the OpenAI BipedalWalker-v2 problem? If not what would be a good approach? If yes; how long would it take to converge?
I run the algorithm for 20000 episodes, each episode has an avg of 400 steps, for each step, I immediately update the weights. The results are not better than random. I have tried different standard deviations (for my normal distribution that represents pi), different NN sizes for the Critic and Actor, and different learning-steps for the optimizer algorithm. The results never improve. I don't know what I'm doing wrong.

My Agent Class
import tensorflow as tf
import numpy as np
import gym
import matplotlib.pyplot as plt

class agent_episodic_continuous_action():
    def __init__(self, lr,gamma,sample_variance, s_size,a_size,dist_type):
       ... #agent parameters

    def save_model(self,path,sess):    
    def load_model(self,path,sess):       
    def weights_init_actor(self,hidd_layer,mean,stddev): #to have control over the weights initialization      
    def weights_init_critic(self,hidd_layer,mean,stddev):  #to have control over the weights initialization            
    def create_actor_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev):  #actor is represented by a fully connected NN      
    def create_critic_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev): #critic is represented by a fully connected NN      
    def critic(self):            
    def get_delta(self,sess):                 
    def normal_dist_prob(self): #Actor pi distribution is a normal distribution whose mean comes from the NN 
    def create_actor_loss(self): 
    def create_critic_loss(self):
    def sample_action(self,sess,state): #Sample actions from the normal dist. Whose mean was aprox. By the NN
    def calculate_actor_loss_gradient(self):
    def calculate_critic_loss_gradient(self):   
    def update_actor_weights(self):
    def update_critic_weights(self):
    def update_I(self):  
    def reset_I(self):      
    def update_time_step_info(self,s,a,r,s1,d):  
    def create_graph_connections(self):
    def bound_actions(self,sess,state,lower_limit,uper_limit):  

Agent instantiation
tf.reset_default_graph()
agent= agent_episodic_continuous_action(learning-step=1e-3,gamma=0.99,pi_stddev=0.02,s_size=24,a_size=4,dist_type="normal")
agent.create_actor_brain(hidden_layers=[12,5],hidden_layers_fct="relu",output_layer="linear",mean=0.0,stddev=0.14)
agent.create_critic_brain(hidden_layers=[12,5],hidden_layers_fct="relu",output_layer="linear",mean=0.0,stddev=0.14)
agent.create_graph_connections()

path = "/home/diego/Desktop/Study/RL/projects/models/biped/model.ckt"   
env = gym.make('BipedalWalker-v2')
uper_action_limit = env.action_space.high
lower_action_limit = env.action_space.low   
total_returns=[]

Training loops
with tf.Session() as sess:
    try:
        sess.run(agent.init)
        sess.graph.finalize()
        #agent.load_model(path,sess)        
        for i in range(1000): 
            agent.reset_I()
            s = env.reset()    
            d = False
            while (not d):
                a=agent.bound_actions(sess,s,lower_action_limit,uper_action_limit)  
                s1,r,d,_ = env.step(a)
                #env.render()
                agent.update_time_step_info([s],[a],[r],[s1],d)                 
                agent.get_delta(sess)
                sess.run([agent.update_critic_weights,agent.update_actor_weights],feed_dict={agent.state_in:agent.time_step_info['s']})
                agent.update_I()  
                s = s1
        agent.save_model(path,sess)    
    except Exception as e:
        print(e)

**Tags:** reinforcement-learning, tensorflow, python, open-ai

> **Comments:**
> **username_1**: I think this could be a good, on-topic question, but the inclusion of **all** the code is a distraction. I suggest give the high level hyperparameters, and just link the full code it in case someone feels like replicating your experiment. There *might* be a bug or misunderstanding in your implementation, but realistically no-one here is going to read through 100s of lines of your project code to help find it. However, you could still get an answer to the core question: How long should this experiment take to succeed, and is the specific agent type you have chosen suitable?  
> **username_0**: @NeilSlater Thank you, I will Edit the question following your suggestion.  
> **username_1**: I would expect the actor to have 8 outputs representing $(\mu_0, \sigma_0, \mu_1, \sigma_1, \mu_2, \sigma_2, \mu_3, \sigma_3)$ for the 4 action dimensions, and the critic to have 1 output representing state value. Not clear if that is what you have implemented here (it would also be valid to have 5 outputs in the actor, sharing the std dev, or some other arrangement - looking at part of the method it seems you might be using a fixed std dev throughout, which may be valid theoretically but would need careful tuning). Can you explain the policy representation that your actor network is using?  
> **username_1**: It's very hard to find a basic "vanilla" Actor-Critic implementation, almost all the available sources are A3C implementations, which would not be a fair comparison. That makes this question hard to answer simply (by loading up Gym and trying some hyperparams with someone else's tested agent). I am looking into it though . . .  
> **username_0**: @NeilSlater Thank you for your interest. I think I'm close to solve it. What I'm doing is solving the cart-pole problem using the same code. I had a bug in the gradient calculation. I will post the code once is correct maybe somebody finds it useful.  
> **username_0**: @NeilSlater Yes I was using a fixed std. The policy I was using is represented by a multivariate normal distribution in which the mean is calculated by a fully connected NN and the stddev is fixed (diagonal identity matrix multiplied by a constant) I will implement   (μ0,σ0,μ1,σ1,μ2,σ2,μ3,σ3) as you suggested.  

---

## 0 answers

