Body: <p>I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow.</p>
  <p>I'm following this pseudo-code taken from the book <em>Reinforcement Learning An Introduction by Richard S. Sutton and Andrew G. Barto.</em></p>
  <p><a href="https://i.sstatic.net/Ysx0Y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ysx0Y.png" alt="enter image description here" /></a></p>
  <p>in summary, my question can be reduced to the following:</p>
  <ul>
  <li>Is it a good idea to implement a one-step actor-critic algorithm to solve the OpenAI BipedalWalker-v2 problem? If not what would be a good approach? If yes; how long would it take to converge?</li>
  <li>I run the algorithm for 20000 episodes, each episode has an avg of 400 steps, for each step, I immediately update the weights. The results are not better than random. I have tried different standard deviations (for my normal distribution that represents pi), different NN sizes for the Critic and Actor, and different learning-steps for the optimizer algorithm. The results never improve. I don't know what I'm doing wrong.</li>
  </ul>
  <h1>My Agent Class</h1>
  <pre><code>import tensorflow as tf
  import numpy as np
  import gym
  import matplotlib.pyplot as plt
  
  class agent_episodic_continuous_action():
      def __init__(self, lr,gamma,sample_variance, s_size,a_size,dist_type):
         ... #agent parameters
  
      def save_model(self,path,sess):    
      def load_model(self,path,sess):       
      def weights_init_actor(self,hidd_layer,mean,stddev): #to have control over the weights initialization      
      def weights_init_critic(self,hidd_layer,mean,stddev):  #to have control over the weights initialization            
      def create_actor_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev):  #actor is represented by a fully connected NN      
      def create_critic_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev): #critic is represented by a fully connected NN      
      def critic(self):            
      def get_delta(self,sess):                 
      def normal_dist_prob(self): #Actor pi distribution is a normal distribution whose mean comes from the NN 
      def create_actor_loss(self): 
      def create_critic_loss(self):
      def sample_action(self,sess,state): #Sample actions from the normal dist. Whose mean was aprox. By the NN
      def calculate_actor_loss_gradient(self):
      def calculate_critic_loss_gradient(self):   
      def update_actor_weights(self):
      def update_critic_weights(self):
      def update_I(self):  
      def reset_I(self):      
      def update_time_step_info(self,s,a,r,s1,d):  
      def create_graph_connections(self):
      def bound_actions(self,sess,state,lower_limit,uper_limit):  
  </code></pre>
  <h1>Agent instantiation</h1>
  <pre><code>tf.reset_default_graph()
  agent= agent_episodic_continuous_action(learning-step=1e-3,gamma=0.99,pi_stddev=0.02,s_size=24,a_size=4,dist_type=&quot;normal&quot;)
  agent.create_actor_brain(hidden_layers=[12,5],hidden_layers_fct=&quot;relu&quot;,output_layer=&quot;linear&quot;,mean=0.0,stddev=0.14)
  agent.create_critic_brain(hidden_layers=[12,5],hidden_layers_fct=&quot;relu&quot;,output_layer=&quot;linear&quot;,mean=0.0,stddev=0.14)
  agent.create_graph_connections()
  
  path = &quot;/home/diego/Desktop/Study/RL/projects/models/biped/model.ckt&quot;   
  env = gym.make('BipedalWalker-v2')
  uper_action_limit = env.action_space.high
  lower_action_limit = env.action_space.low   
  total_returns=[]
  </code></pre>
  <h1>Training loops</h1>
  <pre><code>with tf.Session() as sess:
      try:
          sess.run(agent.init)
          sess.graph.finalize()
          #agent.load_model(path,sess)        
          for i in range(1000): 
              agent.reset_I()
              s = env.reset()    
              d = False
              while (not d):
                  a=agent.bound_actions(sess,s,lower_action_limit,uper_action_limit)  
                  s1,r,d,_ = env.step(a)
                  #env.render()
                  agent.update_time_step_info([s],[a],[r],[s1],d)                 
                  agent.get_delta(sess)
                  sess.run([agent.update_critic_weights,agent.update_actor_weights],feed_dict={agent.state_in:agent.time_step_info['s']})
                  agent.update_I()  
                  s = s1
          agent.save_model(path,sess)    
      except Exception as e:
          print(e)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-10T15:04:46.270Z
Comments.Text: I think this could be a good, on-topic question, but the inclusion of **all** the code is a distraction. I suggest give the high level hyperparameters, and just link the full code it in case someone feels like replicating your experiment. There *might* be a bug or misunderstanding in your implementation, but realistically no-one here is going to read through 100s of lines of your project code to help find it. However, you could still get an answer to the core question: How long should this experiment take to succeed, and is the specific agent type you have chosen suitable?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-10T15:06:20.330Z
Comments.Text: @NeilSlater Thank you, I will Edit the question following your suggestion.
Comments.UserDisplayName: Diego Orellana
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-10T19:59:08.927Z
Comments.Text: I would expect the actor to have 8 outputs representing $(\mu_0, \sigma_0, \mu_1, \sigma_1, \mu_2, \sigma_2, \mu_3, \sigma_3)$ for the 4 action dimensions, and the critic to have 1 output representing state value. Not clear if that is what you have implemented here (it would also be valid to have 5 outputs in the actor, sharing the std dev, or some other arrangement - looking at part of the method it seems you might be using a fixed std dev throughout, which may be valid theoretically but would need careful tuning). Can you explain the policy representation that your actor network is using?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-10T20:12:30.273Z
Comments.Text: It's very hard to find a basic "vanilla" Actor-Critic implementation, almost all the available sources are A3C implementations, which would not be a fair comparison. That makes this question hard to answer simply (by loading up Gym and trying some hyperparams with someone else's tested agent). I am looking into it though . . .
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-10T21:34:03.723Z
Comments.Text: @NeilSlater Thank you for your interest. I think I'm close to solve it. What I'm doing is solving the cart-pole problem using the same code. I had a bug in the gradient calculation. I will post the code once is correct maybe somebody finds it useful.
Comments.UserDisplayName: Diego Orellana
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-11T00:04:09.497Z
Comments.Text: @NeilSlater Yes I was using a fixed std. The policy I was using is represented by a multivariate normal distribution in which the mean is calculated by a fully connected NN and the stddev is fixed (diagonal identity matrix multiplied by a constant) I will implement   (μ0,σ0,μ1,σ1,μ2,σ2,μ3,σ3) as you suggested.
Comments.UserDisplayName: Diego Orellana
Tags: <reinforcement-learning><tensorflow><python><open-ai>
Title: How many episodes does it take for a vanilla one-step actor-critic agent to master the OpenAI BipedalWalker-v2 problem?
UserDisplayName: Diego Orellana
ThreadText: # How many episodes does it take for a vanilla one-step actor-critic agent to master the OpenAI BipedalWalker-v2 problem?
  
  (Asked by: username_0 on 2018-09-05)
  
  I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow.
  I'm following this pseudo-code taken from the book Reinforcement Learning An Introduction by Richard S. Sutton and Andrew G. Barto.
  
  in summary, my question can be reduced to the following:
  
  Is it a good idea to implement a one-step actor-critic algorithm to solve the OpenAI BipedalWalker-v2 problem? If not what would be a good approach? If yes; how long would it take to converge?
  I run the algorithm for 20000 episodes, each episode has an avg of 400 steps, for each step, I immediately update the weights. The results are not better than random. I have tried different standard deviations (for my normal distribution that represents pi), different NN sizes for the Critic and Actor, and different learning-steps for the optimizer algorithm. The results never improve. I don't know what I'm doing wrong.
  
  My Agent Class
  import tensorflow as tf
  import numpy as np
  import gym
  import matplotlib.pyplot as plt
  
  class agent_episodic_continuous_action():
      def __init__(self, lr,gamma,sample_variance, s_size,a_size,dist_type):
         ... #agent parameters
  
      def save_model(self,path,sess):    
      def load_model(self,path,sess):       
      def weights_init_actor(self,hidd_layer,mean,stddev): #to have control over the weights initialization      
      def weights_init_critic(self,hidd_layer,mean,stddev):  #to have control over the weights initialization            
      def create_actor_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev):  #actor is represented by a fully connected NN      
      def create_critic_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev): #critic is represented by a fully connected NN      
      def critic(self):            
      def get_delta(self,sess):                 
      def normal_dist_prob(self): #Actor pi distribution is a normal distribution whose mean comes from the NN 
      def create_actor_loss(self): 
      def create_critic_loss(self):
      def sample_action(self,sess,state): #Sample actions from the normal dist. Whose mean was aprox. By the NN
      def calculate_actor_loss_gradient(self):
      def calculate_critic_loss_gradient(self):   
      def update_actor_weights(self):
      def update_critic_weights(self):
      def update_I(self):  
      def reset_I(self):      
      def update_time_step_info(self,s,a,r,s1,d):  
      def create_graph_connections(self):
      def bound_actions(self,sess,state,lower_limit,uper_limit):  
  
  Agent instantiation
  tf.reset_default_graph()
  agent= agent_episodic_continuous_action(learning-step=1e-3,gamma=0.99,pi_stddev=0.02,s_size=24,a_size=4,dist_type="normal")
  agent.create_actor_brain(hidden_layers=[12,5],hidden_layers_fct="relu",output_layer="linear",mean=0.0,stddev=0.14)
  agent.create_critic_brain(hidden_layers=[12,5],hidden_layers_fct="relu",output_layer="linear",mean=0.0,stddev=0.14)
  agent.create_graph_connections()
  
  path = "/home/diego/Desktop/Study/RL/projects/models/biped/model.ckt"   
  env = gym.make('BipedalWalker-v2')
  uper_action_limit = env.action_space.high
  lower_action_limit = env.action_space.low   
  total_returns=[]
  
  Training loops
  with tf.Session() as sess:
      try:
          sess.run(agent.init)
          sess.graph.finalize()
          #agent.load_model(path,sess)        
          for i in range(1000): 
              agent.reset_I()
              s = env.reset()    
              d = False
              while (not d):
                  a=agent.bound_actions(sess,s,lower_action_limit,uper_action_limit)  
                  s1,r,d,_ = env.step(a)
                  #env.render()
                  agent.update_time_step_info([s],[a],[r],[s1],d)                 
                  agent.get_delta(sess)
                  sess.run([agent.update_critic_weights,agent.update_actor_weights],feed_dict={agent.state_in:agent.time_step_info['s']})
                  agent.update_I()  
                  s = s1
          agent.save_model(path,sess)    
      except Exception as e:
          print(e)
  
  **Tags:** reinforcement-learning, tensorflow, python, open-ai
  > **Comments:**
  > **username_1**: I think this could be a good, on-topic question, but the inclusion of **all** the code is a distraction. I suggest give the high level hyperparameters, and just link the full code it in case someone feels like replicating your experiment. There *might* be a bug or misunderstanding in your implementation, but realistically no-one here is going to read through 100s of lines of your project code to help find it. However, you could still get an answer to the core question: How long should this experiment take to succeed, and is the specific agent type you have chosen suitable?  
  > **username_0**: @NeilSlater Thank you, I will Edit the question following your suggestion.  
  > **username_1**: I would expect the actor to have 8 outputs representing $(\mu_0, \sigma_0, \mu_1, \sigma_1, \mu_2, \sigma_2, \mu_3, \sigma_3)$ for the 4 action dimensions, and the critic to have 1 output representing state value. Not clear if that is what you have implemented here (it would also be valid to have 5 outputs in the actor, sharing the std dev, or some other arrangement - looking at part of the method it seems you might be using a fixed std dev throughout, which may be valid theoretically but would need careful tuning). Can you explain the policy representation that your actor network is using?  
  > **username_1**: It's very hard to find a basic "vanilla" Actor-Critic implementation, almost all the available sources are A3C implementations, which would not be a fair comparison. That makes this question hard to answer simply (by loading up Gym and trying some hyperparams with someone else's tested agent). I am looking into it though . . .  
  > **username_0**: @NeilSlater Thank you for your interest. I think I'm close to solve it. What I'm doing is solving the cart-pole problem using the same code. I had a bug in the gradient calculation. I will post the code once is correct maybe somebody finds it useful.  
  > **username_0**: @NeilSlater Yes I was using a fixed std. The policy I was using is represented by a multivariate normal distribution in which the mean is calculated by a fully connected NN and the stddev is fixed (diagonal identity matrix multiplied by a constant) I will implement   (μ0,σ0,μ1,σ1,μ2,σ2,μ3,σ3) as you suggested.  
  
  ---
  
  ## 0 answers
  
  