Answers.Body: <p>Disclaimer: <strong>Without the full code, we can only speculate.</strong> I encourage you to post the full code on <a href="http://colab.research.google.com/" rel="nofollow noreferrer">Google Colab</a> or something like this.
  In the meanwhile, here is my <em>point of view</em>:</p>
  <hr />
  <h3>The Problem</h3>
  <p>Looks like your model has found some &quot;<em>master action</em>&quot; that always leads to zero loss, no matter what the state is. So it's not necessarily <em>bad</em>, it's just <em>unexpected</em> according to your point of view.</p>
  <blockquote>
  <p>An example for that would be <strong>pausing</strong> the game - so you never loose.</p>
  </blockquote>
  <p>You might not like it, but in de model's point of view, it's absolutely nailing it!</p>
  <h3>The Solution</h3>
  <p>So how to convince the actor not to <em>pause the game</em>?</p>
  <p><strong>Not by changing the model, or tuning hyper-parameters, but by reformulating the problem.</strong> In this example, instead of just penalizing the model for failing, you should reward if for winning, so pausing is no longer the best option.</p>
  <h3>Conclusion</h3>
  <p>It might not be a problem in the Machine Learning model, but in your environment and reward models. As we don't have access to that, it's hard to provide an answer.</p>
  <hr />
  <h3>Edit:</h3>
  <p>You are the CartPole-v0 environment:</p>
  <blockquote>
  <p>A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center.</p>
  </blockquote>
  <p>Source: <a href="https://gym.openai.com/envs/CartPole-v0/" rel="nofollow noreferrer">https://gym.openai.com/envs/CartPole-v0/</a></p>
  <p>It is a solvable problem. Probably your model has just learned how to solve it after a few hundreds generations. (The link shows a table with &quot;Episodes before solve&quot; for each algorithm, showing numbers consistent to yours).</p>
  <h3>TL;DR: It's not a bug, it's a feature!</h3>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-29T17:38:06.733Z
Answers.Text: The original code is here  https://github.com/KokoMind/Actor-Critic-TF/tree/d062227c1fc610139a6f026fa63bb6069230bdc8
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-06T06:46:29.473Z
Answers.Text: The problem was in the state and the reward.
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-29T16:56:58.180Z
Answers.LastActivityDate: 2021-08-29T21:05:51.427Z
Answers.LastEditDate: 2021-08-29T21:05:51.427Z
Answers.UserDisplayName: Andre Goulart
Answers.Body: <p>If the loss is zero, all gradients should be zero as well, so you should take a look at the computed gradients.
  There might be a problem with momentum or the scheduled lr which might still apply very little updates which eventually lead to this <a href="https://datascience.stackexchange.com/a/22221">policy colapse</a>.</p>
  <p>On a side note I would also call <code>reduce_mean</code> on the actor loss since you're optimizing the expected value.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-27T14:19:17.903Z
Answers.Text: My code was like this `actorloss = tf.reduce_mean(tf.losses.huber_loss(picked_action_prob, actortarget, delta=1.0), name='actorloss')` but nothing change.
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-28T11:42:17.510Z
Answers.Text: Are the gradients zero?
Answers.UserDisplayName: tnfru
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-28T14:07:51.133Z
Answers.Text: No, it is not zero.
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-28T14:11:36.683Z
Answers.Text: But your loss is? Because if it is there might be some kind of momentum from your optimizer applying steps
Answers.UserDisplayName: tnfru
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-28T14:16:23.583Z
Answers.Text: I have changed it, and train the model again but it is not changing.
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-29T11:28:33.067Z
Answers.Text: You need to find out the source of the non zero gradient. Your policy keeps moving (probably in the same direction) while your loss is zero. This eventually leads to selecting the same action almost every time and the policy colapse by this.
  
  You could also try lr decay, but I feel like it only minimizes the problem instead of solving it.
Answers.UserDisplayName: tnfru
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-27T08:25:54.203Z
Answers.LastActivityDate: 2021-08-27T08:25:54.203Z
Answers.UserDisplayName: tnfru
Body: <p>I'm working on an <a href="https://github.com/KokoMind/Actor-Critic-TF/tree/d062227c1fc610139a6f026fa63bb6069230bdc8" rel="nofollow noreferrer">advantage actor-critic (A2C)</a> reinforcement learning model, but when I test the model after I trained for 3500 episodes, I start to get almost the same action for all testing episodes. While if I trained the system for less than 850 episodes, I got different actions. The value of <code>state</code> is always different, and around 850 episodes, the <code>loss</code> becomes zero.</p>
  <p>Here is the Actor and critic Network</p>
  <pre><code>        with g.as_default():
              #==============================actor==============================#
              actorstate = tf.placeholder(dtype=tf.float32, shape=n_input, name='state')
              actoraction = tf.placeholder(dtype=tf.int32, name='action')
              actortarget = tf.placeholder(dtype=tf.float32, name='target')
  
              hidden_layer1 = tf.layers.dense(inputs=tf.expand_dims(actorstate, 0), units=500, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              hidden_layer2 = tf.layers.dense(inputs=hidden_layer1, units=250, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              hidden_layer3 = tf.layers.dense(inputs=hidden_layer2, units=120, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              output_layer = tf.layers.dense(inputs=hidden_layer3, units=n_output, kernel_initializer=tf.zeros_initializer())
              action_probs = tf.squeeze(tf.nn.softmax(output_layer))
              picked_action_prob = tf.gather(action_probs, actoraction)
  
              actorloss = -tf.log(picked_action_prob) * actortarget
              # actorloss = tf.reduce_mean(tf.losses.huber_loss(picked_action_prob, actortarget, delta=1.0), name='actorloss')
  
              actoroptimizer1 = tf.train.AdamOptimizer(learning_rate=var.learning_rate)
  
              if var.opt == 2:
                  actoroptimizer1 = tf.train.RMSPropOptimizer(learning_rate=var.learning_rate, momentum=0.95,
                                                              epsilon=0.01)
              elif var.opt == 0:
                  actoroptimizer1 = tf.train.GradientDescentOptimizer(learning_rate=var.learning_rate)
  
              actortrain_op = actoroptimizer1.minimize(actorloss)
  
              init = tf.global_variables_initializer()
              saver = tf.train.Saver(max_to_keep=var.n)
  
          p = tf.Graph()
          with p.as_default():
              #==============================critic==============================#
              criticstate = tf.placeholder(dtype=tf.float32, shape=n_input, name='state')
              critictarget = tf.placeholder(dtype=tf.float32, name='target')
  
              hidden_layer4 = tf.layers.dense(inputs=tf.expand_dims(criticstate, 0), units=500, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              hidden_layer5 = tf.layers.dense(inputs=hidden_layer4, units=250, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              hidden_layer6 = tf.layers.dense(inputs=hidden_layer5, units=120, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              output_layer2 = tf.layers.dense(inputs=hidden_layer6, units=1, kernel_initializer=tf.zeros_initializer())
              value_estimate = tf.squeeze(output_layer2)
  
              criticloss= tf.reduce_mean(tf.losses.huber_loss(output_layer2, critictarget,delta = 0.5), name='criticloss')
              optimizer2 = tf.train.AdamOptimizer(learning_rate=var.learning_rateMADDPG_c)
              if var.opt == 2:
                  optimizer2 = tf.train.RMSPropOptimizer(learning_rate=var.learning_rate_c, momentum=0.95,
                                                              epsilon=0.01)
              elif var.opt == 0:
                  optimizer2 = tf.train.GradientDescentOptimizer(learning_rate=var.learning_rateMADDPG_c)
  
              update_step2 = optimizer2.minimize(criticloss)
  
              init2 = tf.global_variables_initializer()
              saver2 = tf.train.Saver(max_to_keep=var.n)
  
   
  </code></pre>
  <p>This is the choice of action.</p>
  <pre><code>def take_action(self, state):
                  &quot;&quot;&quot;Take the action&quot;&quot;&quot;
                  action_probs = self.actor.predict(state)
                  action = np.random.choice(np.arange(len(action_probs)), p=action_probs)
                  return action
  </code></pre>
  <p>This is the <code>actor.predict</code> function.</p>
  <pre><code>def predict(self, s):
          return self._sess.run(self._action_probs, {self._state: s})
  </code></pre>
  <p>Any Idea what causing this?</p>
  <p><strong>Update</strong></p>
  <p>Change the learning rate, state, and the reward solve the problem where I reduce the size of the state and also added switching cost to the reward.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-22T21:12:03.460Z
Comments.Text: Not really possible to answer without seeing the full code. Can you post a colab notebook with a MWE? At the least we need more details such as a) what is the MDP, b) how are actions parametrized. You say you're getting the same action (I assume you mean over all timesteps in the episode) - is it possible that that is the optimal policy?
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-22T21:19:23.257Z
Comments.Text: I got different actions when I train the model and it converges perfectly, but it is not changing during the testing for all the episodes.
Comments.UserDisplayName: I_Al-thamary
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-22T21:44:11.673Z
Comments.Text: The code is taken from https://www.datahubbs.com/policy-gradients-and-advantage-actor-critic/ and https://stackoverflow.com/questions/45428574/actor-critic-model-never-converges
Comments.UserDisplayName: I_Al-thamary
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-23T19:50:36.497Z
Comments.Text: @Taw, I have added the code.
Comments.UserDisplayName: I_Al-thamary
Tags: <reinforcement-learning><python><actor-critic-methods><advantage-actor-critic><a3c>
Title: Why I got the same action when testing the A2C?
UserDisplayName: I_Al-thamary
ThreadText: # Why I got the same action when testing the A2C?
  
  (Asked by: username_0 on 2020-01-16)
  
  I'm working on an [advantage actor-critic (A2C)](https://github.com/KokoMind/Actor-Critic-TF/tree/d062227c1fc610139a6f026fa63bb6069230bdc8) reinforcement learning model, but when I test the model after I trained for 3500 episodes, I start to get almost the same action for all testing episodes. While if I trained the system for less than 850 episodes, I got different actions. The value of `state` is always different, and around 850 episodes, the `loss` becomes zero.
  
  Here is the Actor and critic Network
  
  ```
          with g.as_default():
              #==============================actor==============================#
              actorstate = tf.placeholder(dtype=tf.float32, shape=n_input, name='state')
              actoraction = tf.placeholder(dtype=tf.int32, name='action')
              actortarget = tf.placeholder(dtype=tf.float32, name='target')
  
              hidden_layer1 = tf.layers.dense(inputs=tf.expand_dims(actorstate, 0), units=500, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              hidden_layer2 = tf.layers.dense(inputs=hidden_layer1, units=250, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              hidden_layer3 = tf.layers.dense(inputs=hidden_layer2, units=120, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              output_layer = tf.layers.dense(inputs=hidden_layer3, units=n_output, kernel_initializer=tf.zeros_initializer())
              action_probs = tf.squeeze(tf.nn.softmax(output_layer))
              picked_action_prob = tf.gather(action_probs, actoraction)
  
              actorloss = -tf.log(picked_action_prob) * actortarget
              # actorloss = tf.reduce_mean(tf.losses.huber_loss(picked_action_prob, actortarget, delta=1.0), name='actorloss')
  
              actoroptimizer1 = tf.train.AdamOptimizer(learning_rate=var.learning_rate)
  
              if var.opt == 2:
                  actoroptimizer1 = tf.train.RMSPropOptimizer(learning_rate=var.learning_rate, momentum=0.95,
                                                              epsilon=0.01)
              elif var.opt == 0:
                  actoroptimizer1 = tf.train.GradientDescentOptimizer(learning_rate=var.learning_rate)
  
              actortrain_op = actoroptimizer1.minimize(actorloss)
  
              init = tf.global_variables_initializer()
              saver = tf.train.Saver(max_to_keep=var.n)
  
          p = tf.Graph()
          with p.as_default():
              #==============================critic==============================#
              criticstate = tf.placeholder(dtype=tf.float32, shape=n_input, name='state')
              critictarget = tf.placeholder(dtype=tf.float32, name='target')
  
              hidden_layer4 = tf.layers.dense(inputs=tf.expand_dims(criticstate, 0), units=500, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              hidden_layer5 = tf.layers.dense(inputs=hidden_layer4, units=250, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              hidden_layer6 = tf.layers.dense(inputs=hidden_layer5, units=120, activation=tf.nn.relu, kernel_initializer=tf.zeros_initializer())
              output_layer2 = tf.layers.dense(inputs=hidden_layer6, units=1, kernel_initializer=tf.zeros_initializer())
              value_estimate = tf.squeeze(output_layer2)
  
              criticloss= tf.reduce_mean(tf.losses.huber_loss(output_layer2, critictarget,delta = 0.5), name='criticloss')
              optimizer2 = tf.train.AdamOptimizer(learning_rate=var.learning_rateMADDPG_c)
              if var.opt == 2:
                  optimizer2 = tf.train.RMSPropOptimizer(learning_rate=var.learning_rate_c, momentum=0.95,
                                                              epsilon=0.01)
              elif var.opt == 0:
                  optimizer2 = tf.train.GradientDescentOptimizer(learning_rate=var.learning_rateMADDPG_c)
  
              update_step2 = optimizer2.minimize(criticloss)
  
              init2 = tf.global_variables_initializer()
              saver2 = tf.train.Saver(max_to_keep=var.n)
  
   
  
  ```
  
  This is the choice of action.
  
  ```
  def take_action(self, state):
                  """Take the action"""
                  action_probs = self.actor.predict(state)
                  action = np.random.choice(np.arange(len(action_probs)), p=action_probs)
                  return action
  
  ```
  
  This is the `actor.predict` function.
  
  ```
  def predict(self, s):
          return self._sess.run(self._action_probs, {self._state: s})
  
  ```
  
  Any Idea what causing this?
  
  **Update**
  
  Change the learning rate, state, and the reward solve the problem where I reduce the size of the state and also added switching cost to the reward.
  
  **Tags:** reinforcement-learning, python, actor-critic-methods, advantage-actor-critic, a3c
  
  > **Comments:**
  > **username_1**: Not really possible to answer without seeing the full code. Can you post a colab notebook with a MWE? At the least we need more details such as a) what is the MDP, b) how are actions parametrized. You say you're getting the same action (I assume you mean over all timesteps in the episode) - is it possible that that is the optimal policy?  
  > **username_0**: I got different actions when I train the model and it converges perfectly, but it is not changing during the testing for all the episodes.  
  > **username_0**: The code is taken from https://www.datahubbs.com/policy-gradients-and-advantage-actor-critic/ and https://stackoverflow.com/questions/45428574/actor-critic-model-never-converges  
  > **username_0**: @username_1, I have added the code.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-08-29. Score: 1 | Accepted answer
  
  Disclaimer: **Without the full code, we can only speculate.** I encourage you to post the full code on [Google Colab](http://colab.research.google.com/) or something like this.
  In the meanwhile, here is my *point of view*:
  
  ---
  
  ### The Problem
  
  Looks like your model has found some "*master action*" that always leads to zero loss, no matter what the state is. So it's not necessarily *bad*, it's just *unexpected* according to your point of view.
  
  > An example for that would be **pausing** the game - so you never loose.
  
  You might not like it, but in de model's point of view, it's absolutely nailing it!
  
  ### The Solution
  
  So how to convince the actor not to *pause the game*?
  
  **Not by changing the model, or tuning hyper-parameters, but by reformulating the problem.** In this example, instead of just penalizing the model for failing, you should reward if for winning, so pausing is no longer the best option.
  
  ### Conclusion
  
  It might not be a problem in the Machine Learning model, but in your environment and reward models. As we don't have access to that, it's hard to provide an answer.
  
  ---
  
  ### Edit:
  
  You are the CartPole-v0 environment:
  
  > A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center.
  
  Source: <https://gym.openai.com/envs/CartPole-v0/>
  
  It is a solvable problem. Probably your model has just learned how to solve it after a few hundreds generations. (The link shows a table with "Episodes before solve" for each algorithm, showing numbers consistent to yours).
  
  ### TL;DR: It's not a bug, it's a feature!
  
  > **Comments:**
  > **username_0**: The original code is here  https://github.com/KokoMind/Actor-Critic-TF/tree/d062227c1fc610139a6f026fa63bb6069230bdc8  
  > **username_0**: The problem was in the state and the reward.  
  
  ---
  
  ### Answer by username_3 on 2021-08-27. Score: 0
  
  If the loss is zero, all gradients should be zero as well, so you should take a look at the computed gradients.
  There might be a problem with momentum or the scheduled lr which might still apply very little updates which eventually lead to this [policy colapse](https://datascience.stackexchange.com/a/22221).
  
  On a side note I would also call `reduce_mean` on the actor loss since you're optimizing the expected value.
  
  > **Comments:**
  > **username_0**: My code was like this `actorloss = tf.reduce_mean(tf.losses.huber_loss(picked_action_prob, actortarget, delta=1.0), name='actorloss')` but nothing change.  
  > **username_3**: Are the gradients zero?  
  > **username_0**: No, it is not zero.  
  > **username_3**: But your loss is? Because if it is there might be some kind of momentum from your optimizer applying steps  
  > **username_0**: I have changed it, and train the model again but it is not changing.  
  > **username_3**: You need to find out the source of the non zero gradient. Your policy keeps moving (probably in the same direction) while your loss is zero. This eventually leads to selecting the same action almost every time and the policy colapse by this.  You could also try lr decay, but I feel like it only minimizes the problem instead of solving it.  
  
  ---
  
  