Answers.Body: <blockquote>
  <p>First, is it even possible to use DDPG for multi-dimensional
  continuous action spaces?</p>
  </blockquote>
  <p>Yes, DDPG was primarily developed to deal with continuous action space you can find out more <a href="https://towardsdatascience.com/deep-deterministic-policy-gradient-ddpg-theory-and-implementation-747a3010e82f" rel="nofollow noreferrer">here</a>, <a href="https://towardsdatascience.com/deep-deterministic-policy-gradients-explained-2d94655a9b7b" rel="nofollow noreferrer">here</a> and <a href="https://openreview.net/pdf?id=HyxnH64KwS" rel="nofollow noreferrer">here</a>.</p>
  <blockquote>
  <p>I have not found any code examples to learn from and many of the
  papers I have read are near the limit of my understanding in this
  area.</p>
  </blockquote>
  <p>You can check <a href="https://keras-rl.readthedocs.io/en/latest/agents/ddpg/" rel="nofollow noreferrer">kerasRL</a> ddpg implementation it is quite easy to understand and can solve your problem of a model with multiple outputs..</p>
  <blockquote>
  <p>why might my actor network be outputting values clustered near its
  max/min values, and why would the values in either cluster all be the
  same?</p>
  </blockquote>
  <p>I think it is just a convergence problem try out small learning rates and/or small target_model_update param.</p>
  <p>If this doesn't work for you check TRPO and/or PPO..</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-21T13:45:35.120Z
Answers.Text: Can we use multi-agent deep deterministic policy gradient for discrete actions?
Answers.UserDisplayName: mohamed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-29T10:31:50.990Z
Answers.LastActivityDate: 2020-12-29T10:31:50.990Z
Answers.UserDisplayName: BAKYAC
Answers.Body: <blockquote>
  <p>why might my actor network be outputting values clustered near its
  max/min values, and why would the values in either cluster all be the
  same?</p>
  </blockquote>
  <p>Try to normalise the input to NNs. Try Layer Norms also.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-03T15:19:11.430Z
Answers.LastActivityDate: 2024-08-03T15:19:11.430Z
Answers.UserDisplayName: Mohammad Ammar
Body: <p>I am relatively new to reinforcement learning, and I am trying to implement a reinforcement learning algorithm that can do continuous control in a custom environment. The state of the environment is composed of 5 sequential observations of 11 continuous values that represent cells (making the observation space 55 continuous values total), and the action is 11 continuous values representing multipliers of those cell totals.</p>
  <p>After preliminary research, I decided to use Deep Deterministic Policy Gradient (DDPG) as my control algorithm because of its ability to deal with both discrete states and actions. However, most of the examples, including <a href="https://keras.io/examples/rl/ddpg_pendulum/" rel="nofollow noreferrer">the one that I am basing my implementation off of</a>, have only a single continuously valued action as the output. I have tried to naively change the agent network from outputting a single value to output a vector of values, but the agent does not improve as all, and the  set of outputs seems to split into two groups near either the maximum value or the minimum value (I believe the tanh activation on the output has something to do with it) with the values in those groups changing in unison.</p>
  <p>I have two questions about my problems.</p>
  <ol>
  <li><p>First, is it even possible to use DDPG for multi-dimensional continuous action spaces? My research leads me to believe it is, but I have not found any code examples to learn from and many of the papers I have read are near the limit of my understanding in this area.</p>
  </li>
  <li><p>Second, why might my actor network be outputting values clustered near its max/min values, and why would the values in either cluster all be the same?</p>
  </li>
  </ol>
  <p>Again, I am fairly new to reinforcement learning, so any advice or recommendations would be greatly appreciated, thanks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-04T03:42:30.280Z
Comments.Text: "I have tried to naively change the agent network from outputting a single value to output a vector of values"
  Hello, I tried the same method,is it right to change the agent network from outputting a single value to output a vector of values? If it's not right,how can I get multi outputs from actor of DDPG?
Comments.UserDisplayName: ERGOU CHEN
Tags: <reinforcement-learning><keras><ddpg><control-problem>
Title: Using DDPG for control in multi-dimensional continuous action space?
UserDisplayName: Dale Larie
ThreadText: # Using DDPG for control in multi-dimensional continuous action space?
  
  (Asked by: username_0 on 2020-08-17)
  
  I am relatively new to reinforcement learning, and I am trying to implement a reinforcement learning algorithm that can do continuous control in a custom environment. The state of the environment is composed of 5 sequential observations of 11 continuous values that represent cells (making the observation space 55 continuous values total), and the action is 11 continuous values representing multipliers of those cell totals.
  
  After preliminary research, I decided to use Deep Deterministic Policy Gradient (DDPG) as my control algorithm because of its ability to deal with both discrete states and actions. However, most of the examples, including [the one that I am basing my implementation off of](https://keras.io/examples/rl/ddpg_pendulum/), have only a single continuously valued action as the output. I have tried to naively change the agent network from outputting a single value to output a vector of values, but the agent does not improve as all, and the set of outputs seems to split into two groups near either the maximum value or the minimum value (I believe the tanh activation on the output has something to do with it) with the values in those groups changing in unison.
  
  I have two questions about my problems.
  
  1. First, is it even possible to use DDPG for multi-dimensional continuous action spaces? My research leads me to believe it is, but I have not found any code examples to learn from and many of the papers I have read are near the limit of my understanding in this area.
  2. Second, why might my actor network be outputting values clustered near its max/min values, and why would the values in either cluster all be the same?
  
  Again, I am fairly new to reinforcement learning, so any advice or recommendations would be greatly appreciated, thanks.
  
  **Tags:** reinforcement-learning, keras, ddpg, control-problem
  
  > **Comments:**
  > **username_1**: "I have tried to naively change the agent network from outputting a single value to output a vector of values" Hello, I tried the same method,is it right to change the agent network from outputting a single value to output a vector of values? If it's not right,how can I get multi outputs from actor of DDPG?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-12-29. Score: 1 | Accepted answer
  
  > First, is it even possible to use DDPG for multi-dimensional
  > continuous action spaces?
  
  Yes, DDPG was primarily developed to deal with continuous action space you can find out more [here](https://towardsdatascience.com/deep-deterministic-policy-gradient-ddpg-theory-and-implementation-747a3010e82f), [here](https://towardsdatascience.com/deep-deterministic-policy-gradients-explained-2d94655a9b7b) and [here](https://openreview.net/pdf?id=HyxnH64KwS).
  
  > I have not found any code examples to learn from and many of the
  > papers I have read are near the limit of my understanding in this
  > area.
  
  You can check [kerasRL](https://keras-rl.readthedocs.io/en/latest/agents/ddpg/) ddpg implementation it is quite easy to understand and can solve your problem of a model with multiple outputs..
  
  > why might my actor network be outputting values clustered near its
  > max/min values, and why would the values in either cluster all be the
  > same?
  
  I think it is just a convergence problem try out small learning rates and/or small target\_model\_update param.
  
  If this doesn't work for you check TRPO and/or PPO..
  
  > **Comments:**
  > **username_3**: Can we use multi-agent deep deterministic policy gradient for discrete actions?  
  
  ---
  
  ### Answer by username_4 on 2024-08-03. Score: 0
  
  > why might my actor network be outputting values clustered near its
  > max/min values, and why would the values in either cluster all be the
  > same?
  
  Try to normalise the input to NNs. Try Layer Norms also.
  
  
  ---
  
  