Answers.Body: <p>I'm using my own implementation of A2C (Advantage Actor Critic) in an industrial application based on Markov Process (present state alone provides sufficient knowledge to make an optimal decision). It's simple and versatile, its performance proven in many different applications. The results so far have been promising.</p>
  <p>One of my colleagues had issues with solving a simple task of mapping images to coordinates with OpenAI's Stable Baselines implementations of PPO and TRPO. Hence I'm biased against this framework.</p>
  <p>My suggestion is to try the simplest model and if that doesn't satisfy your expectations for performance, then try something fancier. Once you've made a pipeline for learning, switching to a different algorithm is relatively time inexpensive.</p>
  <p>Here is a list of algorithms for continuous action and state space from the <a href="https://en.wikipedia.org/wiki/Reinforcement_learning" rel="nofollow noreferrer">wikipedia</a> article about RL:</p>
  <p><a href="https://i.sstatic.net/RHIrT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RHIrT.png" alt="list of RL algorithms with continuous action and state space" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-28T10:10:53.170Z
Answers.LastActivityDate: 2020-07-28T10:16:43.780Z
Answers.LastEditDate: 2020-07-28T10:16:43.780Z
Answers.UserDisplayName: conscious_process
Body: <p>I'm trying to decide which policy improvement algorithm to use in the context of my problem. But let me emerge you into the problem</p>
  <p><strong>Problem</strong></p>
  <p>I want to move a set of points in a 3D space. Depending on how the points move, the environment gives a positive or negative reward. Further, the environment does not split up into episodes, so it is a continuing problem. The state space is high-dimensional (a lot of states are possible) and many states can be similar (so state aliasing can appear), also states are continuous. The problem is dense in rewards, so for every transition, there will be a negative or positive reward, depending on the previous state.</p>
  <p>A state is represented as a vector with dimension N (initially it will be something like ~100, but in the future, I want to work with vectors up to 1000).</p>
  <p>In the case of action, it is described by a matrix 3xN, where N is the same as in the case of the state. The first dimension comes from the fact, that action is 3D displacement.</p>
  <p><strong>What I have done so far</strong></p>
  <p>Since actions are continuous, I have narrowed down my search to policy gradient methods. Further, I researched methods, that work with continuous state spaces. I found a deep deterministic policy gradient (DDPG) and the Proximal Policy Gradient (PPO) would fit here. Theoretically, they should work but I'm unsure and any advice would be gold here.</p>
  <p><strong>Questions</strong></p>
  <p>Would those algorithms be suitable for the problem (PPO or DDPG)?
  There are other policy improvement algorithms that would work here or a family of policy improvement algorithms?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-26T22:38:57.203Z
Comments.Text: You could try https://arxiv.org/pdf/1804.08617v1.pdf. It's from Deepmind (the authors of DDPG). They also provided code.
Comments.UserDisplayName: Schach21
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-27T18:09:00.763Z
Comments.Text: Thanks @Schach21, every piece is a valueable one.
Comments.UserDisplayName: Daniel Wiczew
Tags: <reinforcement-learning><policy-gradients>
Title: Choosing a policy improvement algorithm for a continuing problem with continuous action and state-space
UserDisplayName: Daniel Wiczew
ThreadText: # Choosing a policy improvement algorithm for a continuing problem with continuous action and state-space
  
  (Asked by: username_0 on 2020-07-22)
  
  I'm trying to decide which policy improvement algorithm to use in the context of my problem. But let me emerge you into the problem
  
  **Problem**
  
  I want to move a set of points in a 3D space. Depending on how the points move, the environment gives a positive or negative reward. Further, the environment does not split up into episodes, so it is a continuing problem. The state space is high-dimensional (a lot of states are possible) and many states can be similar (so state aliasing can appear), also states are continuous. The problem is dense in rewards, so for every transition, there will be a negative or positive reward, depending on the previous state.
  
  A state is represented as a vector with dimension N (initially it will be something like ~100, but in the future, I want to work with vectors up to 1000).
  
  In the case of action, it is described by a matrix 3xN, where N is the same as in the case of the state. The first dimension comes from the fact, that action is 3D displacement.
  
  **What I have done so far**
  
  Since actions are continuous, I have narrowed down my search to policy gradient methods. Further, I researched methods, that work with continuous state spaces. I found a deep deterministic policy gradient (DDPG) and the Proximal Policy Gradient (PPO) would fit here. Theoretically, they should work but I'm unsure and any advice would be gold here.
  
  **Questions**
  
  Would those algorithms be suitable for the problem (PPO or DDPG)?
  There are other policy improvement algorithms that would work here or a family of policy improvement algorithms?
  
  **Tags:** reinforcement-learning, policy-gradients
  > **Comments:**
  > **username_1**: You could try https://arxiv.org/pdf/1804.08617v1.pdf. It's from Deepmind (the authors of DDPG). They also provided code.  
  > **username_0**: Thanks @Schach21, every piece is a valueable one.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-28. Score: 1
  
  I'm using my own implementation of A2C (Advantage Actor Critic) in an industrial application based on Markov Process (present state alone provides sufficient knowledge to make an optimal decision). It's simple and versatile, its performance proven in many different applications. The results so far have been promising.
  
  One of my colleagues had issues with solving a simple task of mapping images to coordinates with OpenAI's Stable Baselines implementations of PPO and TRPO. Hence I'm biased against this framework.
  
  My suggestion is to try the simplest model and if that doesn't satisfy your expectations for performance, then try something fancier. Once you've made a pipeline for learning, switching to a different algorithm is relatively time inexpensive.
  
  Here is a list of algorithms for continuous action and state space from the [wikipedia](https://en.wikipedia.org/wiki/Reinforcement_learning) article about RL:
  
  [![list of RL algorithms with continuous action and state space](https://i.sstatic.net/RHIrT.png)](https://i.sstatic.net/RHIrT.png)
  
  
  ---
  
  