Answers.Body: <p>(1) You might want look into RND (Random network distillation) which allows usage of a curiosity-based exploration bonus for the agent as an intrinsic reward. You can use the intrinsic reward to complement the sparse extrinsic reward return by the environment. </p>
  
  <p>The general idea is to have a randomly initialized fixed target network which encodes the next state &amp; a predictor network is trained to predict the output of the target network. The prediction error is used to "quantify the novelty of new experience". Stronger novelty will be a good indication for the agent that it maybe worthwhile to explore more. </p>
  
  <p>The authors of this (A) <a href="https://arxiv.org/pdf/1810.12894.pdf" rel="nofollow noreferrer">paper</a> were able to achieve SOTA performance in Montezuma's Revenge, which is notorious for it's sparse reward.</p>
  
  <p>In appendix A.1, It is mentioned that: "An exploration bonus can be used with any RL algorithm by modifying the rewards used to train the
  model (i.e., rt = it + et)." It is also mentioned that the authors combined this exploration bonus with PPO (which also works in continuous action space). In A.2, a pseudo code is provided.</p>
  
  <p>I would also recommend this (B) <a href="https://arxiv.org/pdf/1908.02388.pdf" rel="nofollow noreferrer">paper</a> (see section 3) if you're interested in exploring the available Bonus-Based Exploration Methods out there which may help in tackling hard exploration games with sparse rewards.</p>
  
  <p>With regards to high stochasticity &amp; variance, I found an interesting remark (on page 3, under Figure 2) in this (C) <a href="https://arxiv.org/pdf/1708.04133.pdf" rel="nofollow noreferrer">paper</a>: </p>
  
  <p>"our investigation of DDPG on different network configurations shows that for the Hopper environment, DDPG is quite unstable no matter the network architecture. This can be attributed partially to the high variance of DDPG itself, but also to the increased stochasticity of the Hopper
  task."</p>
  
  <p>The remark was made in the context where the authors were trying to "tune DDPG to reproduce results from other works even when using their reported hyper-parameter settings".</p>
  
  <p>Have a look <a href="https://spinningup.openai.com/en/latest/spinningup/bench.html#hopper-pytorch-versions" rel="nofollow noreferrer">here</a> for a different benchmark on how DDPG fair against other algorithms.</p>
  
  <p>(2) From the information provided, I can't conclusively provide you a quantitative assessment on DDPG performance for your specific problem. However, I would recommend the following:</p>
  
  <p>(a) I will encourage you to try different RL algorithms when face with a difficult problem so that you can benchmark &amp; find out which is more suitable. Also in (A), the authors mentioned, "PPO is a policy gradient method that we have found to require little tuning for good performance." </p>
  
  <p>(b) Try different sets of hyperparameters. There are many ways to tune them systematically but discussion about this will be out of scope for this question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T03:42:19.953Z
Answers.LastActivityDate: 2020-04-26T06:42:18.690Z
Answers.LastEditDate: 2020-04-26T06:42:18.690Z
Answers.UserDisplayName: Huan
Body: <p>I'm working on a RL problem with the following properties:</p>
  
  <ol>
  <li>The rewards are <strong>extremely sparse</strong> i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering as that would lead to a different optimization problem.</li>
  <li><strong>Actions are continuous</strong>. Discretization should not be used.</li>
  <li>The amount of <strong>stochasticity in the environment is very high</strong> i.e. for a fixed deterministic policy the variance of returns is very high.</li>
  </ol>
  
  <p>More specifically, the RL agent represents the investor, the terminal reward represents the utility of the terminal wealth (hence the sparsity), actions represent portfolio positions (hence the continuity) and the environment represents the financial market (hence the high stochasticity).</p>
  
  <p>I've been trying to use DDPG with a set of "commonly used" hyperparameters (as I have no idea have to tune them besides experimenting which lasts too long) but so far (after 10000 episodes) it seems that nothing is happening.</p>
  
  <p><a href="https://i.sstatic.net/yjA9O.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yjA9O.png" alt="enter image description here"></a></p>
  
  <p>My questions are the following:</p>
  
  <ol>
  <li>Given the nature of the problem I'm trying to solve (sparse rewards, continuous actions, stochasticity) is there a particular (D)RL algorithm that would lend itself well to it?</li>
  <li>How likely is it that DDPG simply won't converge to a reasonable solution (due to the peculiarities of the problem itself) no matter what set of hyperparameters I choose?</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-23T10:37:05.447Z
Comments.Text: You seem to be using the right kind of algorithm (I would think that most policy gradient methods could at least be applied here, although some may perform better than others). Your problem may well be intractable though. Do you have any evidence that it is solvable given the state data you are using?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-23T10:43:01.220Z
Comments.Text: The optimization problem I'm trying to solve has been approximated analytically - the approximation to the optimal solution depends on three variables (t-remaining time, q-current inventory and s-current price). It is precisely those variables that I've included in my state space. Therefore, RL should in theory be able to learn this function, provided enough episodes and an NN function approximator with enough capacity.
Comments.UserDisplayName: BGa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-18T17:52:31.147Z
Comments.Text: Use Introduction Multi agent sysrtem
Comments.UserDisplayName: adeleye ifeoluwa seun
Tags: <reinforcement-learning><rewards><policy-gradients><ddpg>
Title: Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity
UserDisplayName: BGa
ThreadText: # Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity
  
  (Asked by: username_0 on 2020-04-23)
  
  I'm working on a RL problem with the following properties:
  
  1. The rewards are **extremely sparse** i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering as that would lead to a different optimization problem.
  2. **Actions are continuous**. Discretization should not be used.
  3. The amount of **stochasticity in the environment is very high** i.e. for a fixed deterministic policy the variance of returns is very high.
  
  More specifically, the RL agent represents the investor, the terminal reward represents the utility of the terminal wealth (hence the sparsity), actions represent portfolio positions (hence the continuity) and the environment represents the financial market (hence the high stochasticity).
  
  I've been trying to use DDPG with a set of "commonly used" hyperparameters (as I have no idea have to tune them besides experimenting which lasts too long) but so far (after 10000 episodes) it seems that nothing is happening.
  
  [![enter image description here](https://i.sstatic.net/yjA9O.png)](https://i.sstatic.net/yjA9O.png)
  
  My questions are the following:
  
  1. Given the nature of the problem I'm trying to solve (sparse rewards, continuous actions, stochasticity) is there a particular (D)RL algorithm that would lend itself well to it?
  2. How likely is it that DDPG simply won't converge to a reasonable solution (due to the peculiarities of the problem itself) no matter what set of hyperparameters I choose?
  
  **Tags:** reinforcement-learning, rewards, policy-gradients, ddpg
  > **Comments:**
  > **username_1**: You seem to be using the right kind of algorithm (I would think that most policy gradient methods could at least be applied here, although some may perform better than others). Your problem may well be intractable though. Do you have any evidence that it is solvable given the state data you are using?  
  > **username_0**: The optimization problem I'm trying to solve has been approximated analytically - the approximation to the optimal solution depends on three variables (t-remaining time, q-current inventory and s-current price). It is precisely those variables that I've included in my state space. Therefore, RL should in theory be able to learn this function, provided enough episodes and an NN function approximator with enough capacity.  
  > **username_2**: Use Introduction Multi agent sysrtem  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-04-26. Score: 4
  
  (1) You might want look into RND (Random network distillation) which allows usage of a curiosity-based exploration bonus for the agent as an intrinsic reward. You can use the intrinsic reward to complement the sparse extrinsic reward return by the environment.
  
  The general idea is to have a randomly initialized fixed target network which encodes the next state & a predictor network is trained to predict the output of the target network. The prediction error is used to "quantify the novelty of new experience". Stronger novelty will be a good indication for the agent that it maybe worthwhile to explore more.
  
  The authors of this (A) [paper](https://arxiv.org/pdf/1810.12894.pdf) were able to achieve SOTA performance in Montezuma's Revenge, which is notorious for it's sparse reward.
  
  In appendix A.1, It is mentioned that: "An exploration bonus can be used with any RL algorithm by modifying the rewards used to train the
  model (i.e., rt = it + et)." It is also mentioned that the authors combined this exploration bonus with PPO (which also works in continuous action space). In A.2, a pseudo code is provided.
  
  I would also recommend this (B) [paper](https://arxiv.org/pdf/1908.02388.pdf) (see section 3) if you're interested in exploring the available Bonus-Based Exploration Methods out there which may help in tackling hard exploration games with sparse rewards.
  
  With regards to high stochasticity & variance, I found an interesting remark (on page 3, under Figure 2) in this (C) [paper](https://arxiv.org/pdf/1708.04133.pdf):
  
  "our investigation of DDPG on different network configurations shows that for the Hopper environment, DDPG is quite unstable no matter the network architecture. This can be attributed partially to the high variance of DDPG itself, but also to the increased stochasticity of the Hopper
  task."
  
  The remark was made in the context where the authors were trying to "tune DDPG to reproduce results from other works even when using their reported hyper-parameter settings".
  
  Have a look [here](https://spinningup.openai.com/en/latest/spinningup/bench.html#hopper-pytorch-versions) for a different benchmark on how DDPG fair against other algorithms.
  
  (2) From the information provided, I can't conclusively provide you a quantitative assessment on DDPG performance for your specific problem. However, I would recommend the following:
  
  (a) I will encourage you to try different RL algorithms when face with a difficult problem so that you can benchmark & find out which is more suitable. Also in (A), the authors mentioned, "PPO is a policy gradient method that we have found to require little tuning for good performance."
  
  (b) Try different sets of hyperparameters. There are many ways to tune them systematically but discussion about this will be out of scope for this question.
  
  
  ---
  
  