Answers.Body: <p>If your problem is a multi-armed bandit then the algorithms that you listed are not suited for the job. Those algorithms are for situations where the agent's actions has an influence on the subsequent states encountered. You should look for solutions designed for multi-armed bandit scenarios such as LinUCB.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-06T02:18:17.733Z
Answers.LastActivityDate: 2023-08-06T02:18:17.733Z
Answers.UserDisplayName: languageoftheuniverse
Body: <p><strong>Environment</strong></p>
  <p>I have a static timeseries environment meaning the environment is the same.<br />
  This problem is a multi armed bandit problem.</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Time</th>
  <th>t0</th>
  <th>t1</th>
  <th>t2</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>State</td>
  <td>s0</td>
  <td>s1</td>
  <td>s2</td>
  </tr>
  <tr>
  <td>Score</td>
  <td>10</td>
  <td>0.1</td>
  <td>0.2</td>
  </tr>
  <tr>
  <td>Class</td>
  <td>1</td>
  <td>0</td>
  <td>0</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Keep work = 1<br />
  Discard work = 0.1<br />
  Class = (Score&gt;9)</p>
  <p><strong>Rewards</strong></p>
  <p>Reward = <span class="math-container">$\frac{Score}{Work}$</span></p>
  <p>I have various states in the dataset which are ordered chronologically and each state contains a reward from (0 -&gt; inf)
  rewards come in 2 types a very small reward then very sparsely a high reward is given.</p>
  <p>To calculate the reward from a state is an expensive operation.
  The model can discard states to increase the reward density
  reward per work.</p>
  <p>For example lets consider 2 cases\</p>
  <ul>
  <li>model 0: do nothing</li>
  <li>model 1: optimal</li>
  </ul>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Model</th>
  <th>Metric</th>
  <th>t0</th>
  <th>t1</th>
  <th>t2</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>model 0</td>
  <td>decision</td>
  <td>keep</td>
  <td>keep</td>
  <td>keep</td>
  </tr>
  <tr>
  <td>model 0</td>
  <td>score</td>
  <td>10</td>
  <td>0.1</td>
  <td>0.2</td>
  </tr>
  <tr>
  <td>model 0</td>
  <td>work</td>
  <td>1</td>
  <td>1</td>
  <td>1</td>
  </tr>
  <tr>
  <td>model 1</td>
  <td>decision</td>
  <td>keep</td>
  <td>discard</td>
  <td>discard</td>
  </tr>
  <tr>
  <td>model 1</td>
  <td>score</td>
  <td>10</td>
  <td>0</td>
  <td>0</td>
  </tr>
  <tr>
  <td>model 1</td>
  <td>work</td>
  <td>1</td>
  <td>0.1</td>
  <td>0.1</td>
  </tr>
  </tbody>
  </table>
  </div>
  <ul>
  <li><p>model 0: reward = <span class="math-container">$\frac{10+0.1+0.2}{1+1+1}$</span> = 3.43</p>
  </li>
  <li><p>model 1: reward = <span class="math-container">$\frac{10+0+0}{1+0.1+0.1}$</span> = 8.33</p>
  </li>
  </ul>
  <p>To avoid the model discarding most states to make work value low a penalty can be applied if the model discards too many states.
  this is due to if keep/discard work is decreased reward will increase.
  As <span class="math-container">$reward\propto\frac{1}{work}$</span></p>
  <p><strong>States</strong></p>
  <p>Each timestep state is related by being an input to a process and is monotonic
  The state is equal to the process with the timestamp as the input
  Sn = P(tn)</p>
  <p><strong>Domain Knowledge</strong></p>
  <ol>
  <li>P is unknown</li>
  <li>high rewards are very sparse &gt;1:10000</li>
  </ol>
  <p><strong>Attempts so far/Ideas</strong><br />
  I have already tried using Bayesian optimisation on this problem with minimal luck.</p>
  <p>there are many different types of reinforcement learning algorithms.
  The ones applicable are</p>
  <ol>
  <li>output is classification(category) for keep/discard
  continuous state space -&gt; discrete action space</li>
  </ol>
  <p><a href="https://stackoverflow.com/questions/58550725/best-reinforcement-learning-algorithm-for-continuous-state-space-and-discrete-ac">stackoverflow</a></p>
  <ul>
  <li>TD3</li>
  <li>DDPG</li>
  <li>SAC</li>
  <li>PPO</li>
  </ul>
  <ol start="2">
  <li><p>output is decision boundary(value) for classification
  continuous state space -&gt; continuous action space</p>
  </li>
  <li><p>output is feature(value) which is then classified
  continuous state space -&gt; continuous action space</p>
  </li>
  </ol>
  <p><a href="https://stackoverflow.com/questions/7098625/how-can-i-apply-reinforcement-learning-to-continuous-action-spaces">stackoverflow</a>
  <a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.75.7658&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">CACLA paper</a></p>
  <ul>
  <li>A2C</li>
  <li>REINFORCE</li>
  <li>CACLA</li>
  </ul>
  <p><strong>Questions</strong></p>
  <ol>
  <li><p>What reinforcement learning algorithm should I use?</p>
  </li>
  <li><p>given the problem is 1,2 or 3 most sensible?</p>
  </li>
  <li><p>any other considerations I should be aware of?</p>
  </li>
  </ol>
  <p>Thanks</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-14T15:02:11.687Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Tags: <reinforcement-learning><classification><time-series><multi-armed-bandits>
Title: What reinforcement learning algorithm should I use for the following problem?
UserDisplayName: adamwest
ThreadText: # What reinforcement learning algorithm should I use for the following problem?
  
  (Asked by: username_0 on 2023-03-14)
  
  **Environment**
  
  I have a static timeseries environment meaning the environment is the same.  
  This problem is a multi armed bandit problem.
  
  | Time | t0 | t1 | t2 |
  | --- | --- | --- | --- |
  | State | s0 | s1 | s2 |
  | Score | 10 | 0.1 | 0.2 |
  | Class | 1 | 0 | 0 |
  
  Keep work = 1  
  Discard work = 0.1  
  Class = (Score>9)
  
  **Rewards**
  
  Reward = $\frac{Score}{Work}$
  
  I have various states in the dataset which are ordered chronologically and each state contains a reward from (0 -> inf)
  rewards come in 2 types a very small reward then very sparsely a high reward is given.
  
  To calculate the reward from a state is an expensive operation.
  The model can discard states to increase the reward density
  reward per work.
  
  For example lets consider 2 cases\
  
  * model 0: do nothing
  * model 1: optimal
  
  | Model | Metric | t0 | t1 | t2 |
  | --- | --- | --- | --- | --- |
  | model 0 | decision | keep | keep | keep |
  | model 0 | score | 10 | 0.1 | 0.2 |
  | model 0 | work | 1 | 1 | 1 |
  | model 1 | decision | keep | discard | discard |
  | model 1 | score | 10 | 0 | 0 |
  | model 1 | work | 1 | 0.1 | 0.1 |
  
  * model 0: reward = $\frac{10+0.1+0.2}{1+1+1}$ = 3.43
  * model 1: reward = $\frac{10+0+0}{1+0.1+0.1}$ = 8.33
  
  To avoid the model discarding most states to make work value low a penalty can be applied if the model discards too many states.
  this is due to if keep/discard work is decreased reward will increase.
  As $reward\propto\frac{1}{work}$
  
  **States**
  
  Each timestep state is related by being an input to a process and is monotonic
  The state is equal to the process with the timestamp as the input
  Sn = P(tn)
  
  **Domain Knowledge**
  
  1. P is unknown
  2. high rewards are very sparse >1:10000
  
  **Attempts so far/Ideas**  
  I have already tried using Bayesian optimisation on this problem with minimal luck.
  
  there are many different types of reinforcement learning algorithms.
  The ones applicable are
  
  1. output is classification(category) for keep/discard
     continuous state space -> discrete action space
  
  [stackoverflow](https://stackoverflow.com/questions/58550725/best-reinforcement-learning-algorithm-for-continuous-state-space-and-discrete-ac)
  
  * TD3
  * DDPG
  * SAC
  * PPO
  
  2. output is decision boundary(value) for classification
     continuous state space -> continuous action space
  3. output is feature(value) which is then classified
     continuous state space -> continuous action space
  
  [stackoverflow](https://stackoverflow.com/questions/7098625/how-can-i-apply-reinforcement-learning-to-continuous-action-spaces)
  [CACLA paper](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.75.7658&rep=rep1&type=pdf)
  
  * A2C
  * REINFORCE
  * CACLA
  
  **Questions**
  
  1. What reinforcement learning algorithm should I use?
  2. given the problem is 1,2 or 3 most sensible?
  3. any other considerations I should be aware of?
  
  Thanks
  
  **Tags:** reinforcement-learning, classification, time-series, multi-armed-bandits
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-08-06. Score: 0
  
  If your problem is a multi-armed bandit then the algorithms that you listed are not suited for the job. Those algorithms are for situations where the agent's actions has an influence on the subsequent states encountered. You should look for solutions designed for multi-armed bandit scenarios such as LinUCB.
  
  
  ---
  
  