Answers.Body: <h2>TLDR: Simplify your agent.</h2>
  <h4>Context:</h4>
  <p>As you've noticed, it's not a hard game and it does not require a complex policy.</p>
  <p>You'd need a single neuron to solve it perfectly: <code>1 if (state &lt; 0.78) else 0</code></p>
  <h3>Problem:</h3>
  <p>However, the default agent comes with a big neural network (I believe it's a 2 layered 64 fully connected neural network). So you have <strong>thousands</strong> of parameters trying to solve a simple problem. And that's why it takes <strong>thousands</strong> of time-steps.</p>
  <h3>Solution:</h3>
  <p>So first thing I'd recommend is to use a simpler policy model.</p>
  <p>If it doesn't completely solve the problem, then simplify your model by diving deeper on your agent and disabling the functions you think aren't helpful.
  <a href="https://tensorforce.readthedocs.io/en/latest/modules/policies.html" rel="nofollow noreferrer">https://tensorforce.readthedocs.io/en/latest/modules/policies.html</a></p>
  <p>Lastly, if you really need to nail it, you can use some meta-learning for tuning the hyper-parameters.</p>
  <hr />
  <p><em>Edit: adding Meta-Learning</em></p>
  <p>Meta-Learning is a broad concept of using machine learning for setting up your machine learning architecture and/or hyperparameters. It's an automation of your trial and error process, exploring and exploiting different configurations in a search for a good model.</p>
  <p>A related concept is auto-ML.</p>
  <p>Keep in mind that:</p>
  <ol>
  <li>It's a very extensive computational process, since you need to train and evaluate thousand or millions of models.</li>
  <li>You'll need to explicitly define your definition of a good model:
  Will you reward it to be simple? Are you looking for fast training speed? Does in need to be light for deploying? Or you want a huge, complex, but accurate model?</li>
  </ol>
  <p>In your case, if the training time is too big, you could limit the training time, so you'll find the best model that can be trained in a fixed short time window. And once again, keep in mind it will take <strong>way more time</strong> to find a good model this way, than just selecting a not ideal architecture and train it extensively.</p>
  <p>Here is a <a href="http://youtube.com/watch?v=2z0ofe2lpz4" rel="nofollow noreferrer">Siraj's video</a> talking about the general concept.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T09:54:57.077Z
Answers.Text: This is a valid point, indeed. Can you please explain more about the meta-learning, as it seems to be directly relevant to the main question. Thank you
Answers.UserDisplayName: penkovsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T09:58:16.850Z
Answers.Text: I actually tried to reduce the number of layers to a single hidden layer and reducing the number of neurons. Doesn't help much.
Answers.UserDisplayName: penkovsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T10:02:18.937Z
Answers.Text: For instance, after training a single layer of 16 neurons (with different number of timesteps and episodes), during evaluation, the agent dies after about 10 timesteps.
Answers.UserDisplayName: penkovsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-13T20:36:00.400Z
Answers.Text: I've updated the orignal post with meta-learning brief concepts.
Answers.UserDisplayName: Andre Goulart
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T11:07:48.193Z
Answers.Text: I see, by meta-learning you only mean hyper-parameter tuning. Thank you for your effort to try to answer the question. I still cannot accept this answer as my main question was: "How to reduce the number of trials (i.e. the number of episodes) before the agent learns?" Since effectively hyperparameter tuning does not directly address this question. I think that the closest solution would be learning by observation/behavior cloning. It would be nice if anyone with experience could confirm that.
Answers.UserDisplayName: penkovsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T11:14:18.840Z
Answers.Text: As an appreciation of your effort, I decided to award the reputation bounty anyway. Thank you.
Answers.UserDisplayName: penkovsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-16T18:23:53.267Z
Answers.Text: Yes. But keep in mind that not only the hyper-parameters can be tuned to reduce the number of trials, but you can also use it meta-learning to set the whole architecture.
  So I agree I did not give you a straightforward answer, but that's a valid method to find it yourself, through very an intensive computational process.
  Either way, thanks for the bounty! :)
Answers.UserDisplayName: Andre Goulart
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-18T07:33:43.653Z
Answers.Text: Tuning the hyper-parameters will not work as every environment run takes a significant time (what I presented above was an oversimplified version, whereas in my case we have to deal with realworld equipment). The problem I want to solve is to train the agent without waiting two years until it learns :) So hyper-parameter tuning is definitely not the way to go.
Answers.UserDisplayName: penkovsky
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-12T07:02:03.807Z
Answers.LastActivityDate: 2021-08-13T20:41:28.250Z
Answers.LastEditDate: 2021-08-13T20:41:28.250Z
Answers.UserDisplayName: Andre Goulart
Body: <p>The initial environment state is 0.25. Each time step the agent performs a discrete <code>action</code> of <code>0</code> or <code>1</code>. If action is <code>1</code>, then the new state will be <code>state + 0.1</code>. If action is <code>0</code>, the new state will be <code>state - random() * 0.2</code>. The reward is <code>state - 0.5</code>, however if <code>state &gt; 0.98</code> (or <code>state &lt; 0</code>) the agent dies (with no reward).</p>
  <p>First question: How do I teach the agent not to be too greedy? How to verify that the agent learned?</p>
  <p>Main question: <strong>How to reduce the number of trials (i.e. the number of episodes) before the agent learns?</strong></p>
  <p>I would also appreciate any relevant references.</p>
  <p>Here is the <a href="https://gist.githubusercontent.com/masterdezign/c82391618bd73aa67e03988f8c4b518c/raw/cd261faab14c05c13601cc4acde0be404079fa80/simple.py" rel="nofollow noreferrer">environment</a> and here is <a href="https://gist.githubusercontent.com/masterdezign/c82391618bd73aa67e03988f8c4b518c/raw/cd261faab14c05c13601cc4acde0be404079fa80/run.py" rel="nofollow noreferrer">what I tried</a>.</p>
  <p><a href="https://gist.github.com/masterdezign/c82391618bd73aa67e03988f8c4b518c" rel="nofollow noreferrer">It works, however</a>:</p>
  <ol>
  <li><p>It took 1000 episodes of max 2000 timesteps, which is unacceptable
  for me (I wish to drastically reduce the number of episodes and timesteps).</p>
  </li>
  <li><p>The behavior is far from optimal. Ideally, the agent should
  choose action <code>0</code> only if the state is larger than 0.88 (or something below that and within a small interval such as 0.01). [Edit] However, the threshold is 0.75, that forces the agent to choose <code>0</code> even if it could safely choose <code>1</code>, e.g. following 0.8 -&gt; 0.76 -&gt; 0.75 -&gt; 0.74 trajectory before choosing
  <code>1</code> again.</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-10T12:30:23.920Z
Comments.Text: Some ideas to reduce the amount of direct interaction with an environment: behavior cloning, off-policy algorithms. By observing, rather than interacting, these algorithms  should partially alleviate the amount of trial and error.
Comments.UserDisplayName: penkovsky
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-10T12:37:00.870Z
Comments.Text: Which is great if it is extremely costly to simulate an environment.
Comments.UserDisplayName: penkovsky
Tags: <reinforcement-learning><deep-rl><exploration-exploitation-tradeoff><exploration-strategies>
Title: How to reduce the number of episodes before the agent learns in this game?
UserDisplayName: penkovsky
ThreadText: # How to reduce the number of episodes before the agent learns in this game?
  
  (Asked by: username_0 on 2021-08-07)
  
  The initial environment state is 0.25. Each time step the agent performs a discrete `action` of `0` or `1`. If action is `1`, then the new state will be `state + 0.1`. If action is `0`, the new state will be `state - random() * 0.2`. The reward is `state - 0.5`, however if `state > 0.98` (or `state < 0`) the agent dies (with no reward).
  
  First question: How do I teach the agent not to be too greedy? How to verify that the agent learned?
  
  Main question: **How to reduce the number of trials (i.e. the number of episodes) before the agent learns?**
  
  I would also appreciate any relevant references.
  
  Here is the [environment](https://gist.githubusercontent.com/masterdezign/c82391618bd73aa67e03988f8c4b518c/raw/cd261faab14c05c13601cc4acde0be404079fa80/simple.py) and here is [what I tried](https://gist.githubusercontent.com/masterdezign/c82391618bd73aa67e03988f8c4b518c/raw/cd261faab14c05c13601cc4acde0be404079fa80/run.py).
  
  [It works, however](https://gist.github.com/masterdezign/c82391618bd73aa67e03988f8c4b518c):
  
  1. It took 1000 episodes of max 2000 timesteps, which is unacceptable
     for me (I wish to drastically reduce the number of episodes and timesteps).
  2. The behavior is far from optimal. Ideally, the agent should
     choose action `0` only if the state is larger than 0.88 (or something below that and within a small interval such as 0.01). [Edit] However, the threshold is 0.75, that forces the agent to choose `0` even if it could safely choose `1`, e.g. following 0.8 -> 0.76 -> 0.75 -> 0.74 trajectory before choosing
     `1` again.
  
  **Tags:** reinforcement-learning, deep-rl, exploration-exploitation-tradeoff, exploration-strategies
  
  > **Comments:**
  > **username_0**: Some ideas to reduce the amount of direct interaction with an environment: behavior cloning, off-policy algorithms. By observing, rather than interacting, these algorithms  should partially alleviate the amount of trial and error.  
  > **username_0**: Which is great if it is extremely costly to simulate an environment.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-12. Score: 1
  
  ## TLDR: Simplify your agent.
  
  #### Context:
  
  As you've noticed, it's not a hard game and it does not require a complex policy.
  
  You'd need a single neuron to solve it perfectly: `1 if (state < 0.78) else 0`
  
  ### Problem:
  
  However, the default agent comes with a big neural network (I believe it's a 2 layered 64 fully connected neural network). So you have **thousands** of parameters trying to solve a simple problem. And that's why it takes **thousands** of time-steps.
  
  ### Solution:
  
  So first thing I'd recommend is to use a simpler policy model.
  
  If it doesn't completely solve the problem, then simplify your model by diving deeper on your agent and disabling the functions you think aren't helpful.
  <https://tensorforce.readthedocs.io/en/latest/modules/policies.html>
  
  Lastly, if you really need to nail it, you can use some meta-learning for tuning the hyper-parameters.
  
  ---
  
  *Edit: adding Meta-Learning*
  
  Meta-Learning is a broad concept of using machine learning for setting up your machine learning architecture and/or hyperparameters. It's an automation of your trial and error process, exploring and exploiting different configurations in a search for a good model.
  
  A related concept is auto-ML.
  
  Keep in mind that:
  
  1. It's a very extensive computational process, since you need to train and evaluate thousand or millions of models.
  2. You'll need to explicitly define your definition of a good model:
     Will you reward it to be simple? Are you looking for fast training speed? Does in need to be light for deploying? Or you want a huge, complex, but accurate model?
  
  In your case, if the training time is too big, you could limit the training time, so you'll find the best model that can be trained in a fixed short time window. And once again, keep in mind it will take **way more time** to find a good model this way, than just selecting a not ideal architecture and train it extensively.
  
  Here is a [Siraj's video](http://youtube.com/watch?v=2z0ofe2lpz4) talking about the general concept.
  
  > **Comments:**
  > **username_0**: This is a valid point, indeed. Can you please explain more about the meta-learning, as it seems to be directly relevant to the main question. Thank you  
  > **username_0**: I actually tried to reduce the number of layers to a single hidden layer and reducing the number of neurons. Doesn't help much.  
  > **username_0**: For instance, after training a single layer of 16 neurons (with different number of timesteps and episodes), during evaluation, the agent dies after about 10 timesteps.  
  > **username_1**: I've updated the orignal post with meta-learning brief concepts.  
  > **username_0**: I see, by meta-learning you only mean hyper-parameter tuning. Thank you for your effort to try to answer the question. I still cannot accept this answer as my main question was: "How to reduce the number of trials (i.e. the number of episodes) before the agent learns?" Since effectively hyperparameter tuning does not directly address this question. I think that the closest solution would be learning by observation/behavior cloning. It would be nice if anyone with experience could confirm that.  
  > **username_0**: As an appreciation of your effort, I decided to award the reputation bounty anyway. Thank you.  
  > **username_1**: Yes. But keep in mind that not only the hyper-parameters can be tuned to reduce the number of trials, but you can also use it meta-learning to set the whole architecture. So I agree I did not give you a straightforward answer, but that's a valid method to find it yourself, through very an intensive computational process. Either way, thanks for the bounty! :)  
  > **username_0**: Tuning the hyper-parameters will not work as every environment run takes a significant time (what I presented above was an oversimplified version, whereas in my case we have to deal with realworld equipment). The problem I want to solve is to train the agent without waiting two years until it learns :) So hyper-parameter tuning is definitely not the way to go.  
  
  ---
  
  