Body: <p>I'm trying to train an agent on a self-written 2d env, and it just doesn't converge to the solution.
  <a href="https://i.sstatic.net/C7cOn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/C7cOn.png" alt="The environment" /></a></p>
  <p>It is basically a 2d game where you have to move a small circle around the screen and try to avoid collisions with randomly moving &quot;enemy&quot; circles and the edge of the screen. The positions of the enemies are initialized randomly, at a minimum distance of 2 diameters from the enemy.
  The player circle has <span class="math-container">$n$</span> sensors (lasers) that measure the distance and speed of the closest object found.</p>
  <p>The <em>observation space</em> is continuous and is made of concatenated distances and speeds, hence has the dimension of <span class="math-container">$\mathbb{R}^{n * 3}$</span>.<br />
  I scale the distances by the length of the screen diagonal.</p>
  <p>The <em>action space</em> is discrete (multidiscrete in my implementation) <span class="math-container">$[dx, dy] \in \{-1, 0, 1\}$</span></p>
  <p>The <em>reward</em> is +1 for every game step made without collisions.</p>
  <p>I use PPO implementation from Stable Baselines, but the return variance just gets bigger over the training. In addition to that, the agent hasn't run away from the enemies even once. I tried even setting the negative reward, to test if he can learn the <em><strong>suicide</strong></em> behavior, but no results either.</p>
  <p>I thought maybe it's just possible for some degenerate policies like going to the corner of the screen and staying there to gain big returns, and that jeopardizes the training. Then I increased the number of enemies, thinking that it will enforce the agent to learn actually to avoid the enemies, but it didn't work as well.</p>
  <p>I'm really out of ideas at this point and would appreciate some help on debugging this.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-16T12:32:49.757Z
Comments.Text: What diagnostics have you already done? What behaviour are you observing for the failing agent? Do you have unit testing around the base environment, so you know for sure that the sensors are working as intended? Are the sensors wide - so anything within the sensor angle is detected? Or are they narrow and might miss unaligned enemies?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-16T19:18:31.390Z
Comments.Text: Could you please elaborate on "diagnostics"?
  Right now he moves to the lower part of the screen and then moves up and down for some time until he hits the enemy. I'd say the best he's got so far was that he learned not to collide with the border. This was observed in different policies.
  The sensors have been tested and are working properly.
Comments.UserDisplayName: debrises
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-16T19:24:52.610Z
Comments.Text: Sensors are basically just rays. They can miss some enemies that are far away, but the problem is that the agent doesn't even learn to flee the enemies standing very close.
Comments.UserDisplayName: debrises
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-16T19:25:04.110Z
Comments.Text: By diagnostics, I mean collecting data about training progress and outcomes in detail. E.g. graphing loss, reward, also being able to spot check an episode and view  behaviour with visualisation of the environment. I was wondering about this because of your concern over "degenerate policies" - have you actually observed the agent behaving according to such policies?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-16T19:30:52.213Z
Comments.Text: Have you calculated the coverage of the sensors - what percentage of enemies are detected at any given range? WIthout knowing more, that would be my first concern, that there was too much hidden state due to small movements making enemies visible then invisible again. Unless the agent has a memory this could be a major problem.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-02T18:19:45.740Z
Comments.Text: Thanks a lot for the advice! I found a bug in how my observation was made and it started working. What helped me was that I visualized the collected state, which involved inverse transformation and a step-by-step review of my env. As a side note, the variance was still getting over the training, but it is related to the env dynamics.
Comments.UserDisplayName: debrises
Tags: <reinforcement-learning><training><convergence><on-policy-methods><online-learning>
Title: How to fix high variance of the returns on a 2d env?
UserDisplayName: debrises
ThreadText: # How to fix high variance of the returns on a 2d env?
  
  (Asked by: username_0 on 2021-11-16)
  
  I'm trying to train an agent on a self-written 2d env, and it just doesn't converge to the solution.
  [![The environment](https://i.sstatic.net/C7cOn.png)](https://i.sstatic.net/C7cOn.png)
  
  It is basically a 2d game where you have to move a small circle around the screen and try to avoid collisions with randomly moving "enemy" circles and the edge of the screen. The positions of the enemies are initialized randomly, at a minimum distance of 2 diameters from the enemy.
  The player circle has $n$ sensors (lasers) that measure the distance and speed of the closest object found.
  
  The *observation space* is continuous and is made of concatenated distances and speeds, hence has the dimension of $\mathbb{R}^{n \* 3}$.  
  I scale the distances by the length of the screen diagonal.
  
  The *action space* is discrete (multidiscrete in my implementation) $[dx, dy] \in \{-1, 0, 1\}$
  
  The *reward* is +1 for every game step made without collisions.
  
  I use PPO implementation from Stable Baselines, but the return variance just gets bigger over the training. In addition to that, the agent hasn't run away from the enemies even once. I tried even setting the negative reward, to test if he can learn the ***suicide*** behavior, but no results either.
  
  I thought maybe it's just possible for some degenerate policies like going to the corner of the screen and staying there to gain big returns, and that jeopardizes the training. Then I increased the number of enemies, thinking that it will enforce the agent to learn actually to avoid the enemies, but it didn't work as well.
  
  I'm really out of ideas at this point and would appreciate some help on debugging this.
  
  **Tags:** reinforcement-learning, training, convergence, on-policy-methods, online-learning
  > **Comments:**
  > **username_1**: What diagnostics have you already done? What behaviour are you observing for the failing agent? Do you have unit testing around the base environment, so you know for sure that the sensors are working as intended? Are the sensors wide - so anything within the sensor angle is detected? Or are they narrow and might miss unaligned enemies?  
  > **username_0**: Could you please elaborate on "diagnostics"? Right now he moves to the lower part of the screen and then moves up and down for some time until he hits the enemy. I'd say the best he's got so far was that he learned not to collide with the border. This was observed in different policies. The sensors have been tested and are working properly.  
  > **username_0**: Sensors are basically just rays. They can miss some enemies that are far away, but the problem is that the agent doesn't even learn to flee the enemies standing very close.  
  > **username_1**: By diagnostics, I mean collecting data about training progress and outcomes in detail. E.g. graphing loss, reward, also being able to spot check an episode and view  behaviour with visualisation of the environment. I was wondering about this because of your concern over "degenerate policies" - have you actually observed the agent behaving according to such policies?  
  > **username_1**: Have you calculated the coverage of the sensors - what percentage of enemies are detected at any given range? WIthout knowing more, that would be my first concern, that there was too much hidden state due to small movements making enemies visible then invisible again. Unless the agent has a memory this could be a major problem.  
  > **username_0**: Thanks a lot for the advice! I found a bug in how my observation was made and it started working. What helped me was that I visualized the collected state, which involved inverse transformation and a step-by-step review of my env. As a side note, the variance was still getting over the training, but it is related to the env dynamics.  
  
  ---
  
  ## 0 answers
  
  