Answers.Body: <p>My guess is that you haven't trained long enough, but there are things that can be done to possibly accelerate learning.</p>
  <p>It depends on what you want the policy to do in the final version. If you want it to be able to be spawned at a random position on the map and park in some other (random) position on the map, then that is how you should train it.</p>
  <p>Training in steps can be useful, for example training with a fixed start and terminal point, then randomize the parking spot, then randomize the starting position. In general, giving the agent fully randomization will take longer than setting multiple, sequential goals.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-04T19:16:03.603Z
Answers.Text: Tnx for the tip. I will try training the model for longer. The reason I wasn't sure before was that when I tried it (randomizing spawn location every N step), whenever the agent changed position, the loss exploded to an absurd number like 4mil+, it did go down eventually but didn't learn anything. I will try increasing the number of total steps and a reasonable amount of steps before randomizing the spawn location.
Answers.UserDisplayName: ashesofphoenix
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T12:33:25.593Z
Answers.Text: It could be a combination of more training and the right parameters. When you expand the observation space, the problem becomes more difficult and the algorithms needs to try harder to learn and explore. The main thing I have learned in ML is it is more like an experimental science, rather than a logical math-based problem. Even though the learning is built on some basic math results.
Answers.UserDisplayName: Elfurd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-04T18:18:11.347Z
Answers.LastActivityDate: 2022-03-04T18:18:11.347Z
Answers.UserDisplayName: Elfurd
Answers.Body: <p>I am correct in my understanding that you only provide the agent with the state of the car, i.e. a global x and y position, its angle, velocity, and steering angle?</p>
  <p>How does the agent know that it is coming closer to the goal if it is not provided with information about where the goal is? Without this observation of the goal, the agent is operating blindly. That explains why it is so difficult for the agent to reach the goal and impossible when you randomize the starting position.</p>
  <p>If my assumptions are correct, the agent takes random actions which are unlikely to reach the goal, but due to the law of large numbers after enough episodes, the agent will reach the goal at random and it can learn to remember this path if given enough reward. But if you then randomize the starting position the agent cannot apply the knowledge it has learned previously because the sequence of actions to reach the goal would now be different. Essentially, there is no correlation between what goal you want the agent to achieve and your state and action space.</p>
  <p>To circumvent this problem, I suggest you add additional state information, here are a few suggestions:</p>
  <ul>
  <li>The global x and y position of the goal</li>
  <li>A distance measure measuring the distance from the agent and to the goal. Either the <a href="https://en.wikipedia.org/wiki/Euclidean_distance" rel="noreferrer">Euclidean distance</a> or <a href="https://en.wikipedia.org/wiki/Taxicab_geometry" rel="noreferrer">Manhattan distance</a>.</li>
  <li>Both of the above</li>
  </ul>
  <hr />
  <p>I also support the suggestion of <a href="https://ai.stackexchange.com/users/52966/elfurd">Elfurd</a>: &quot;<em>Training in steps can be useful</em>&quot;. This is called <a href="https://ronan.collobert.com/pub/matos/2009_curriculum_icml.pdf" rel="noreferrer">curriculum learning</a> and the idea is to present easier training examples to the agent at the beginning of training and steadily increase the difficulty of the environment. In turn, the agent will reach the goal in the easier environments, obtain some reward, and learn. It can then apply what it has learned in the more advanced environments once it progresses through the curriculum.</p>
  <p>In your environment, this could be as simple as decreasing the size of the gird world in early training. Or you could spawn the agent close to the goal so that the agent is more likely to reach the goal with just a few random actions, alternatively, you can also randomize the goal close to the starting position of the agent if it has to start from a specified position and then increase the distance to where the goal is sampled.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-05T09:08:03.420Z
Answers.Text: From the description it looks like the parking space is always at 0,0
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-05T09:45:36.787Z
Answers.Text: Yeah, it seems so. But I imagine they want to be able to start from any arbitrary position and end at any arbitrary position in the environment.
Answers.UserDisplayName: Lars
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-07T20:59:35.393Z
Answers.Text: Thank you. As you mentioned, I was missing additional state information, namely the distance from the agent to the goal.  With this agent is able to learn a good policy faster. I will also try experimenting with curriculum learning to see if the agent will have an easier and faster time learning a good policy.
Answers.UserDisplayName: ashesofphoenix
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-05T07:50:27.940Z
Answers.LastActivityDate: 2022-03-05T07:50:27.940Z
Answers.UserDisplayName: Lars
Body: <p>I am new to RL and I'm currently working on implementing a DQN and DDPG agent for a 2D car parking environment. I want to train my agent so that it can successfully traverse the env and park in the designated goal in the middle.</p>
  <p>So, my question is: <strong>what are the best practices when training an agent for a changing environment?</strong></p>
  <p>In my case, my goal is that a car can randomly spawn every episode anywhere in the dark grey-ish area and always successfully parks in the middle. My problem, in this case, is that if I for example train the agent from only one specified location, it usually won't know how to perform if it's spawned somewhere else.</p>
  <p>I also tried making it so that the car starting location gets randomly updated every N step, but unfortunately came to no success.</p>
  <p>It may be possible that I've not trained for long enough and with a sufficient number of steps in between the &quot;position resets&quot;, but I still want to ask if there are any general practices in the cases like this?</p>
  <p><img src="https://i.sstatic.net/5zVhC.png" alt="Seen here" /></p>
  
Tags: <reinforcement-learning><training><dqn><environment><reward-shaping>
Title: How to deal with changing environment in reinforcement learning
UserDisplayName: ashesofphoenix
ThreadText: # How to deal with changing environment in reinforcement learning
  
  (Asked by: username_0 on 2022-03-04)
  
  I am new to RL and I'm currently working on implementing a DQN and DDPG agent for a 2D car parking environment. I want to train my agent so that it can successfully traverse the env and park in the designated goal in the middle.
  
  So, my question is: **what are the best practices when training an agent for a changing environment?**
  
  In my case, my goal is that a car can randomly spawn every episode anywhere in the dark grey-ish area and always successfully parks in the middle. My problem, in this case, is that if I for example train the agent from only one specified location, it usually won't know how to perform if it's spawned somewhere else.
  
  I also tried making it so that the car starting location gets randomly updated every N step, but unfortunately came to no success.
  
  It may be possible that I've not trained for long enough and with a sufficient number of steps in between the "position resets", but I still want to ask if there are any general practices in the cases like this?
  
  ![Seen here](https://i.sstatic.net/5zVhC.png)
  
  **Tags:** reinforcement-learning, training, dqn, environment, reward-shaping
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-03-05. Score: 5 | Accepted answer
  
  I am correct in my understanding that you only provide the agent with the state of the car, i.e. a global x and y position, its angle, velocity, and steering angle?
  
  How does the agent know that it is coming closer to the goal if it is not provided with information about where the goal is? Without this observation of the goal, the agent is operating blindly. That explains why it is so difficult for the agent to reach the goal and impossible when you randomize the starting position.
  
  If my assumptions are correct, the agent takes random actions which are unlikely to reach the goal, but due to the law of large numbers after enough episodes, the agent will reach the goal at random and it can learn to remember this path if given enough reward. But if you then randomize the starting position the agent cannot apply the knowledge it has learned previously because the sequence of actions to reach the goal would now be different. Essentially, there is no correlation between what goal you want the agent to achieve and your state and action space.
  
  To circumvent this problem, I suggest you add additional state information, here are a few suggestions:
  
  * The global x and y position of the goal
  * A distance measure measuring the distance from the agent and to the goal. Either the [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance) or [Manhattan distance](https://en.wikipedia.org/wiki/Taxicab_geometry).
  * Both of the above
  
  ---
  
  I also support the suggestion of [username_1](https://ai.stackexchange.com/users/52966/username_1): "*Training in steps can be useful*". This is called [curriculum learning](https://ronan.collobert.com/pub/matos/2009_curriculum_icml.pdf) and the idea is to present easier training examples to the agent at the beginning of training and steadily increase the difficulty of the environment. In turn, the agent will reach the goal in the easier environments, obtain some reward, and learn. It can then apply what it has learned in the more advanced environments once it progresses through the curriculum.
  
  In your environment, this could be as simple as decreasing the size of the gird world in early training. Or you could spawn the agent close to the goal so that the agent is more likely to reach the goal with just a few random actions, alternatively, you can also randomize the goal close to the starting position of the agent if it has to start from a specified position and then increase the distance to where the goal is sampled.
  
  > **Comments:**
  > **username_3**: From the description it looks like the parking space is always at 0,0  
  > **username_2**: Yeah, it seems so. But I imagine they want to be able to start from any arbitrary position and end at any arbitrary position in the environment.  
  > **username_0**: Thank you. As you mentioned, I was missing additional state information, namely the distance from the agent to the goal.  With this agent is able to learn a good policy faster. I will also try experimenting with curriculum learning to see if the agent will have an easier and faster time learning a good policy.  
  
  ---
  
  ### Answer by username_1 on 2022-03-04. Score: 1
  
  My guess is that you haven't trained long enough, but there are things that can be done to possibly accelerate learning.
  
  It depends on what you want the policy to do in the final version. If you want it to be able to be spawned at a random position on the map and park in some other (random) position on the map, then that is how you should train it.
  
  Training in steps can be useful, for example training with a fixed start and terminal point, then randomize the parking spot, then randomize the starting position. In general, giving the agent fully randomization will take longer than setting multiple, sequential goals.
  
  > **Comments:**
  > **username_0**: Tnx for the tip. I will try training the model for longer. The reason I wasn't sure before was that when I tried it (randomizing spawn location every N step), whenever the agent changed position, the loss exploded to an absurd number like 4mil+, it did go down eventually but didn't learn anything. I will try increasing the number of total steps and a reasonable amount of steps before randomizing the spawn location.  
  > **username_1**: It could be a combination of more training and the right parameters. When you expand the observation space, the problem becomes more difficult and the algorithms needs to try harder to learn and explore. The main thing I have learned in ML is it is more like an experimental science, rather than a logical math-based problem. Even though the learning is built on some basic math results.  
  
  ---
  
  