Answers.Body: <p>This is technically a partially observable MDP (POMDP), because some of the environment state is not visible to the agent. However, there are degrees of non-Markov states, and the environment you describe is pretty close to Markovian, with new details being revealed to the agent effectively at random as time progresses.</p>
  <p>So the good news is that a DQN-based agent should be able to cope. One caveat is that you should code the state so that it keeps a memory of the obstacles seen so far (the state is the agent's knowledge of the map and its position on it). Another is that there is no RL solution that will be optimal from the perspective of an observer that knows the location of obstacles in advance. The best any agent can do will be a policy that does well on average given the limited knowledge that the agent has. If a set of obstacles is generated that forms a dead end, the agent may waste time steps discovering that, whilst an observer who can see the obstacles could find a more optimal path that doesn't need to approach the blockage.</p>
  <p>There's a difference here between writing an agent to make use of knowledge you as designer has, as opposed to one that learns the hidden state details by itself. It <em>is</em> possible to design agents - DQN or otherwise - that figure out how to represent discoveries in POMDPs by themselves, but it's far easier to code an update to a state/memory yourself if you can and assuming your goal is to solve the environment, not to develop the most sophisticated general learning agent possible.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T14:10:39.157Z
Answers.Text: Thank you for your reply.  According to your explanation, the best way to proceed is to somehow encode this information related to the obstacles' locations into the code? Otherwise I should go with POMDPS? But then in testing the algorithm how can we add this extra knowledge? What if we make the problem really simple and not looking for the shortest path and instead focusing on fertilizing the target fruits and avoiding the obstacles. I think with my current state representation it cannot solve it. The current state includes the agent location and target fruit type.
Answers.UserDisplayName: Mamora
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T14:24:51.507Z
Answers.Text: @Mamora Make the state the agent's current knowledge of the map, including position of any obstacles it has discovered. If the space is very small, and number of obstacles fixed, you can do this by enumerating all possible states (see the Open AI Taxi environment for something similar - here's a worked solution: https://towardsdatascience.com/solving-the-taxi-environment-with-q-learning-a-tutorial-c76c22fc5d8f ). If the environment is larger, e.g. 16x16 grid with 10 or more movable obstacles, then a grid map "image" and CNN for the agent is the way to go.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T14:36:43.783Z
Answers.Text: The environment is quite big like 300*250 cells and one or two stationary obstacles but being located randomly.
Answers.UserDisplayName: Mamora
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T18:36:22.843Z
Answers.Text: So the obstacles are likely to be larger shapes, covering many pixels, and the agent only observe the parts of them that it is adjacent to? 300x250 is a bit large for a starting problem - it may take some effort to tune the CNN and representation. All of this is really different set of questions/answers though - I'd encourage you to try, and perhaps ask again if you get stuck. If I just predict what your challenges are in advance, you miss the chance to try stuff and learn about it even if it doesn't quite work. Plus I could well be wrong
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-05T08:11:19.760Z
Answers.Text: Thanks @Neil Slater for your answer and the effort. The obstacles are covering several pixels and the idea is that the agent only sees the adjacent parts of it. I am still stuck :D mostly for the state representation. I was thinking about how to proceed, one way is that because the fruits' location and the grid size are known in advance, I can give the robot the initial path to do the task and then whenever it encounters an obstacle it should learn to find a detour to finish the task. But then the state representation should be the whole field? It is too big and I want not to use CNN
Answers.UserDisplayName: Mamora
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-05T09:16:59.863Z
Answers.Text: @Mamora I have successfully coded an agent working in 2D space without using a grid or CNN. Remember you don't have to input the state representation directly to the neural network, you can use feature engineering to make things simpler. Istill have questions though, such as how the agent moves, and you haven't already asked about how to implement your state. Your question will need to give full information about the environment, not general issues you face or are not sure of
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-05T10:03:00.910Z
Answers.Text: The action is four direction which moves the agent one cell in the selected direction and it is deterministic. The state at the moment includes current location (cell coordinates) and % of the target fruits being fertilized. I also added the location of the obstacle, which is a rectangle (2 corners of the obstacle is given to the agent). But still it is not enough. There is not much left about the environment. Agent has to move around to collect the target fruit, not hitting the obstacle. Fertilizing wrong fruit also has punishment but way less than hitting obstacle.
Answers.UserDisplayName: Mamora
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-03T13:48:59.063Z
Answers.LastActivityDate: 2023-07-03T14:05:49.227Z
Answers.LastEditDate: 2023-07-03T14:05:49.227Z
Answers.UserDisplayName: Neil Slater
Body: <p>The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without hitting obstacles. Those locations that need to be visited are fruits that requires the agent to fertilize them. There are different fruits but the agent only have to fertilize specific types. The only available information to the agent is its current location and the type of the fruits to be fertilized. It is possible to know the location of the target crops but not the obstacle beforehand. The fixed information in each episode is the size of the field/grid-world, the type and locations of the fruits, the initial state of the agent.</p>
  <p>My question is  - can I train the agent with a DQN algorithm, <strong>avoiding obstacles</strong> and still finding the <strong>optimal path</strong> to finish the task?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-03T13:10:52.677Z
Comments.Text: Could you please give some more information regarding the rules that the obstacles follow? In your previous post you said that the agent *can* see obstacles, but only nearby ones - is that still the case? Also, it will make a difference to the best approach to use depending one when new locations for obstacles are generated, and what kinds of rule are used to generate those new locations. For example there is a big difference between random positions chosen once at the beginning of each episode, and positions that change on every time stamp. Also between completely random and items that move
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-03T13:27:13.790Z
Comments.Text: Obstacles are generated at the beginning of each episode and will not each time step. But it does not matter as the agent does not have access to the whole field. It is a simulation which is supposed to be realistic. But for simplicity we assume that the obstacles when they are generated they will not move for the whole episode. About the seeing part, in the training phase at the moment I assume that by giving high penalties, the agent will learn the location of the obstacles and/or not to hit it. But it is not in the state representation of the agent
Comments.UserDisplayName: Mamora
Tags: <reinforcement-learning><deep-rl><dqn><robotics>
Title: Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?
UserDisplayName: Mamora
ThreadText: # Can I train an agent with DQN, avoiding obstacles and still finding the optimal path to finish the task?
  
  (Asked by: username_0 on 2023-07-03)
  
  The agent is supposed to visit specific locations (which is also different each time) and it may encounters obstacles. The goal is to visit those locations with the shortest path possible without hitting obstacles. Those locations that need to be visited are fruits that requires the agent to fertilize them. There are different fruits but the agent only have to fertilize specific types. The only available information to the agent is its current location and the type of the fruits to be fertilized. It is possible to know the location of the target crops but not the obstacle beforehand. The fixed information in each episode is the size of the field/grid-world, the type and locations of the fruits, the initial state of the agent.
  
  My question is - can I train the agent with a DQN algorithm, **avoiding obstacles** and still finding the **optimal path** to finish the task?
  
  **Tags:** reinforcement-learning, deep-rl, dqn, robotics
  > **Comments:**
  > **username_1**: Could you please give some more information regarding the rules that the obstacles follow? In your previous post you said that the agent *can* see obstacles, but only nearby ones - is that still the case? Also, it will make a difference to the best approach to use depending one when new locations for obstacles are generated, and what kinds of rule are used to generate those new locations. For example there is a big difference between random positions chosen once at the beginning of each episode, and positions that change on every time stamp. Also between completely random and items that move  
  > **username_0**: Obstacles are generated at the beginning of each episode and will not each time step. But it does not matter as the agent does not have access to the whole field. It is a simulation which is supposed to be realistic. But for simplicity we assume that the obstacles when they are generated they will not move for the whole episode. About the seeing part, in the training phase at the moment I assume that by giving high penalties, the agent will learn the location of the obstacles and/or not to hit it. But it is not in the state representation of the agent  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-03. Score: 4
  
  This is technically a partially observable MDP (POMDP), because some of the environment state is not visible to the agent. However, there are degrees of non-Markov states, and the environment you describe is pretty close to Markovian, with new details being revealed to the agent effectively at random as time progresses.
  
  So the good news is that a DQN-based agent should be able to cope. One caveat is that you should code the state so that it keeps a memory of the obstacles seen so far (the state is the agent's knowledge of the map and its position on it). Another is that there is no RL solution that will be optimal from the perspective of an observer that knows the location of obstacles in advance. The best any agent can do will be a policy that does well on average given the limited knowledge that the agent has. If a set of obstacles is generated that forms a dead end, the agent may waste time steps discovering that, whilst an observer who can see the obstacles could find a more optimal path that doesn't need to approach the blockage.
  
  There's a difference here between writing an agent to make use of knowledge you as designer has, as opposed to one that learns the hidden state details by itself. It *is* possible to design agents - DQN or otherwise - that figure out how to represent discoveries in POMDPs by themselves, but it's far easier to code an update to a state/memory yourself if you can and assuming your goal is to solve the environment, not to develop the most sophisticated general learning agent possible.
  
  > **Comments:**
  > **username_0**: Thank you for your reply.  According to your explanation, the best way to proceed is to somehow encode this information related to the obstacles' locations into the code? Otherwise I should go with POMDPS? But then in testing the algorithm how can we add this extra knowledge? What if we make the problem really simple and not looking for the shortest path and instead focusing on fertilizing the target fruits and avoiding the obstacles. I think with my current state representation it cannot solve it. The current state includes the agent location and target fruit type.  
  > **username_1**: @Mamora Make the state the agent's current knowledge of the map, including position of any obstacles it has discovered. If the space is very small, and number of obstacles fixed, you can do this by enumerating all possible states (see the Open AI Taxi environment for something similar - here's a worked solution: https://towardsdatascience.com/solving-the-taxi-environment-with-q-learning-a-tutorial-c76c22fc5d8f ). If the environment is larger, e.g. 16x16 grid with 10 or more movable obstacles, then a grid map "image" and CNN for the agent is the way to go.  
  > **username_0**: The environment is quite big like 300*250 cells and one or two stationary obstacles but being located randomly.  
  > **username_1**: So the obstacles are likely to be larger shapes, covering many pixels, and the agent only observe the parts of them that it is adjacent to? 300x250 is a bit large for a starting problem - it may take some effort to tune the CNN and representation. All of this is really different set of questions/answers though - I'd encourage you to try, and perhaps ask again if you get stuck. If I just predict what your challenges are in advance, you miss the chance to try stuff and learn about it even if it doesn't quite work. Plus I could well be wrong  
  > **username_0**: Thanks @Neil Slater for your answer and the effort. The obstacles are covering several pixels and the idea is that the agent only sees the adjacent parts of it. I am still stuck :D mostly for the state representation. I was thinking about how to proceed, one way is that because the fruits' location and the grid size are known in advance, I can give the robot the initial path to do the task and then whenever it encounters an obstacle it should learn to find a detour to finish the task. But then the state representation should be the whole field? It is too big and I want not to use CNN  
  > **username_1**: @Mamora I have successfully coded an agent working in 2D space without using a grid or CNN. Remember you don't have to input the state representation directly to the neural network, you can use feature engineering to make things simpler. Istill have questions though, such as how the agent moves, and you haven't already asked about how to implement your state. Your question will need to give full information about the environment, not general issues you face or are not sure of  
  > **username_0**: The action is four direction which moves the agent one cell in the selected direction and it is deterministic. The state at the moment includes current location (cell coordinates) and % of the target fruits being fertilized. I also added the location of the obstacle, which is a rectangle (2 corners of the obstacle is given to the agent). But still it is not enough. There is not much left about the environment. Agent has to move around to collect the target fruit, not hitting the obstacle. Fertilizing wrong fruit also has punishment but way less than hitting obstacle.  
  
  ---
  
  