Answers.Body: <p>A quick search about reinforcement learning applied to video games will lead you to countless tutorials that describe exactly what you're asking for.</p>
  <p>With images the way to go is usually deep reinforcement learning. A convolutional neural network (or any other deep learning architecture) is used to process the image and compress it to a latent vector used as the &quot;environment&quot; seen by the agent.</p>
  <p>Given that you can then apply whatever reinforcement learning algorithm (sarsa, q-learning, monte carlo tree search, etc.) to train the agent itself on a specific task, in this case reaching the end pixels area without hitting obstacles.</p>
  <p>If you're familiar with python a good starting point is <a href="https://gym.openai.com/" rel="nofollow noreferrer">OpenAI Gym</a>, and I would say in particular the <a href="https://github.com/Kautenja/gym-super-mario-bros" rel="nofollow noreferrer">Super Mario</a> tutorial, since conceptually the game is basically the same as you're task of interest.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-19T08:15:03.307Z
Answers.Text: Alright, Thanks for the information. I shall look into this!!
Answers.UserDisplayName: Pinkman8144
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-18T16:56:18.097Z
Answers.LastActivityDate: 2021-11-18T16:56:18.097Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I have an image which consists of a start and an end point, the journey has some obstacles which have to be avoided.</p>
  <ul>
  <li>Is it possible to train an RL agent using such images to find the best path avoiding objects.</li>
  <li>Or what algorithm should be used in order to find the best path avoiding objects, where the input is an image.</li>
  </ul>
  <p>For example a picture of a person on a field track and there are obstacles in between from the start to the end point. I want to predict the series of actions that are required to reach the final position.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-18T16:23:27.007Z
Comments.Text: Could you give an example of the image? There are some good basic pathfinding algorithms that don't need machine learning, but that do need clean data to operation. If we are dealing with natural photos here, you may have a couple of choices, depending on what other data you have.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-18T16:26:17.597Z
Comments.Text: Also, for RL you will need well-defined actions, and be able to follow through with them to get feedback on state. For any machine learning at all, you will need some way to collect and/or assess data for correctness. It is not clear from your question what your experiment set up is, so which methods you have a good chance to apply. Could you please explain more about your task - e.g. if the person is in one place, and receives an instruction from the AI, what happens next, what feedback does AI get (in training, and for finished model or agent)
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-19T08:10:21.770Z
Comments.Text: Alright the main problem statement I have is an image of a road with potholes, if I could somehow add some variables to indicate the position of the potholes to the state space of the environment, would i be able to train an agent to avoid those pot holes using a generic action space of up, down, left and right.
Comments.UserDisplayName: Pinkman8144
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-19T08:12:50.377Z
Comments.Text: If possible I would like to map the image to a grid of a fixed size so as to map the position of the agent according to the grid co-ordinates which would act as the state space, presence of a pothole etc. The reward function would then be decided accordingly.
Comments.UserDisplayName: Pinkman8144
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-19T08:20:24.327Z
Comments.Text: From your description I think Edoardo's answer is a reasonable fit. I am still not 100% certain about the details - I am not sure if your output is a plan for multiple actions starting from a single photo, or something more interactive. Either way, you will need some way to simulate actions being taken in the environment - at the very least for training, and possibly for real if the problem is for planning multiple actions in advance based on a single photo.
Comments.UserDisplayName: Neil Slater
Tags: <machine-learning><reinforcement-learning><deep-learning><computer-vision><deep-rl>
Title: Is it possible to train an RL agent using images?
UserDisplayName: Pinkman8144
ThreadText: # Is it possible to train an RL agent using images?
  
  (Asked by: username_0 on 2021-11-18)
  
  I have an image which consists of a start and an end point, the journey has some obstacles which have to be avoided.
  
  * Is it possible to train an RL agent using such images to find the best path avoiding objects.
  * Or what algorithm should be used in order to find the best path avoiding objects, where the input is an image.
  
  For example a picture of a person on a field track and there are obstacles in between from the start to the end point. I want to predict the series of actions that are required to reach the final position.
  
  **Tags:** machine-learning, reinforcement-learning, deep-learning, computer-vision, deep-rl
  > **Comments:**
  > **username_1**: Could you give an example of the image? There are some good basic pathfinding algorithms that don't need machine learning, but that do need clean data to operation. If we are dealing with natural photos here, you may have a couple of choices, depending on what other data you have.  
  > **username_1**: Also, for RL you will need well-defined actions, and be able to follow through with them to get feedback on state. For any machine learning at all, you will need some way to collect and/or assess data for correctness. It is not clear from your question what your experiment set up is, so which methods you have a good chance to apply. Could you please explain more about your task - e.g. if the person is in one place, and receives an instruction from the AI, what happens next, what feedback does AI get (in training, and for finished model or agent)  
  > **username_0**: Alright the main problem statement I have is an image of a road with potholes, if I could somehow add some variables to indicate the position of the potholes to the state space of the environment, would i be able to train an agent to avoid those pot holes using a generic action space of up, down, left and right.  
  > **username_0**: If possible I would like to map the image to a grid of a fixed size so as to map the position of the agent according to the grid co-ordinates which would act as the state space, presence of a pothole etc. The reward function would then be decided accordingly.  
  > **username_1**: From your description I think Edoardo's answer is a reasonable fit. I am still not 100% certain about the details - I am not sure if your output is a plan for multiple actions starting from a single photo, or something more interactive. Either way, you will need some way to simulate actions being taken in the environment - at the very least for training, and possibly for real if the problem is for planning multiple actions in advance based on a single photo.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-11-18. Score: 1
  
  A quick search about reinforcement learning applied to video games will lead you to countless tutorials that describe exactly what you're asking for.
  
  With images the way to go is usually deep reinforcement learning. A convolutional neural network (or any other deep learning architecture) is used to process the image and compress it to a latent vector used as the "environment" seen by the agent.
  
  Given that you can then apply whatever reinforcement learning algorithm (sarsa, q-learning, monte carlo tree search, etc.) to train the agent itself on a specific task, in this case reaching the end pixels area without hitting obstacles.
  
  If you're familiar with python a good starting point is [OpenAI Gym](https://gym.openai.com/), and I would say in particular the [Super Mario](https://github.com/Kautenja/gym-super-mario-bros) tutorial, since conceptually the game is basically the same as you're task of interest.
  
  > **Comments:**
  > **username_0**: Alright, Thanks for the information. I shall look into this!!  
  
  ---
  
  