Answers.Body: <p>In general, you need to actively explore the environment to gather data to train your Q network. However, especially in your self-driving car example, you might be looking for Batch RL. In Batch RL you start with a given, fixed dataset of transitions (state, action, reward, next state) and you learn a policy (or Q function) based on the dataset without exploring. The Batch-Constrained Q-learning algorithm (BCQ) is a good example of this.</p>
  <p>With regards to acquiring the data necessary for Batch RL, that's another story. I suppose one could collect data through sensors in a car, and label datapoints with rewards according to some reward function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T15:14:49.260Z
Answers.Text: Ah, you're right, my mistake. Edited my response to address this.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T12:32:26.987Z
Answers.LastActivityDate: 2020-07-06T15:14:30.390Z
Answers.LastEditDate: 2020-07-06T15:14:30.390Z
Answers.UserDisplayName: harwiltz
Body: <p>Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works.</p>
  <p>Is it that we are exploring the environment at the same time that we are training the Q network? If so, is there not a way to train the Q network before actually going out into the real world? And then, aren't there millions of possible states in the real world? So, how does RL or I guess the neural network generalize so that it can function during rush hour, empty roads, etc.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-05T23:40:24.863Z
Comments.Text: Do you know why you train the neural network in the first place? This the first question that needs to be answered to answer your main question, although, to be honest, I am not sure if you're asking if we can train the neural network in simulation first and then apply it to the actual real-world problem. Can you clarify this?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-05T23:58:09.777Z
Comments.Text: Yes. Training the neural network replaces the value policy iteration that is done with the Q table. And so the neural network is used to approximate a value function. Additionally, I am assuming that you can prob train in simulation first before applying it to the real-world problem (I believe MIT did that).
Comments.UserDisplayName: Huzaifah Shamim
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-06T00:04:46.490Z
Comments.Text: You're right about the neural network. But let me try to understand what your questions are: 1. "Can we train the Q network while exploring the environment?", and 2. "Can we train the Q network in simulation before training it in the real-world? If so, how do we deal with the discrepancies between the simulation and the real-world?" Are these your two questions? Anyway, these two questions are distinct enough to be asked in their separate post, so I suggest that you do that (i.e. create a new post where you ask your second question).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><q-learning><dqn>
Title: How does training for DQN work if messing up in the environment in costly?
UserDisplayName: Huzaifah Shamim
ThreadText: # How does training for DQN work if messing up in the environment in costly?
  
  (Asked by: username_0 on 2020-07-05)
  
  Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works.
  
  Is it that we are exploring the environment at the same time that we are training the Q network? If so, is there not a way to train the Q network before actually going out into the real world? And then, aren't there millions of possible states in the real world? So, how does RL or I guess the neural network generalize so that it can function during rush hour, empty roads, etc.
  
  **Tags:** reinforcement-learning, q-learning, dqn
  > **Comments:**
  > **username_1**: Do you know why you train the neural network in the first place? This the first question that needs to be answered to answer your main question, although, to be honest, I am not sure if you're asking if we can train the neural network in simulation first and then apply it to the actual real-world problem. Can you clarify this?  
  > **username_0**: Yes. Training the neural network replaces the value policy iteration that is done with the Q table. And so the neural network is used to approximate a value function. Additionally, I am assuming that you can prob train in simulation first before applying it to the real-world problem (I believe MIT did that).  
  > **username_1**: You're right about the neural network. But let me try to understand what your questions are: 1. "Can we train the Q network while exploring the environment?", and 2. "Can we train the Q network in simulation before training it in the real-world? If so, how do we deal with the discrepancies between the simulation and the real-world?" Are these your two questions? Anyway, these two questions are distinct enough to be asked in their separate post, so I suggest that you do that (i.e. create a new post where you ask your second question).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-06. Score: 1 | Accepted answer
  
  In general, you need to actively explore the environment to gather data to train your Q network. However, especially in your self-driving car example, you might be looking for Batch RL. In Batch RL you start with a given, fixed dataset of transitions (state, action, reward, next state) and you learn a policy (or Q function) based on the dataset without exploring. The Batch-Constrained Q-learning algorithm (BCQ) is a good example of this.
  
  With regards to acquiring the data necessary for Batch RL, that's another story. I suppose one could collect data through sensors in a car, and label datapoints with rewards according to some reward function.
  
  > **Comments:**
  > **username_2**: Ah, you're right, my mistake. Edited my response to address this.  
  
  ---
  
  