<p>In general, you need to actively explore the environment to gather data to train your Q network. However, especially in your self-driving car example, you might be looking for Batch RL. In Batch RL you start with a given, fixed dataset of transitions (state, action, reward, next state) and you learn a policy (or Q function) based on the dataset without exploring. The Batch-Constrained Q-learning algorithm (BCQ) is a good example of this.</p>
<p>With regards to acquiring the data necessary for Batch RL, that's another story. I suppose one could collect data through sensors in a car, and label datapoints with rewards according to some reward function.</p>

CC BY-SA 4.0
2020-07-06T15:14:49.260Z
Ah, you're right, my mistake. Edited my response to address this.
harwiltz
CC BY-SA 4.0
2020-07-06T12:32:26.987Z
2020-07-06T15:14:30.390Z
2020-07-06T15:14:30.390Z
harwiltz
<p>Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works.</p>
<p>Is it that we are exploring the environment at the same time that we are training the Q network? If so, is there not a way to train the Q network before actually going out into the real world? And then, aren't there millions of possible states in the real world? So, how does RL or I guess the neural network generalize so that it can function during rush hour, empty roads, etc.</p>

CC BY-SA 4.0
2020-07-05T23:40:24.863Z
Do you know why you train the neural network in the first place? This the first question that needs to be answered to answer your main question, although, to be honest, I am not sure if you're asking if we can train the neural network in simulation first and then apply it to the actual real-world problem. Can you clarify this?
nbro
CC BY-SA 4.0
2020-07-05T23:58:09.777Z
Yes. Training the neural network replaces the value policy iteration that is done with the Q table. And so the neural network is used to approximate a value function. Additionally, I am assuming that you can prob train in simulation first before applying it to the real-world problem (I believe MIT did that).
Huzaifah Shamim
CC BY-SA 4.0
2020-07-06T00:04:46.490Z
You're right about the neural network. But let me try to understand what your questions are: 1. "Can we train the Q network while exploring the environment?", and 2. "Can we train the Q network in simulation before training it in the real-world? If so, how do we deal with the discrepancies between the simulation and the real-world?" Are these your two questions? Anyway, these two questions are distinct enough to be asked in their separate post, so I suggest that you do that (i.e. create a new post where you ask your second question).
nbro
<reinforcement-learning><q-learning><dqn>
How does training for DQN work if messing up in the environment in costly?
Huzaifah Shamim
# How does training for DQN work if messing up in the environment in costly?

(Asked by: username_0 on 2020-07-05)

Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works.

Is it that we are exploring the environment at the same time that we are training the Q network? If so, is there not a way to train the Q network before actually going out into the real world? And then, aren't there millions of possible states in the real world? So, how does RL or I guess the neural network generalize so that it can function during rush hour, empty roads, etc.

**Tags:** reinforcement-learning, q-learning, dqn

> **Comments:**
> **username_1**: Do you know why you train the neural network in the first place? This the first question that needs to be answered to answer your main question, although, to be honest, I am not sure if you're asking if we can train the neural network in simulation first and then apply it to the actual real-world problem. Can you clarify this?  
> **username_0**: Yes. Training the neural network replaces the value policy iteration that is done with the Q table. And so the neural network is used to approximate a value function. Additionally, I am assuming that you can prob train in simulation first before applying it to the real-world problem (I believe MIT did that).  
> **username_1**: You're right about the neural network. But let me try to understand what your questions are: 1. "Can we train the Q network while exploring the environment?", and 2. "Can we train the Q network in simulation before training it in the real-world? If so, how do we deal with the discrepancies between the simulation and the real-world?" Are these your two questions? Anyway, these two questions are distinct enough to be asked in their separate post, so I suggest that you do that (i.e. create a new post where you ask your second question).  

---

## 1 answers

### Answer by username_2 on 2020-07-06. Score: 1 | Accepted answer

In general, you need to actively explore the environment to gather data to train your Q network. However, especially in your self-driving car example, you might be looking for Batch RL. In Batch RL you start with a given, fixed dataset of transitions (state, action, reward, next state) and you learn a policy (or Q function) based on the dataset without exploring. The Batch-Constrained Q-learning algorithm (BCQ) is a good example of this.

With regards to acquiring the data necessary for Batch RL, that's another story. I suppose one could collect data through sensors in a car, and label datapoints with rewards according to some reward function.

> **Comments:**
> **username_2**: Ah, you're right, my mistake. Edited my response to address this.  

---

