Answers.Body: <p>There are many techniques for training an RL agent without explicitly interacting with an environment, some of which are cited in the paper you linked. Heck, even using experience replay like in the foundational <a href="https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf" rel="nofollow noreferrer">DQN paper</a> is a way of doing this. However, while many models utilize some sort of pre-training for the sake of safety or speed, there are a couple of reasons why an environment is <em>also</em> used whenever possible.</p>
  
  <p>Eventually, your RL agent will be placed in an environment to take its own actions. This is why we train RL agents. I'm assuming that, per your question, learning does not happen during this phase. </p>
  
  <p><strong>Maybe your agent encounters a novel situation</strong><br>
  Hopefully, the experience your agent learns from is extensive enough to include every possible state-action pair <span class="math-container">$(s,a)$</span> that your agent will ever encounter. If it isn't, your agent won't have learned about these situations, and it will always perform suboptimally in them. This lack of coverage over the state-action space could be caused by stochasticity or nonstationarity in the environment.</p>
  
  <p><strong>Maybe the teacher isn't perfect</strong><br>
  If you don't allow your agent to learn from its own experience, it will only ever perform as well as the agent that collected the demonstration data. That's an upper bound on performance that we have no reason to set for ourselves.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-11T10:15:55.283Z
Answers.Text: Thanks a lot Philip. More specifically, I'm planning to pre-train a recommender system from a generated dataset using user embeddings and last interactions as the state and the items to recommend as the actions to take. Therefore, I'm very concerned about including every possible state-action par, because although the states may be very close one from another, it won't be the same exactly. Does it really matter?
Answers.UserDisplayName: Angelo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-11T11:08:22.033Z
Answers.Text: It depends on how well your model can generalize between similar states. More training data always helps with this. [Regularization](http://cs231n.github.io/neural-networks-2/#reg) could also help. But, again, it's very possible your system needs to learn from its own experience--through trial and error after deployment--in order to learn optimal behavior.
Answers.UserDisplayName: Philip Raeisghasem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-11T13:25:12.533Z
Answers.Text: Of course, I'm only talking about the pre-deployment phase. Thanks a lot.
Answers.UserDisplayName: Angelo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-11T10:04:56.840Z
Answers.LastActivityDate: 2019-03-11T10:04:56.840Z
Answers.UserDisplayName: Philip Raeisghasem
Body: <p>Following <a href="https://arxiv.org/pdf/1704.03732.pdf" rel="nofollow noreferrer">Deep Q-learning from Demonstrations</a>, I'd like to avoid potentially unsafe behavior during early learning by making use of supervised learning with demonstration data. However, the <a href="https://github.com/AurelianTactics/dqfd-with-keras/blob/master/keras_dqfd.py" rel="nofollow noreferrer">implementation</a> I'm following still uses an environment. Can I train my agent without an environment at all?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-11T06:44:08.607Z
Comments.Text: Most of this question is off-topic for this community, as we don't address implementation details. We can help you understand that paper and general things about environments, though.
Comments.UserDisplayName: Philip Raeisghasem
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-11T08:56:21.950Z
Comments.Text: Sorry, I'm newbie here. I addressed the implementation to make people see that people still uses Environments to "learn from Demonstrations" while the devil is in learning only from a certain dataset, isn't it?
Comments.UserDisplayName: Angelo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-11T09:17:56.173Z
Comments.Text: I didn't mention you. Sorry again @PhilipRaeisghasem.
Comments.UserDisplayName: Angelo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-11T09:20:25.067Z
Comments.Text: Hah, it's ok. I'm considering how to best answer your question.
Comments.UserDisplayName: Philip Raeisghasem
Tags: <reinforcement-learning><q-learning><deep-rl><environment>
Title: Is there a way to train an RL agent without any environment?
UserDisplayName: Angelo
ThreadText: # Is there a way to train an RL agent without any environment?
  
  (Asked by: username_0 on 2019-03-06)
  
  Following [Deep Q-learning from Demonstrations](https://arxiv.org/pdf/1704.03732.pdf), I'd like to avoid potentially unsafe behavior during early learning by making use of supervised learning with demonstration data. However, the [implementation](https://github.com/AurelianTactics/dqfd-with-keras/blob/master/keras_dqfd.py) I'm following still uses an environment. Can I train my agent without an environment at all?
  
  **Tags:** reinforcement-learning, q-learning, deep-rl, environment
  
  > **Comments:**
  > **username_1**: Most of this question is off-topic for this community, as we don't address implementation details. We can help you understand that paper and general things about environments, though.  
  > **username_0**: Sorry, I'm newbie here. I addressed the implementation to make people see that people still uses Environments to "learn from Demonstrations" while the devil is in learning only from a certain dataset, isn't it?  
  > **username_0**: I didn't mention you. Sorry again @username_1.  
  > **username_1**: Hah, it's ok. I'm considering how to best answer your question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-11. Score: 2 | Accepted answer
  
  There are many techniques for training an RL agent without explicitly interacting with an environment, some of which are cited in the paper you linked. Heck, even using experience replay like in the foundational [DQN paper](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) is a way of doing this. However, while many models utilize some sort of pre-training for the sake of safety or speed, there are a couple of reasons why an environment is *also* used whenever possible.
  
  Eventually, your RL agent will be placed in an environment to take its own actions. This is why we train RL agents. I'm assuming that, per your question, learning does not happen during this phase.
  
  **Maybe your agent encounters a novel situation**  
  Hopefully, the experience your agent learns from is extensive enough to include every possible state-action pair $(s,a)$ that your agent will ever encounter. If it isn't, your agent won't have learned about these situations, and it will always perform suboptimally in them. This lack of coverage over the state-action space could be caused by stochasticity or nonstationarity in the environment.
  
  **Maybe the teacher isn't perfect**  
  If you don't allow your agent to learn from its own experience, it will only ever perform as well as the agent that collected the demonstration data. That's an upper bound on performance that we have no reason to set for ourselves.
  
  > **Comments:**
  > **username_0**: Thanks a lot username_1. More specifically, I'm planning to pre-train a recommender system from a generated dataset using user embeddings and last interactions as the state and the items to recommend as the actions to take. Therefore, I'm very concerned about including every possible state-action par, because although the states may be very close one from another, it won't be the same exactly. Does it really matter?  
  > **username_1**: It depends on how well your model can generalize between similar states. More training data always helps with this. [Regularization](http://cs231n.github.io/neural-networks-2/#reg) could also help. But, again, it's very possible your system needs to learn from its own experience--through trial and error after deployment--in order to learn optimal behavior.  
  > **username_0**: Of course, I'm only talking about the pre-deployment phase. Thanks a lot.  
  
  ---
  
  