Answers.Body: <p>There are several ways to tackle this, although exploration is definitely not a solved problem yet ;)</p>
  <p>In general, I believe the right thing to do here is to measure the uncertainty of your policy or Q-value estimates and use that to construct some sort of exploration bonus. An intuitive example is given in <a href="https://arxiv.org/abs/1810.12894" rel="nofollow noreferrer" title="Exploration by Random Network Distillation">Exploration by Random Network Distillation</a>. They make two randomly initialized neural nets, one of which is never updated. At every transition, they feed transition data through both neural nets and use the difference in output between them as an estimate of uncertainty, and this quantity is added to the reward. Then they update the modifiable neural net towards the other one. This way, on a completely novel transition, the two neural nets will likely have very different outputs so the reward will be augmented a lot. Of course, this will hopefully encourage the agent to explore.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-25T14:13:56.833Z
Answers.Text: Can you break this down please , it’s quite daunting
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-25T17:31:18.187Z
Answers.Text: You have one fixed NN and another NN, both initialized randomly. Let's say these networks take states as inputs and outputs scalars. On a new state, both networks will output random scalars x and y that are likely different. Add |x-y| to the reward. Train the first network to minimize (x-y)^2. Once you've seen a state many times, the first network and the fixed network should match, so |x-y| ~= 0, so there's very little bonus.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-25T21:48:24.893Z
Answers.Text: Ohh this is deep Q learning, using target networks and our normal Q network
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-25T21:52:55.997Z
Answers.Text: Where you have a fixed target network which improves stability for our gradient descent and when it finall converges, we train our fixed networks with the updated weights of the Q network, am I right
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-25T22:50:54.253Z
Answers.Text: No, the two networks I'm talking about are not Q networks. I recommend you read the paper that I linked.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-26T07:42:00.563Z
Answers.Text: Chuckwudi, the NN are used as a distance function between states.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-25T14:12:49.923Z
Answers.LastActivityDate: 2020-06-25T14:12:49.923Z
Answers.UserDisplayName: harwiltz
Body: <p>Let's say we have a dynamic environment: a new state gets added after 2000 episodes have been done. So, we leave room for exploration, so that it can discover the new state.</p>
  <p>When it gets to that new state, it has no idea of the Q values, and, since we're past 2000 episodes, our exploration rate is very low. What happens if try to exploit when all Q values are 0?</p>
  
Tags: <reinforcement-learning><environment><exploration-exploitation-tradeoff>
Title: How to deal with the addition of a new state to the environment during training?
UserDisplayName: Chukwudi
ThreadText: # How to deal with the addition of a new state to the environment during training?
  
  (Asked by: username_0 on 2020-06-25)
  
  Let's say we have a dynamic environment: a new state gets added after 2000 episodes have been done. So, we leave room for exploration, so that it can discover the new state.
  
  When it gets to that new state, it has no idea of the Q values, and, since we're past 2000 episodes, our exploration rate is very low. What happens if try to exploit when all Q values are 0?
  
  **Tags:** reinforcement-learning, environment, exploration-exploitation-tradeoff
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-25. Score: 1
  
  There are several ways to tackle this, although exploration is definitely not a solved problem yet ;)
  
  In general, I believe the right thing to do here is to measure the uncertainty of your policy or Q-value estimates and use that to construct some sort of exploration bonus. An intuitive example is given in [Exploration by Random Network Distillation](https://arxiv.org/abs/1810.12894 "Exploration by Random Network Distillation"). They make two randomly initialized neural nets, one of which is never updated. At every transition, they feed transition data through both neural nets and use the difference in output between them as an estimate of uncertainty, and this quantity is added to the reward. Then they update the modifiable neural net towards the other one. This way, on a completely novel transition, the two neural nets will likely have very different outputs so the reward will be augmented a lot. Of course, this will hopefully encourage the agent to explore.
  
  > **Comments:**
  > **username_0**: Can you break this down please , it’s quite daunting  
  > **username_1**: You have one fixed NN and another NN, both initialized randomly. Let's say these networks take states as inputs and outputs scalars. On a new state, both networks will output random scalars x and y that are likely different. Add |x-y| to the reward. Train the first network to minimize (x-y)^2. Once you've seen a state many times, the first network and the fixed network should match, so |x-y| ~= 0, so there's very little bonus.  
  > **username_0**: Ohh this is deep Q learning, using target networks and our normal Q network  
  > **username_0**: Where you have a fixed target network which improves stability for our gradient descent and when it finall converges, we train our fixed networks with the updated weights of the Q network, am I right  
  > **username_1**: No, the two networks I'm talking about are not Q networks. I recommend you read the paper that I linked.  
  > **username_2**: Chuckwudi, the NN are used as a distance function between states.  
  
  ---
  
  