Answers.Body: <p>Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll probably find that it doesn't contain a term for exploration vs exploitation.</p>
  <p>Instead, the choice of exploration vs exploitation, and the tradeoff, is about how the data used to train the neural network are generated.</p>
  <p>So in a rough sense, what's happening to the NN is that during exploration, the network is being trained on novel data (not always, but it's more likely), while during exploitation, the network is being trained on well worn paths.</p>
  <p>Then even though the gradient formula is the same, the gradient may be smaller and the weights may change less during exploitation, because the loss is typically lower when choosing the optimal action, and with some loss functions (like MSE, but not L1 loss) a bigger loss results in a bigger gradient.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-07T22:33:37.610Z
Answers.Text: "...you'll probably find that it doesn't contain a term for exploration vs exploitation" - this is not true - think of $\epsilon$ in $\epsilon$-greedy, but maybe you meant something else?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-07T22:37:41.337Z
Answers.Text: But  is not used in the objective (aka loss) function. Sorry if my terms were confusing there, I'll add (AKA loss).
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-08T11:55:09.353Z
Answers.Text: Ok. Note that the objective function (e.g. minimization of the discounted reward) may contain a discount factor, which can play a role in or affect the exploitation-exploration dilemma. So, this dilemma is actually not just a practical issue.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-07T22:27:16.527Z
Answers.LastActivityDate: 2022-06-07T22:46:40.467Z
Answers.LastEditDate: 2022-06-07T22:46:40.467Z
Answers.UserDisplayName: Lee Reeves
Body: <p>I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is exploiting? Or somehow being updated to a lesser degree?</p>
  <p>If this question is algorithm-dependent, I'm mostly interested in PPO, but any insight is appreciated.</p>
  
Tags: <reinforcement-learning><deep-rl><proximal-policy-optimization><exploration-exploitation-tradeoff>
Title: Mathematically, what is happening differently in the neural net during exploration vs. exploitation?
UserDisplayName: Vladimir Belik
ThreadText: # Mathematically, what is happening differently in the neural net during exploration vs. exploitation?
  
  (Asked by: username_0 on 2022-06-07)
  
  I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is exploiting? Or somehow being updated to a lesser degree?
  
  If this question is algorithm-dependent, I'm mostly interested in PPO, but any insight is appreciated.
  
  **Tags:** reinforcement-learning, deep-rl, proximal-policy-optimization, exploration-exploitation-tradeoff
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-07. Score: 7 | Accepted answer
  
  Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll probably find that it doesn't contain a term for exploration vs exploitation.
  
  Instead, the choice of exploration vs exploitation, and the tradeoff, is about how the data used to train the neural network are generated.
  
  So in a rough sense, what's happening to the NN is that during exploration, the network is being trained on novel data (not always, but it's more likely), while during exploitation, the network is being trained on well worn paths.
  
  Then even though the gradient formula is the same, the gradient may be smaller and the weights may change less during exploitation, because the loss is typically lower when choosing the optimal action, and with some loss functions (like MSE, but not L1 loss) a bigger loss results in a bigger gradient.
  
  > **Comments:**
  > **username_2**: "...you'll probably find that it doesn't contain a term for exploration vs exploitation" - this is not true - think of $\epsilon$ in $\epsilon$-greedy, but maybe you meant something else?  
  > **username_1**: But  is not used in the objective (aka loss) function. Sorry if my terms were confusing there, I'll add (AKA loss).  
  > **username_2**: Ok. Note that the objective function (e.g. minimization of the discounted reward) may contain a discount factor, which can play a role in or affect the exploitation-exploration dilemma. So, this dilemma is actually not just a practical issue.  
  
  ---
  
  