Answers.Body: <p>L2 regularization pushes weights to be more normally distributed (see <a href="http://qwone.com/%7Ejason/writing/l2gaussian.pdf" rel="nofollow noreferrer">here</a>), so your output will get smoothed out. However, reward functions are pretty discrete (e.g. did the mouse get the cheese or not?). Maybe you could try other priors for the weights? The Cauchy distribution is sharper, and has loss
  <span class="math-container">$$\frac{\gamma^2}{2}\ln\left(1 + \left(\frac{w}{\gamma}\right)^2\right)$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-12T02:28:07.317Z
Answers.LastActivityDate: 2024-01-12T02:28:07.317Z
Answers.UserDisplayName: programjames
Body: <p>I have a deep-Q-network-type reinforcement learner in a <a href="https://github.com/Farama-Foundation/Minigrid" rel="nofollow noreferrer">minigrid</a>-type environment. After learning I can put the agent in a series of contrived situations and measure its Q values, and then infer its effective discount rate from these Q values (e.g. infer the discount factor based on how the value for moving forward changes with proximity to the goal).</p>
  <p>When I measure the effective discount factor this way, it matches the explicit discount factor () setting I used.</p>
  <p><em>But</em> if I add a very strong L2 regularization (weight decay) to the network, the inferred discount factor decreases, even though I didn't change the agent's  setting.</p>
  <p>Could someone help me think through why this happens?
  Thanks!</p>
  
Tags: <reinforcement-learning><dqn><discount-factor>
Title: Relationship between regularization and (effective) discounting in deep Q learning
UserDisplayName: dinnums
ThreadText: # Relationship between regularization and (effective) discounting in deep Q learning
  
  (Asked by: username_0 on 2024-01-11)
  
  I have a deep-Q-network-type reinforcement learner in a [minigrid](https://github.com/Farama-Foundation/Minigrid)-type environment. After learning I can put the agent in a series of contrived situations and measure its Q values, and then infer its effective discount rate from these Q values (e.g. infer the discount factor based on how the value for moving forward changes with proximity to the goal).
  
  When I measure the effective discount factor this way, it matches the explicit discount factor () setting I used.
  
  *But* if I add a very strong L2 regularization (weight decay) to the network, the inferred discount factor decreases, even though I didn't change the agent's setting.
  
  Could someone help me think through why this happens?
  Thanks!
  
  **Tags:** reinforcement-learning, dqn, discount-factor
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-12. Score: 0
  
  L2 regularization pushes weights to be more normally distributed (see [here](http://qwone.com/%7Ejason/writing/l2gaussian.pdf)), so your output will get smoothed out. However, reward functions are pretty discrete (e.g. did the mouse get the cheese or not?). Maybe you could try other priors for the weights? The Cauchy distribution is sharper, and has loss
  $$\frac{\gamma^2}{2}\ln\left(1 + \left(\frac{w}{\gamma}\right)^2\right)$$
  
  
  ---
  
  