Answers.Body: <p>Replacement you suggest is replacement of random variable by its expectation in forward part of TD. It would make IQN into modification of <a href="https://arxiv.org/abs/1707.06887" rel="nofollow noreferrer">C51</a> with randomly sampled function approximator instead of discrete distribution. Both distribution produced and especially exploration behavior with your replacement would be very different. The authors of paper explicitly said that "more randomness" in their opinion benefit training, so reducing randomness would go aginst spirit of the paiper.  That they produce similar results on single toy test mean very little. IQN could be better then C51 or it could be worse then C51 but single toy example is not enough to say they are close. Nevertheless I agree that IQN looks overly complex and may require more training time, C51 approach could be more practical. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-04T11:27:03.897Z
Answers.LastActivityDate: 2019-04-04T11:27:03.897Z
Answers.UserDisplayName: mirror2image
Body: <p>IQN paper (<a href="https://arxiv.org/abs/1806.06923" rel="nofollow noreferrer">https://arxiv.org/abs/1806.06923</a>) uses distributional bellman target:
  <span class="math-container">$$ \delta^{\tau,\tau'}_t = r_t + \gamma Z_{\tau'}(x_{t+1}, \pi_{\beta}(x_{t+1})) - Z_{\tau}(x_t, a_t) $$</span>
  And optimizes: 
  <span class="math-container">$$ L = \frac{1}{N'} \sum^{N}_i \sum^{N'}_j \rho^\kappa_{\tau_i} \delta^{\tau_i,\tau_j}_t $$</span></p>
  
  <p>But similar quantiles can be got just from Q values, when doing so:
  <span class="math-container">$$ \delta^\tau_t = r_t + \gamma \frac{1}{N'} \sum_{j}^{N'} Z_{\tau_j}(x_{t+1}, \pi_{\beta}(x_{t+1})) - Z_\tau(x_t, a_t) \\ = r_t + \gamma Q (x_{t+1}, \pi_\beta(x_{t+1})) - Z_\tau(x_t, a_t) $$</span>
  optimizing:
  <span class="math-container">$$ L = \sum^N_i \rho^{\kappa}_{\tau_i} \delta^{\tau_i}_t $$</span></p>
  
  <p>Both lead to similar performance on CartPole env. The loss function of the 2nd one is more simpler and intuitive (atleast to me). So i was thinking if there are any obvious reason why authors didin't use it?</p>
  
Tags: <deep-learning><reinforcement-learning><papers><atari-games>
Title: IQN bellman target: using Z vs using Q
UserDisplayName: Domas A
ThreadText: # IQN bellman target: using Z vs using Q
  
  (Asked by: username_0 on 2019-04-04)
  
  IQN paper (<https://arxiv.org/abs/1806.06923>) uses distributional bellman target:
  $$ \delta^{\tau,\tau'}\_t = r\_t + \gamma Z\_{\tau'}(x\_{t+1}, \pi\_{\beta}(x\_{t+1})) - Z\_{\tau}(x\_t, a\_t) $$
  And optimizes:
  $$ L = \frac{1}{N'} \sum^{N}\_i \sum^{N'}\_j \rho^\kappa\_{\tau\_i} \delta^{\tau\_i,\tau\_j}\_t $$
  
  But similar quantiles can be got just from Q values, when doing so:
  $$ \delta^\tau\_t = r\_t + \gamma \frac{1}{N'} \sum\_{j}^{N'} Z\_{\tau\_j}(x\_{t+1}, \pi\_{\beta}(x\_{t+1})) - Z\_\tau(x\_t, a\_t) \\ = r\_t + \gamma Q (x\_{t+1}, \pi\_\beta(x\_{t+1})) - Z\_\tau(x\_t, a\_t) $$
  optimizing:
  $$ L = \sum^N\_i \rho^{\kappa}\_{\tau\_i} \delta^{\tau\_i}\_t $$
  
  Both lead to similar performance on CartPole env. The loss function of the 2nd one is more simpler and intuitive (atleast to me). So i was thinking if there are any obvious reason why authors didin't use it?
  
  **Tags:** deep-learning, reinforcement-learning, papers, atari-games
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-04-04. Score: 1
  
  Replacement you suggest is replacement of random variable by its expectation in forward part of TD. It would make IQN into modification of [C51](https://arxiv.org/abs/1707.06887) with randomly sampled function approximator instead of discrete distribution. Both distribution produced and especially exploration behavior with your replacement would be very different. The authors of paper explicitly said that "more randomness" in their opinion benefit training, so reducing randomness would go aginst spirit of the paiper. That they produce similar results on single toy test mean very little. IQN could be better then C51 or it could be worse then C51 but single toy example is not enough to say they are close. Nevertheless I agree that IQN looks overly complex and may require more training time, C51 approach could be more practical.
  
  
  ---
  
  