Answers.Body: <p>In distributional value functions, you can still calculate a single <em>expected</em> action value, by summing over the distribution. You can use that to select a greedy action for <span class="math-container">$\epsilon$</span>-greedy policies.</p>
  <p>So in practice this is the same as the non-distributional approach, but with a sum to get expected Q predicted by the distributional q function</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-21T16:36:48.260Z
Answers.Text: It is my understanding that distributional RL seeks to derive some further actionable insight than the estimated mean Q value of traditional RL.  If we only sum over the distribution does that not result in the mean or something with the same action as the mean?
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-21T17:33:52.393Z
Answers.Text: @foreverska not necessarily, one more direct benefit of a distributional model is that it is more apt and stable. E.g for win/draw/lose results of a game, it's a good choice, and a more informative model even if optimal choice would be the same. Perhaps worth a separate question though
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-21T09:35:51.940Z
Answers.LastActivityDate: 2023-12-21T09:35:51.940Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>I don't think this fully answers your question, but, as a related fact, the return distribution you get with distributional RL can be used as a mean to &quot;alter&quot; the policy in a smart way.</p>
  <p>What I mean is that with the return distribution you also get the <em>uncertainty</em> related to the returns, and from this you can introduce the notion of <em>risk</em>. Basically, you have a distortion risk measure, <span class="math-container">$\beta$</span>, which is a function that can transform your return distribution altering the way the policy would sample actions, since the underlying distribution has changed.</p>
  <p>Once you apply a specific implementation of risk, you can make your policy to act more conservatively for example, therefore expecting to exhibit a safer behavior, e.g., in more uncertain environments.</p>
  <p><a href="https://i.sstatic.net/y79Ke.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/y79Ke.png" alt="enter image description here" /></a></p>
  <ul>
  <li>The picture compares DQN (non-distributional) with IQN (a distributional agent), with different risk measures.</li>
  <li>As you can see the sampling distribution of the actions (so the policy) can be influenced by the specific notion of risk.</li>
  </ul>
  <p>This is called <em>risk-sensitive RL</em>, and is nicely introduced in chapter 4 of the <a href="http://arxiv.org/abs/1806.06923" rel="nofollow noreferrer">IQN paper</a>, which is a more powerful way to estimate the return distribution compared to C51. I suggest you to checkout the paper for a more in depth understanding.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-21T18:55:51.530Z
Answers.LastActivityDate: 2023-12-21T18:55:51.530Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>In &quot;A Distributional Perspective on Reinforcement Learning&quot; Bellamare et. al. 2017.  We find the following phrase.</p>
  <blockquote>
  <p>As in DQN, we use a simple <span class="math-container">$\epsilon$</span>-greedy policy over the expected action values; we leave as future work the many ways in which an agent could select actions on the basis of the full distribution.</p>
  </blockquote>
  <p>Did I miss the strategy used in the paper?  I understand <span class="math-container">$\epsilon$</span>-greedy but how were distributions compared to determine which action is the greediest?  Q is the reward with the maximum likelihood, etc?</p>
  <p>Has there been published research on the &quot;many ways&quot; this can be solved yet?</p>
  
Tags: <reinforcement-learning><deep-rl>
Title: Selection of actions based on distribution of rewards
UserDisplayName: foreverska
ThreadText: # Selection of actions based on distribution of rewards
  
  (Asked by: username_0 on 2023-12-21)
  
  In "A Distributional Perspective on Reinforcement Learning" Bellamare et. al. 2017. We find the following phrase.
  
  > As in DQN, we use a simple $\epsilon$-greedy policy over the expected action values; we leave as future work the many ways in which an agent could select actions on the basis of the full distribution.
  
  Did I miss the strategy used in the paper? I understand $\epsilon$-greedy but how were distributions compared to determine which action is the greediest? Q is the reward with the maximum likelihood, etc?
  
  Has there been published research on the "many ways" this can be solved yet?
  
  **Tags:** reinforcement-learning, deep-rl
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-12-21. Score: 3 | Accepted answer
  
  I don't think this fully answers your question, but, as a related fact, the return distribution you get with distributional RL can be used as a mean to "alter" the policy in a smart way.
  
  What I mean is that with the return distribution you also get the *uncertainty* related to the returns, and from this you can introduce the notion of *risk*. Basically, you have a distortion risk measure, $\beta$, which is a function that can transform your return distribution altering the way the policy would sample actions, since the underlying distribution has changed.
  
  Once you apply a specific implementation of risk, you can make your policy to act more conservatively for example, therefore expecting to exhibit a safer behavior, e.g., in more uncertain environments.
  
  [![enter image description here](https://i.sstatic.net/y79Ke.png)](https://i.sstatic.net/y79Ke.png)
  
  * The picture compares DQN (non-distributional) with IQN (a distributional agent), with different risk measures.
  * As you can see the sampling distribution of the actions (so the policy) can be influenced by the specific notion of risk.
  
  This is called *risk-sensitive RL*, and is nicely introduced in chapter 4 of the [IQN paper](http://arxiv.org/abs/1806.06923), which is a more powerful way to estimate the return distribution compared to C51. I suggest you to checkout the paper for a more in depth understanding.
  
  
  ---
  
  ### Answer by username_1 on 2023-12-21. Score: 3
  
  In distributional value functions, you can still calculate a single *expected* action value, by summing over the distribution. You can use that to select a greedy action for $\epsilon$-greedy policies.
  
  So in practice this is the same as the non-distributional approach, but with a sum to get expected Q predicted by the distributional q function
  
  > **Comments:**
  > **username_0**: It is my understanding that distributional RL seeks to derive some further actionable insight than the estimated mean Q value of traditional RL.  If we only sum over the distribution does that not result in the mean or something with the same action as the mean?  
  > **username_1**: @foreverska not necessarily, one more direct benefit of a distributional model is that it is more apt and stable. E.g for win/draw/lose results of a game, it's a good choice, and a more informative model even if optimal choice would be the same. Perhaps worth a separate question though  
  
  ---
  
  