Answers.Body: <p>The equation you show does not appear in <a href="https://www.aaai.org/Papers/AAAI/2008/AAAI08-227.pdf" rel="nofollow noreferrer">Ziebart et al (2008)</a>. They do provide a description of the computation in Algorithm 1.</p>
  <p>It is the visitation frequency and it is not a probability distribution, so it does not need to be averaged.</p>
  <p>If you look at Equation 2. in <a href="https://arxiv.org/pdf/1806.06877.pdf?ref=https://githubhelp.com" rel="nofollow noreferrer">Arora &amp; Doshi (2020)</a>, you a formulation that describes the Algorithm 1 quite well:</p>
  <p><span class="math-container">$\phi^\pi(s) = \phi^0(s) + \sum_{s'\in\mathcal{S}}P(s,\pi(s),s')\phi^\pi(s')$</span>.</p>
  <p>I am not very satisfied with this formulation, because, in my opinion, there should also be a summation over <span class="math-container">$a\in \mathcal{A}$</span>, like <span class="math-container">$\eta(s)$</span> the expected number of visits, in Equation 9.2 in <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Sutton &amp; Barto (2020).</a>:</p>
  <p><span class="math-container">$\eta(s)=h(s)+\sum_{\bar{s}}\eta(\bar{s})\sum_a\pi(a|\bar{s})p(s|\bar{s},a)$</span>.</p>
  <p>To summarize, in all three descriptions, you just calculate how often a state is visited by policy <span class="math-container">$\pi$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T12:05:25.963Z
Answers.LastActivityDate: 2022-05-30T12:05:25.963Z
Answers.UserDisplayName: Tessa van der Heiden
Body: <p>In the maximum entropy inverse reinforcement learning paper, Ziebart <em>et al</em>. show that the state visitation frequency <span class="math-container">$\rho(s)$</span> of a state <span class="math-container">$s$</span> can be computed as
  <span class="math-container">$$
  \rho_{\pi}(s) = \sum_{t}^{T} P(s_t=s|\pi),
  $$</span>
  which is the sum of the probability that the state being visited at each time step.</p>
  <p>I just don't understand why is it the sum? From my perspective, a frequency should be the less than one, so that it should be the average value
  <span class="math-container">$$
  \rho_{\pi}(s) = \frac{1}{T}\sum_{t}^{T} P(s_t=s|\pi).
  $$</span></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-26T08:34:52.810Z
Comments.Text: My feeling is that they define the first equation and will then normalise it to make it a state distribution.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-28T08:08:41.963Z
Comments.Text: It should be the average and this is rarely mentioned by people except for a IRL summer camp at UCB. You can check this [GithubIssue](https://github.com/yrlu/irl-imitation/issues/1#issuecomment-532552252) for details.
Comments.UserDisplayName: skypitcher
Tags: <reinforcement-learning><inverse-rl>
Title: Why is it that the state visitation frequency equals the sum of state visitation frequency from initial time step to the horizon?
UserDisplayName: skypitcher
ThreadText: # Why is it that the state visitation frequency equals the sum of state visitation frequency from initial time step to the horizon?
  
  (Asked by: username_0 on 2021-04-25)
  
  In the maximum entropy inverse reinforcement learning paper, Ziebart *et al*. show that the state visitation frequency $\rho(s)$ of a state $s$ can be computed as
  $$
  \rho\_{\pi}(s) = \sum\_{t}^{T} P(s\_t=s|\pi),
  $$
  which is the sum of the probability that the state being visited at each time step.
  
  I just don't understand why is it the sum? From my perspective, a frequency should be the less than one, so that it should be the average value
  $$
  \rho\_{\pi}(s) = \frac{1}{T}\sum\_{t}^{T} P(s\_t=s|\pi).
  $$
  
  **Tags:** reinforcement-learning, inverse-rl
  > **Comments:**
  > **username_1**: My feeling is that they define the first equation and will then normalise it to make it a state distribution.  
  > **username_0**: It should be the average and this is rarely mentioned by people except for a IRL summer camp at UCB. You can check this [GithubIssue](https://github.com/yrlu/irl-imitation/issues/1#issuecomment-532552252) for details.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-05-30. Score: 1
  
  The equation you show does not appear in [Ziebart et al (2008)](https://www.aaai.org/Papers/AAAI/2008/AAAI08-227.pdf). They do provide a description of the computation in Algorithm 1.
  
  It is the visitation frequency and it is not a probability distribution, so it does not need to be averaged.
  
  If you look at Equation 2. in [Arora & Doshi (2020)](https://arxiv.org/pdf/1806.06877.pdf?ref=https://githubhelp.com), you a formulation that describes the Algorithm 1 quite well:
  
  $\phi^\pi(s) = \phi^0(s) + \sum\_{s'\in\mathcal{S}}P(s,\pi(s),s')\phi^\pi(s')$.
  
  I am not very satisfied with this formulation, because, in my opinion, there should also be a summation over $a\in \mathcal{A}$, like $\eta(s)$ the expected number of visits, in Equation 9.2 in [Sutton & Barto (2020).](http://incompleteideas.net/book/RLbook2020.pdf):
  
  $\eta(s)=h(s)+\sum\_{\bar{s}}\eta(\bar{s})\sum\_a\pi(a|\bar{s})p(s|\bar{s},a)$.
  
  To summarize, in all three descriptions, you just calculate how often a state is visited by policy $\pi$.
  
  
  ---
  
  