<p>The equation you show does not appear in <a href="https://www.aaai.org/Papers/AAAI/2008/AAAI08-227.pdf" rel="nofollow noreferrer">Ziebart et al (2008)</a>. They do provide a description of the computation in Algorithm 1.</p>
<p>It is the visitation frequency and it is not a probability distribution, so it does not need to be averaged.</p>
<p>If you look at Equation 2. in <a href="https://arxiv.org/pdf/1806.06877.pdf?ref=https://githubhelp.com" rel="nofollow noreferrer">Arora &amp; Doshi (2020)</a>, you a formulation that describes the Algorithm 1 quite well:</p>
<p><span class="math-container">$\phi^\pi(s) = \phi^0(s) + \sum_{s'\in\mathcal{S}}P(s,\pi(s),s')\phi^\pi(s')$</span>.</p>
<p>I am not very satisfied with this formulation, because, in my opinion, there should also be a summation over <span class="math-container">$a\in \mathcal{A}$</span>, like <span class="math-container">$\eta(s)$</span> the expected number of visits, in Equation 9.2 in <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Sutton &amp; Barto (2020).</a>:</p>
<p><span class="math-container">$\eta(s)=h(s)+\sum_{\bar{s}}\eta(\bar{s})\sum_a\pi(a|\bar{s})p(s|\bar{s},a)$</span>.</p>
<p>To summarize, in all three descriptions, you just calculate how often a state is visited by policy <span class="math-container">$\pi$</span>.</p>

CC BY-SA 4.0
2022-05-30T12:05:25.963Z
2022-05-30T12:05:25.963Z
Tessa van der Heiden
<p>In the maximum entropy inverse reinforcement learning paper, Ziebart <em>et al</em>. show that the state visitation frequency <span class="math-container">$\rho(s)$</span> of a state <span class="math-container">$s$</span> can be computed as
<span class="math-container">$$
\rho_{\pi}(s) = \sum_{t}^{T} P(s_t=s|\pi),
$$</span>
which is the sum of the probability that the state being visited at each time step.</p>
<p>I just don't understand why is it the sum? From my perspective, a frequency should be the less than one, so that it should be the average value
<span class="math-container">$$
\rho_{\pi}(s) = \frac{1}{T}\sum_{t}^{T} P(s_t=s|\pi).
$$</span></p>

CC BY-SA 4.0
2021-04-26T08:34:52.810Z
My feeling is that they define the first equation and will then normalise it to make it a state distribution.
David
CC BY-SA 4.0
2021-04-28T08:08:41.963Z
It should be the average and this is rarely mentioned by people except for a IRL summer camp at UCB. You can check this [GithubIssue](https://github.com/yrlu/irl-imitation/issues/1#issuecomment-532552252) for details.
skypitcher
<reinforcement-learning><inverse-rl>
Why is it that the state visitation frequency equals the sum of state visitation frequency from initial time step to the horizon?
skypitcher
# Why is it that the state visitation frequency equals the sum of state visitation frequency from initial time step to the horizon?

(Asked by: username_0 on 2021-04-25)

In the maximum entropy inverse reinforcement learning paper, Ziebart *et al*. show that the state visitation frequency $\rho(s)$ of a state $s$ can be computed as
$$
\rho\_{\pi}(s) = \sum\_{t}^{T} P(s\_t=s|\pi),
$$
which is the sum of the probability that the state being visited at each time step.

I just don't understand why is it the sum? From my perspective, a frequency should be the less than one, so that it should be the average value
$$
\rho\_{\pi}(s) = \frac{1}{T}\sum\_{t}^{T} P(s\_t=s|\pi).
$$

**Tags:** reinforcement-learning, inverse-rl

> **Comments:**
> **username_1**: My feeling is that they define the first equation and will then normalise it to make it a state distribution.  
> **username_0**: It should be the average and this is rarely mentioned by people except for a IRL summer camp at UCB. You can check this [GithubIssue](https://github.com/yrlu/irl-imitation/issues/1#issuecomment-532552252) for details.  

---

## 1 answers

### Answer by username_2 on 2022-05-30. Score: 1

The equation you show does not appear in [Ziebart et al (2008)](https://www.aaai.org/Papers/AAAI/2008/AAAI08-227.pdf). They do provide a description of the computation in Algorithm 1.

It is the visitation frequency and it is not a probability distribution, so it does not need to be averaged.

If you look at Equation 2. in [Arora & Doshi (2020)](https://arxiv.org/pdf/1806.06877.pdf?ref=https://githubhelp.com), you a formulation that describes the Algorithm 1 quite well:

$\phi^\pi(s) = \phi^0(s) + \sum\_{s'\in\mathcal{S}}P(s,\pi(s),s')\phi^\pi(s')$.

I am not very satisfied with this formulation, because, in my opinion, there should also be a summation over $a\in \mathcal{A}$, like $\eta(s)$ the expected number of visits, in Equation 9.2 in [Sutton & Barto (2020).](http://incompleteideas.net/book/RLbook2020.pdf):

$\eta(s)=h(s)+\sum\_{\bar{s}}\eta(\bar{s})\sum\_a\pi(a|\bar{s})p(s|\bar{s},a)$.

To summarize, in all three descriptions, you just calculate how often a state is visited by policy $\pi$.


---

