Answers.Body: <p>Please look at line 5:</p>
  <p>If <span class="math-container">$P(a_{i,j}|s_i)$</span> is equal to the policy that is used for generating the demonstrated trajectories, then it could be the same. However, in inverse RL you don't know <span class="math-container">$P(a_{i,j}|s_i)$</span> and you iteratively approximate it.</p>
  <p>Furthermore, your technique would work if the MDP is deterministic or you would have an infinite amount of trajectories (central limit theorem). Otherwise, your summation neglects the <em>probability</em> of <span class="math-container">$s'$</span> given <span class="math-container">$s,a$</span> pairs, which is why you have <span class="math-container">$P(s_k|a_{i,j},s_i)$</span> in the equation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T10:24:32.773Z
Answers.LastActivityDate: 2022-05-30T10:24:32.773Z
Answers.UserDisplayName: Tessa van der Heiden
Body: <p>I am trying to understand <a href="https://www.aaai.org/Papers/AAAI/2008/AAAI08-227.pdf" rel="nofollow noreferrer">the formulation of the maximum entropy Inverse RL method by Brian Ziebart</a>. Particularly, I am stuck on how to understand the computation of state - visitation frequencies.</p>
  
  <p>In order to do so, they utilize a dynamic programming approach to compute the visitation frequency, in which the next state frequency is calculated based upon the state visitation frequency at the previous time step. </p>
  
  <p>This is the algorithm below where, <span class="math-container">$D_{s_i,t}$</span> is the probability of state <span class="math-container">$s_i$</span> being visited at time step <span class="math-container">$t$</span>.</p>
  
  <p><a href="https://i.sstatic.net/fYQdg.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fYQdg.png" alt="enter image description here"></a></p>
  
  <p>What is the difference between this way of computing state visitation frequency compared to the naive method of summing the total number of times state <span class="math-container">$s_i$</span> appears in the trajectory divided by the trajectory length?</p>
  
Tags: <reinforcement-learning><markov-decision-process><papers>
Title: How is the state-visitation frequency computed in "Maximum Entropy Inverse Reinforcement Learning"?
UserDisplayName: calveeen
ThreadText: # How is the state-visitation frequency computed in "Maximum Entropy Inverse Reinforcement Learning"?
  
  (Asked by: username_0 on 2020-04-07)
  
  I am trying to understand [the formulation of the maximum entropy Inverse RL method by Brian Ziebart](https://www.aaai.org/Papers/AAAI/2008/AAAI08-227.pdf). Particularly, I am stuck on how to understand the computation of state - visitation frequencies.
  
  In order to do so, they utilize a dynamic programming approach to compute the visitation frequency, in which the next state frequency is calculated based upon the state visitation frequency at the previous time step.
  
  This is the algorithm below where, $D\_{s\_i,t}$ is the probability of state $s\_i$ being visited at time step $t$.
  
  [![enter image description here](https://i.sstatic.net/fYQdg.png)](https://i.sstatic.net/fYQdg.png)
  
  What is the difference between this way of computing state visitation frequency compared to the naive method of summing the total number of times state $s\_i$ appears in the trajectory divided by the trajectory length?
  
  **Tags:** reinforcement-learning, markov-decision-process, papers
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-30. Score: 0
  
  Please look at line 5:
  
  If $P(a\_{i,j}|s\_i)$ is equal to the policy that is used for generating the demonstrated trajectories, then it could be the same. However, in inverse RL you don't know $P(a\_{i,j}|s\_i)$ and you iteratively approximate it.
  
  Furthermore, your technique would work if the MDP is deterministic or you would have an infinite amount of trajectories (central limit theorem). Otherwise, your summation neglects the *probability* of $s'$ given $s,a$ pairs, which is why you have $P(s\_k|a\_{i,j},s\_i)$ in the equation.
  
  
  ---
  
  