Body: <p>In the Sec. 13.2 of <a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf" rel="nofollow noreferrer">RL: An Introduction (Sutton &amp; Barto)</a>, the concept of average episode length is discussed for both episodic MDP and continuing MDP.</p>
  <p>In an episodic MDP, the average length of an episode can be calculated as the sum of the stationary distribution of states under policy <span class="math-container">$\pi$</span>, denoted as <span class="math-container">$\sum_{s}d^{\pi}(s)$</span>. In a continuing MDP, the average episode length is defined as 1. How to understand the calculation of the average length in such two cases?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-19T06:10:43.477Z
Comments.Text: I only see your reference claimed "*In the episodic case, the constant of proportionality is the average length of an episode, and in the continuing case it is 1*", where you see it claimed "the average length of an episode can be calculated as the sum of the stationary distribution of states"?
Comments.UserDisplayName: cinch
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-29T12:15:57.513Z
Comments.Text: @cinch Hi, thanks for your reply. I see this statement in the [policy gradient tutorial](https://lilianweng.github.io/posts/2018-04-08-policy-gradient/). It claims "In the episodic case, the constant of proportionality is the average length of an episode; in the continuing case, it is 1."
Comments.UserDisplayName: Yancy Pan
Tags: <reinforcement-learning><markov-decision-process>
Title: In Markov Decision Process, how to understand the calculation of the average length of episode?
UserDisplayName: Yancy Pan
ThreadText: # In Markov Decision Process, how to understand the calculation of the average length of episode?
  
  (Asked by: username_0 on 2024-04-15)
  
  In the Sec. 13.2 of [RL: An Introduction (Sutton & Barto)](http://incompleteideas.net/book/bookdraft2017nov5.pdf), the concept of average episode length is discussed for both episodic MDP and continuing MDP.
  
  In an episodic MDP, the average length of an episode can be calculated as the sum of the stationary distribution of states under policy $\pi$, denoted as $\sum\_{s}d^{\pi}(s)$. In a continuing MDP, the average episode length is defined as 1. How to understand the calculation of the average length in such two cases?
  
  **Tags:** reinforcement-learning, markov-decision-process
  > **Comments:**
  > **username_1**: I only see your reference claimed "*In the episodic case, the constant of proportionality is the average length of an episode, and in the continuing case it is 1*", where you see it claimed "the average length of an episode can be calculated as the sum of the stationary distribution of states"?  
  > **username_0**: @cinch Hi, thanks for your reply. I see this statement in the [policy gradient tutorial](https://lilianweng.github.io/posts/2018-04-08-policy-gradient/). It claims "In the episodic case, the constant of proportionality is the average length of an episode; in the continuing case, it is 1."  
  
  ---
  
  ## 0 answers
  
  