Answers.Body: <p>From my interpretation what it means is that <span class="math-container">$p_t$</span> is the priority value associated with each transition and <span class="math-container">$p_t = max_{i&lt;t} p_i $</span> means that the priority of transition number <span class="math-container">$t$</span> will be the maximum between the values of the priorities of the previous elements.</p>
  
  <p>Example: since <span class="math-container">$p_1$</span> is initialized to <span class="math-container">$1$</span>, all the new experiences will be too:
  <span class="math-container">\begin{equation}
  p_2 = max\{p_1\} = 1,
  \end{equation}</span></p>
  
  <p><span class="math-container">\begin{equation}
  p_3 = max\{p_1,p_2\} = 1,
  \end{equation}</span></p>
  
  <p><span class="math-container">\begin{equation}
  p_4 = max\{p_1,p_2,p_3\} = 1.
  \end{equation}</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-01T10:16:49.260Z
Answers.LastActivityDate: 2019-06-01T10:22:55.153Z
Answers.LastEditDate: 2019-06-01T10:22:55.153Z
Answers.UserDisplayName: Miguel Saraiva
Body: <p>I am having a hard time converting line 6 of the prioritized experience replay algorithm from <a href="https://arxiv.org/pdf/1511.05952.pdf" rel="nofollow noreferrer">the original paper</a> into plain English (see below):
  <a href="https://i.sstatic.net/8Q53w.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8Q53w.jpg" alt="algorithm" /></a></p>
  <p>I understand that new transitions (not visited before) are given maximal priority. On line 6 this would be done for every transition in an initial pass since the history is initialized as empty on line 2.</p>
  <p>I’m having trouble with the notation <span class="math-container">$p_t = \text{max}_{i&lt;t}  p_i$</span>. Can someone please state this in plain English? If <span class="math-container">$t$</span> = 4 for example, then <span class="math-container">$p_t$</span> = 4? How is this equal to max<span class="math-container">$_{i&lt;t}  p_i$</span>.</p>
  <p>It seems in my contrived example here, max<span class="math-container">$_{i&lt;t}  p_i$</span> would be 3. I must be misreading this notation.</p>
  
Tags: <reinforcement-learning><dqn><deep-rl><experience-replay><double-dqn>
Title: What does the notation $p_t = \text{max}_{i<t} p_i$ mean in algorithm 1 of the prioritized experience replay paper?
UserDisplayName: Hanzy
ThreadText: # What does the notation $p_t = \text{max}_{i<t} p_i$ mean in algorithm 1 of the prioritized experience replay paper?
  
  (Asked by: username_0 on 2019-06-01)
  
  I am having a hard time converting line 6 of the prioritized experience replay algorithm from [the original paper](https://arxiv.org/pdf/1511.05952.pdf) into plain English (see below):
  [![algorithm](https://i.sstatic.net/8Q53w.jpg)](https://i.sstatic.net/8Q53w.jpg)
  
  I understand that new transitions (not visited before) are given maximal priority. On line 6 this would be done for every transition in an initial pass since the history is initialized as empty on line 2.
  
  I’m having trouble with the notation $p\_t = \text{max}\_{i<t} p\_i$. Can someone please state this in plain English? If $t$ = 4 for example, then $p\_t$ = 4? How is this equal to max$\_{i<t} p\_i$.
  
  It seems in my contrived example here, max$\_{i<t} p\_i$ would be 3. I must be misreading this notation.
  
  **Tags:** reinforcement-learning, dqn, deep-rl, experience-replay, double-dqn
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-01. Score: 3 | Accepted answer
  
  From my interpretation what it means is that $p\_t$ is the priority value associated with each transition and $p\_t = max\_{i<t} p\_i $ means that the priority of transition number $t$ will be the maximum between the values of the priorities of the previous elements.
  
  Example: since $p\_1$ is initialized to $1$, all the new experiences will be too:
  \begin{equation}
  p\_2 = max\{p\_1\} = 1,
  \end{equation}
  
  \begin{equation}
  p\_3 = max\{p\_1,p\_2\} = 1,
  \end{equation}
  
  \begin{equation}
  p\_4 = max\{p\_1,p\_2,p\_3\} = 1.
  \end{equation}
  
  
  ---
  
  