Answers.Body: <p>for 1) i think your confusing elements touched vs sequences. At each point for N turns you have 2 possible options, therefore you have <span class="math-container">$\prod_{i=1}^N 2$</span> or <span class="math-container">$2^N$</span> possible sequences.   </p>
  
  <p>for 2) The priorities are updated based on the expected rewards. They do not add new elements each time, they update</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-27T03:19:38.360Z
Answers.Text: Thanks for the answer! I was wondering what implications the 2^N has, given the setup of blind cliffwalk. Lets say there are just 3 states = (0,1 and 2). According to me the possible sequences of actions until termination would be the following:
  
  0>0 (via the wrong (W) action in the first state)
  0>1>0  
  0>1>2>0 (via W action in the last state)
  0>1>2>0 (via R action in the last state)
  
  So this gives me 4 possible sequences of actions.
Answers.UserDisplayName: human_ai
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-27T03:26:24.833Z
Answers.Text: According to the formula there should be 2 x 2 x 2 possible sequences of actions. But in each state one action leads to termination which does not propagate further. So does 2 x 2 x 2 make sense here?
Answers.UserDisplayName: human_ai
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-27T05:23:53.617Z
Answers.Text: Priorities are updated based not the expected rewards but on current TD error. That way worst predictions updated more often.
Answers.UserDisplayName: mirror2image
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-26T14:07:33.803Z
Answers.LastActivityDate: 2019-06-26T14:07:33.803Z
Answers.UserDisplayName: mshlis
Body: <p>Paper link : <a href="https://arxiv.org/pdf/1511.05952.pdf" rel="nofollow noreferrer">Prioritized Experience Replay</a> </p>
  
  <p>About the blind cliffwalk setup:</p>
  
  <ol>
  <li>Why is the number of possible action sequences equal to 2^N? I cant think of sequences more than (N + 1) where one sequence is the sequence of all right actions and the other N sequences are due to wrong actions at each state.</li>
  </ol>
  
  <p>Generally for prioritized experience replay:</p>
  
  <ol start="2">
  <li>The replay memory consists of some transitions which are repeated.In the priority queue I feel that there should only be a single priority for each transition to speed up learning. Is there any advantage of having priority values for each repeated instance of the transition?</li>
  </ol>
  
  <p><strong>Edit for 2nd question:</strong></p>
  
  <p>Consider algorithm 1 on page 5 of the article.</p>
  
  <p><a href="https://i.sstatic.net/nFRP7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nFRP7.png" alt="enter image description here"></a></p>
  
  <p>Lets consider one of the transitions to be repeated in the replay memory. If one of them is sampled (line 9) and the priority updated (line 12). Will the priority update on the other instance of the same transition?</p>
  
Tags: <deep-learning><prioritized-sweeping>
Title: Clarifications on "Prioritized Experience Replay" (Deepmind, 2015)
UserDisplayName: human_ai
ThreadText: # Clarifications on "Prioritized Experience Replay" (Deepmind, 2015)
  
  (Asked by: username_0 on 2019-06-26)
  
  Paper link : [Prioritized Experience Replay](https://arxiv.org/pdf/1511.05952.pdf)
  
  About the blind cliffwalk setup:
  
  1. Why is the number of possible action sequences equal to 2^N? I cant think of sequences more than (N + 1) where one sequence is the sequence of all right actions and the other N sequences are due to wrong actions at each state.
  
  Generally for prioritized experience replay:
  
  2. The replay memory consists of some transitions which are repeated.In the priority queue I feel that there should only be a single priority for each transition to speed up learning. Is there any advantage of having priority values for each repeated instance of the transition?
  
  **Edit for 2nd question:**
  
  Consider algorithm 1 on page 5 of the article.
  
  [![enter image description here](https://i.sstatic.net/nFRP7.png)](https://i.sstatic.net/nFRP7.png)
  
  Lets consider one of the transitions to be repeated in the replay memory. If one of them is sampled (line 9) and the priority updated (line 12). Will the priority update on the other instance of the same transition?
  
  **Tags:** deep-learning, prioritized-sweeping
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-26. Score: 1
  
  for 1) i think your confusing elements touched vs sequences. At each point for N turns you have 2 possible options, therefore you have $\prod\_{i=1}^N 2$ or $2^N$ possible sequences.
  
  for 2) The priorities are updated based on the expected rewards. They do not add new elements each time, they update
  
  > **Comments:**
  > **username_0**: Thanks for the answer! I was wondering what implications the 2^N has, given the setup of blind cliffwalk. Lets say there are just 3 states = (0,1 and 2). According to me the possible sequences of actions until termination would be the following:  0>0 (via the wrong (W) action in the first state) 0>1>0   0>1>2>0 (via W action in the last state) 0>1>2>0 (via R action in the last state)  So this gives me 4 possible sequences of actions.  
  > **username_0**: According to the formula there should be 2 x 2 x 2 possible sequences of actions. But in each state one action leads to termination which does not propagate further. So does 2 x 2 x 2 make sense here?  
  > **username_2**: Priorities are updated based not the expected rewards but on current TD error. That way worst predictions updated more often.  
  
  ---
  
  