Body: <p>In deep-rl techniques, if I understand correctly, a replay buffer is used in training the neural networks. The purpose of using the replay buffer is to store the experience and send a (sampled) batch of unit transitions to train neural networks as it is known that neural networks work well for iid data.</p>
  <p>But in games, experience trajectory is important as it contains temporal dynamics. Am I true? If not, all the knowledge required to learn the policy function can be obtained from (out of sequence or randomly sampled) unit transitions alone.</p>
  <p>Which one among the both is correct?</p>
  <p>Note that unit transition in this question refers to <span class="math-container">$(s_t, a_t, r_t, s_{t+1})$</span></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-04T15:56:19.780Z
Comments.Text: This is a confusing question. Please make it clear what your two options are? They are also not currently contradictory - a trajectory can be important and still broken down into parts/reassembled etc. So you need to make it clearer what your concern is about  needing to preserve a trajectory as an irreducable unit. Given that many of the earliest papers on Deep RL solve games using a database of single time step transitions, I believe that you have read them (?) and it is pretty clear that the approach works, then I would need to be be very clear on what your concern is before I could answer
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-05T04:55:05.033Z
Comments.Text: **But in games, experience trajectory is important as it contains temporal dynamics. Am I true?** I don't know whether it is true or not that the entire trajectory should be taken into account to calculate policy function. I am viewing the first option as a sentence and the second action as a collection of all bigrams. It may be the wrong form of viewing in this case. @NeilSlater
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-05T07:10:27.737Z
Comments.Text: Thanks for trying to clarify it. I still don't understand what your problem is - I just have a vague idea that you think whole trajectories are irreducable for some RL problems, but no clear understanding of why you think that. This is probably something fundamental in RL that you misunderstand, since it is reasonable to claim that all MDPs contain "temporal dynamics", and single-step temporal difference learning can be applied to any strict MDP (plus is often superior approach in terms of sample efficiency to monte carlo methods).
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-06T17:55:51.163Z
Comments.Text: @NeilSlater, okay, I understood that policy can be learned from either total trajectory or unit transitions. I think some entities like expected returns are calculated from trajectory alone and since I'm habituated with the definitions that are generally done using expected returns, I'm thinking that trajectory is mandatory. And transition function of environment can be obtained from the collection of unit transitions also...
Comments.UserDisplayName: hanugm
Tags: <neural-networks><deep-rl><experience-replay><iid>
Title: Does deep RL techniques only interested in 'unit transitions' rather than 'whole experience'?
UserDisplayName: hanugm
ThreadText: # Does deep RL techniques only interested in 'unit transitions' rather than 'whole experience'?
  
  (Asked by: username_0 on 2022-07-03)
  
  In deep-rl techniques, if I understand correctly, a replay buffer is used in training the neural networks. The purpose of using the replay buffer is to store the experience and send a (sampled) batch of unit transitions to train neural networks as it is known that neural networks work well for iid data.
  
  But in games, experience trajectory is important as it contains temporal dynamics. Am I true? If not, all the knowledge required to learn the policy function can be obtained from (out of sequence or randomly sampled) unit transitions alone.
  
  Which one among the both is correct?
  
  Note that unit transition in this question refers to $(s\_t, a\_t, r\_t, s\_{t+1})$
  
  **Tags:** neural-networks, deep-rl, experience-replay, iid
  
  > **Comments:**
  > **username_1**: This is a confusing question. Please make it clear what your two options are? They are also not currently contradictory - a trajectory can be important and still broken down into parts/reassembled etc. So you need to make it clearer what your concern is about  needing to preserve a trajectory as an irreducable unit. Given that many of the earliest papers on Deep RL solve games using a database of single time step transitions, I believe that you have read them (?) and it is pretty clear that the approach works, then I would need to be be very clear on what your concern is before I could answer  
  > **username_0**: **But in games, experience trajectory is important as it contains temporal dynamics. Am I true?** I don't know whether it is true or not that the entire trajectory should be taken into account to calculate policy function. I am viewing the first option as a sentence and the second action as a collection of all bigrams. It may be the wrong form of viewing in this case. @username_1  
  > **username_1**: Thanks for trying to clarify it. I still don't understand what your problem is - I just have a vague idea that you think whole trajectories are irreducable for some RL problems, but no clear understanding of why you think that. This is probably something fundamental in RL that you misunderstand, since it is reasonable to claim that all MDPs contain "temporal dynamics", and single-step temporal difference learning can be applied to any strict MDP (plus is often superior approach in terms of sample efficiency to monte carlo methods).  
  > **username_0**: @username_1, okay, I understood that policy can be learned from either total trajectory or unit transitions. I think some entities like expected returns are calculated from trajectory alone and since I'm habituated with the definitions that are generally done using expected returns, I'm thinking that trajectory is mandatory. And transition function of environment can be obtained from the collection of unit transitions also...  
  
  ---
  
  ## 0 answers
  
  