Answers.Body: <p>Because each environment individually satisfies the Markov property, the distribution of the next state <span class="math-container">$s_{t+1}$</span> in any transition depends only on <span class="math-container">$s_t$</span>, <span class="math-container">$a_t$</span>, and the transition probabilities of the active environment <span class="math-container">$e_t$</span>:</p>
  <p><span class="math-container">$$P(s_{t+1} | s_t, a_t) = P_{e_t}(s_t,a_t,s_{t+1})$$</span></p>
  <p>Each environment is independent of the others, so when mixreg builds a combined state <span class="math-container">$(s_i, s_j, \lambda)$</span> and chooses an action <span class="math-container">$a$</span>, the distribution of the next combined state <span class="math-container">$(s_{i+1}, s_{j+1}, \lambda)$</span> is simply the combined distribution:</p>
  <p><span class="math-container">$$P((s_{i+1}, s_{j+1}, \lambda) | (s_i, s_j, \lambda), a) = P_{e_i}(s_i,a,s_{i+1}) P_{e_j}(s_j,a,s_{j+1})$$</span></p>
  <p>This remains conditionally independent of all other states and actions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T22:53:57.920Z
Answers.Text: I see. So the Markov property is satisfied, even though that doesn't mean that there will be any signal to be found/remaining when mixing states. Is that correct? I might be destroying the signal, but Markov property is preserved?
Answers.UserDisplayName: Vladimir Belik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T23:09:16.487Z
Answers.Text: Do you mean the rewards? Those are preserved, either using the reward from the environment with the bigger weight or a convex mixture of the two rewards
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T23:25:09.870Z
Answers.Text: Got it, thank you for your answer! I have a related question here: https://ai.stackexchange.com/questions/35941/how-to-deal-with-sparse-rewards-when-applying-mixup-or-mixreg-to-deep-reinfo  In case you are able and willing to answer.
Answers.UserDisplayName: Vladimir Belik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T23:29:34.637Z
Answers.Text: I'm not sure how mixreg changes things there, but in single environment RL, the agent learns in steps. First it learns that the state immediately before the reward is valuable, then the state immediately before that one, and so on.
Answers.UserDisplayName: Lee Reeves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-15T22:22:58.937Z
Answers.LastActivityDate: 2022-06-15T23:35:20.570Z
Answers.LastEditDate: 2022-06-15T23:35:20.570Z
Answers.UserDisplayName: Lee Reeves
Body: <p>I've read through these two papers:</p>
  <ol>
  <li>(original about &quot;mixup&quot;) <a href="https://arxiv.org/pdf/1710.09412.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1710.09412.pdf</a></li>
  <li>(variant for RL, &quot;mixreg&quot;) <a href="https://arxiv.org/pdf/2010.10814.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2010.10814.pdf</a></li>
  </ol>
  <p>They are about a rather interesting approach to improving model generalization. Here's the thing, though - I can easily see how to use this for supervised learning, as there is always a &quot;reward&quot;/prediction on each &quot;observation&quot;/row-of-data .</p>
  <p>However, even though the second paper (mixreg) talks about applying this to RL specifically, I don't understand how you can manage this.</p>
  <p>How would you preserve the Markov property if you're mixing observations/rewards that aren't necessarily in any way sequential?</p>
  
Tags: <deep-rl><deep-neural-networks><data-augmentation>
Title: How to preserve Markov Property in Deep Reinforcement Learning when using "mixup" or "mixreg"?
UserDisplayName: Vladimir Belik
ThreadText: # How to preserve Markov Property in Deep Reinforcement Learning when using "mixup" or "mixreg"?
  
  (Asked by: username_0 on 2022-06-15)
  
  I've read through these two papers:
  
  1. (original about "mixup") <https://arxiv.org/pdf/1710.09412.pdf>
  2. (variant for RL, "mixreg") <https://arxiv.org/pdf/2010.10814.pdf>
  
  They are about a rather interesting approach to improving model generalization. Here's the thing, though - I can easily see how to use this for supervised learning, as there is always a "reward"/prediction on each "observation"/row-of-data .
  
  However, even though the second paper (mixreg) talks about applying this to RL specifically, I don't understand how you can manage this.
  
  How would you preserve the Markov property if you're mixing observations/rewards that aren't necessarily in any way sequential?
  
  **Tags:** deep-rl, deep-neural-networks, data-augmentation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-15. Score: 1 | Accepted answer
  
  Because each environment individually satisfies the Markov property, the distribution of the next state $s\_{t+1}$ in any transition depends only on $s\_t$, $a\_t$, and the transition probabilities of the active environment $e\_t$:
  
  $$P(s\_{t+1} | s\_t, a\_t) = P\_{e\_t}(s\_t,a\_t,s\_{t+1})$$
  
  Each environment is independent of the others, so when mixreg builds a combined state $(s\_i, s\_j, \lambda)$ and chooses an action $a$, the distribution of the next combined state $(s\_{i+1}, s\_{j+1}, \lambda)$ is simply the combined distribution:
  
  $$P((s\_{i+1}, s\_{j+1}, \lambda) | (s\_i, s\_j, \lambda), a) = P\_{e\_i}(s\_i,a,s\_{i+1}) P\_{e\_j}(s\_j,a,s\_{j+1})$$
  
  This remains conditionally independent of all other states and actions.
  
  > **Comments:**
  > **username_0**: I see. So the Markov property is satisfied, even though that doesn't mean that there will be any signal to be found/remaining when mixing states. Is that correct? I might be destroying the signal, but Markov property is preserved?  
  > **username_1**: Do you mean the rewards? Those are preserved, either using the reward from the environment with the bigger weight or a convex mixture of the two rewards  
  > **username_0**: Got it, thank you for your answer! I have a related question here: https://ai.stackexchange.com/questions/35941/how-to-deal-with-sparse-rewards-when-applying-mixup-or-mixreg-to-deep-reinfo  In case you are able and willing to answer.  
  > **username_1**: I'm not sure how mixreg changes things there, but in single environment RL, the agent learns in steps. First it learns that the state immediately before the reward is valuable, then the state immediately before that one, and so on.  
  
  ---
  
  