Answers.Body: <p>The definition of non-stationarity in multi-agent learning is indeed correct.</p>
  <p>The 'moving target problem' you are referring to is investigated here: <a href="https://openreview.net/forum?id=Qun8fv4qSby" rel="nofollow noreferrer">Transient Non-stationarity and Generalisation in Deep Reinforcement Learning</a>. Both of them together have been discussed in section 5.10.1 in this book: <a href="https://www.marl-book.com/" rel="nofollow noreferrer">Multi-Agent Reinforcement Learning: Foundations and Modern Approaches</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-22T12:01:31.383Z
Answers.LastActivityDate: 2023-08-23T08:01:50.933Z
Answers.LastEditDate: 2023-08-23T08:01:50.933Z
Answers.UserDisplayName: antarteek
Body: <p>If single-agent RL algorithms such as Q-learning is applied to multi-agent systems (e.g. Markov games), the environment from the perspective of the agent is non-stationary, and the agent is faced with a moving target problem, that is, the optimal policy changes as other agents' policies changes. I understand this as the optimal Q-function, <span class="math-container">$Q^*(s,a)$</span>, being the moving target, since the optimal Q-function depends on other agents' policies.</p>
  <p>If the agent learns in the space of the joint action space, then according to several references, the environment is stationary from the perspective of any agent, even though the agents' policies may change over time. Suppose the goal of the agent is to learn a joint optimal policy defined as a Nash equilibrium, from which it bases its actions on. Then, the agent tries to find an optimal Q-function defined as <span class="math-container">$Q_{{\pi^*}}(s,\textbf{a})$</span>, where <strong>a</strong> denotes the joint action and <span class="math-container">$\pi^*$</span> denotes the joint optimal policy. I would then conclude that <span class="math-container">$Q_{{\pi^*}}(s,\textbf{a})$</span> is no longer a moving target, since we are in the space of joint policies, and thereby account for other agents' policies explicitly.</p>
  <p>So as I understand it, non-stationarity and the moving target problem are two sides of the same coin. Is this correctly understood? Or can the environment be stationary while the problem is still a moving-target?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-13T18:13:41.350Z
Comments.Text: it seems as though you're defining the moving target problem and the non-stationarity problem to be the same thing. personally, when I heard the moving target problem, I would think immediately of the fact that the function approximates in DRL is being trained towards a "moving target", in the sense that it bootstraps from itself (i.e. when you update your network, then you're also updating the target, since the target comes in part from the network).
Comments.UserDisplayName: David
Tags: <reinforcement-learning><multi-agent-systems>
Title: What is the relation between non-stationarity and "a moving target problem" in multi-agent reinforcement learning?
UserDisplayName: Necron
ThreadText: # What is the relation between non-stationarity and "a moving target problem" in multi-agent reinforcement learning?
  
  (Asked by: username_0 on 2023-06-13)
  
  If single-agent RL algorithms such as Q-learning is applied to multi-agent systems (e.g. Markov games), the environment from the perspective of the agent is non-stationary, and the agent is faced with a moving target problem, that is, the optimal policy changes as other agents' policies changes. I understand this as the optimal Q-function, $Q^\*(s,a)$, being the moving target, since the optimal Q-function depends on other agents' policies.
  
  If the agent learns in the space of the joint action space, then according to several references, the environment is stationary from the perspective of any agent, even though the agents' policies may change over time. Suppose the goal of the agent is to learn a joint optimal policy defined as a Nash equilibrium, from which it bases its actions on. Then, the agent tries to find an optimal Q-function defined as $Q\_{{\pi^\*}}(s,\textbf{a})$, where **a** denotes the joint action and $\pi^\*$ denotes the joint optimal policy. I would then conclude that $Q\_{{\pi^\*}}(s,\textbf{a})$ is no longer a moving target, since we are in the space of joint policies, and thereby account for other agents' policies explicitly.
  
  So as I understand it, non-stationarity and the moving target problem are two sides of the same coin. Is this correctly understood? Or can the environment be stationary while the problem is still a moving-target?
  
  **Tags:** reinforcement-learning, multi-agent-systems
  > **Comments:**
  > **username_1**: it seems as though you're defining the moving target problem and the non-stationarity problem to be the same thing. personally, when I heard the moving target problem, I would think immediately of the fact that the function approximates in DRL is being trained towards a "moving target", in the sense that it bootstraps from itself (i.e. when you update your network, then you're also updating the target, since the target comes in part from the network).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-08-22. Score: 0
  
  The definition of non-stationarity in multi-agent learning is indeed correct.
  
  The 'moving target problem' you are referring to is investigated here: [Transient Non-stationarity and Generalisation in Deep Reinforcement Learning](https://openreview.net/forum?id=Qun8fv4qSby). Both of them together have been discussed in section 5.10.1 in this book: [Multi-Agent Reinforcement Learning: Foundations and Modern Approaches](https://www.marl-book.com/)
  
  
  ---
  
  