Answers.Body: <p>From comments, you say there is no &quot;outer&quot; goal for picking an adversary other than  scoring highly in an individual episode.</p>
  <p>You could <em>potentially</em> model the initial adversary choice as a partially separate Markov Decision Process (MDP), where choosing the opponent is a single-step episode with return equal to whatever reward the secondary MDP - which played the game - obtains. However, this &quot;outer&quot; MDP is not much of an MDP at all, it is more like a contextual bandit. In addition, the performance of the inner game-playing agent will vary both with the choice of opponent, and over time as it learns to play better against each opponent. This makes the outer MDP non-stationary. It also requires the inner MDP to know what opponent it is facing in order to correctly predict correct choices and/or future rewards.</p>
  <p>That last part - the need for any &quot;inner&quot; agent to be aware of the opponent it is playing against - is likely to be necessary whatever structure you chooose. That choice of opponent needs to be part of the state for this inner agent, because it will have an impact on likely future rewards. A characterisation of the opponents also needs to be part of whatever predictive models you could use for the outer agent.</p>
  <p>A more natural, and probably more useful, MDP model for your problem is to have a single MDP where the first action <span class="math-container">$a_0$</span> is to select the opponent. This matches the language you use to describe the problem, and resolves your issue about trying to run a hierarchy of agents. <a href="https://arxiv.org/abs/1910.04450#:%7E:text=Hierarchical%20Reinforcement%20Learning%20(HRL)%20is,to%20define%20low%2Dlevel%20rewards." rel="nofollow noreferrer">Hierarchical reinforcement learning</a> is a real thing, and very interesting for solving problems which can be broken down into meaningful sub-goals that an agent could discover autonomously, but it does not appear to apply for your problem.</p>
  <p>This leaves you with a practical problem of creating a model that can switch between choosing between two sets of radically different actions. The <em>select an opponent</em> action only occurs at the first state of the game, and the two sets of actions do not overlap at all. However, in terms of the theoretical MDP model this is not an issue at all. It is only a practical issue of how you get to fit your Q function approximator to the two radically different action types. There a a few ways around that. Here are a couple that might work for you:</p>
  <p><strong>One shared network</strong></p>
  <p>Always predict for all kinds of action choice, so the agent still makes predictions for switching opponents all the way to the end of the game. Then filter the action choices down to only those available at any time step. When <span class="math-container">$t=0$</span> only use the predictions for actions for selecting an opponent, for <span class="math-container">$t \ge 1$</span> only use predictions relating to moves in the game.</p>
  <p><strong>Two separate approximators</strong></p>
  <p>Have two function approximators in your agent, use one for predicting reward at <span class="math-container">$t=0$</span> that covers different opponent choices, and use the other for the rest of the game. If <span class="math-container">$n$</span> is small and there is no generalisation between opponents (i.e. no opponent &quot;stats&quot; that give some kind of clue towards the end results), then for the first approximator, you could even use a Q table.</p>
  <p>For update steps you need to know whether any particular action value was modelled in one or other of the Q functions - and this will naturally lead you to bootstrap</p>
  <p><span class="math-container">$$\hat{q}_{o}(s_0, a_0, \theta_0) \leftarrow r_1 + \gamma \text{max}_{a'}\hat{q}_{p}(s_1, a', \theta_1)$$</span></p>
  <p>where <span class="math-container">$\hat{q}_{o}$</span> is your approximate model for action values of selecting opponents (and <span class="math-container">$a_0$</span> must be an opponent choice) at the start of the game, and <span class="math-container">$\hat{q}_{p}$</span> is the nodel you use for the rest of it (and <span class="math-container">$a'$</span> must be a position play in the game). I've misused <span class="math-container">$\leftarrow$</span> here to stand in for whatever process used to update the action value towards the new estimate - a tabular method that would be a rolling average with current estimate, in neural networks of course that is gradient descent using backpropagation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T12:12:33.393Z
Answers.Text: I fully agree with your first three paragraphs, but not sure about the rest. Yeah, that could work, but I feel like completely separating the problems would work better; treat the opponent selection as a MAB, and the playing as an MDP. Inside the MDP, you could augment the state representation to include a one-hot vector to represent the opponent, or even some other kind of representation if meaningful features are available that could allow for generalisation across similar opponents. Treating everything as a single MDP would just exacerbate the credit assignment problem?
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-21T14:26:33.743Z
Answers.Text: @DennisSoemers: "Treating everything as a single MDP would just exacerbate the credit assignment problem?" I don't think so. That issue is a core part of the problem definition and would either appear as "credit assignment" in a single MDP view or high variance and non-stationarity in a two MDP view. In terms of number of samples required, and the values passing between steps for value updates, the two views (single MDP with differing action spaces vs two MDPs) are practically identical.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-03T21:11:24.657Z
Answers.Text: Say I use MAB to select opponents. Should I do (1) or (2)? Here is (1): in each round of the MAB, I select an opponent, and train the DQN (given the chosen opponent). After training, I get the best reward from the trained DQN, and update MAB probabilities and go to the next round of the MAB. Here is (2):  in each round of the MAB, I select an opponent, and play one episode of the DQN (given the chosen opponent). After the terminal state is reached, I get the reward, and update MAB probabilities and go to the next round.
Answers.UserDisplayName: zdm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-04T09:04:07.483Z
Answers.Text: @zdm: If you are able to do this as an experiemnt, then you can test your idea. If you want some theoretical or practical advice from an expert then I think that you should ask a new question on the site. On a short think through, with (1) you will need to train N DQNs (one for each opponent) in sequence and keep them. It may be more reliable than (2) in the long term, but (2) may find your best initial choice earlier and require a lot less training as it will start to focus on the *probably best* opponents before needing to fully converge.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-20T17:14:48.040Z
Answers.LastActivityDate: 2020-06-21T09:03:55.923Z
Answers.LastEditDate: 2020-06-21T09:03:55.923Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have the following situation. An agent plays a game and wants to maximize the accumulated reward as usual, but it can choose its adversary. There are <span class="math-container">$n$</span> adversaries.</p>
  <p>In episode <span class="math-container">$e$</span>, the agent must first select an adversary. Then for each step <span class="math-container">$t$</span> in the episode <span class="math-container">$e$</span>, it plays the game against the chosen adversary. Every step <span class="math-container">$t$</span>, it receives a reward following the chosen action in step <span class="math-container">$t$</span> (for the chosen adversary). How to maximize the expected rewards using DQN? It is clear that choosing the &quot;wrong&quot; (the strongest) adversary won't be a good choice for the agent. Thus, to maximize the accumulated rewards, the agent must take two actions at two different timescales.</p>
  <p>I started solving it using two DQNs, one to decide the adversary to play against and one to play the game against the chosen adversary. I have two duplicate hyperparameters (<code>batch_size</code>, <code>target_update_freq</code>, etc), one for each DQN. Have you ever seen two DQNs like this? Should I train the DQNs simultaneously?</p>
  <p>The results that I am getting is not that good. The accumulated reward is decreasing, the loss isn't always decreasing...</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-20T08:31:28.150Z
Comments.Text: Are episodes in any way connected so that a series of episodes form a "meta-episode" with clear time sequence, consequences for picking a particular opponent in episode 1 as opposed to episode 3, and an accumulation of reward over multiple episodes that matters (e.g. winning a series of games)?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-20T13:27:38.870Z
Comments.Text: I think they should be connected right? but I don't know how. Right now in my modeling, once I choose an adversary and play against it, I see the reward and move to the next episode by resetting the environment. Should I count the number of times I won against an adversary as a meta-episode or something?
Comments.UserDisplayName: zdm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-20T13:31:58.410Z
Comments.Text: It depends what your goal is. Is there any larger structure in the game other than an episode? Does the choice of opponent have any *other* effect than influencing the results of an individual episode? For instance, does it affect the list of opponents possible for the next episode? If not, then perhaps there is no connection and the answer will be different. That is why I am checking in comments for clarification
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-20T14:17:04.350Z
Comments.Text: Yes, right now I am thinking of no connection. I thought of playing it with $n$ independent DQNs---one for each adversary---and then select the one with the best reward. But this does not work for large $n$.
Comments.UserDisplayName: zdm
Tags: <reinforcement-learning><deep-rl>
Title: Two DQNs in two different time scales
UserDisplayName: zdm
ThreadText: # Two DQNs in two different time scales
  
  (Asked by: username_0 on 2020-06-20)
  
  I have the following situation. An agent plays a game and wants to maximize the accumulated reward as usual, but it can choose its adversary. There are $n$ adversaries.
  
  In episode $e$, the agent must first select an adversary. Then for each step $t$ in the episode $e$, it plays the game against the chosen adversary. Every step $t$, it receives a reward following the chosen action in step $t$ (for the chosen adversary). How to maximize the expected rewards using DQN? It is clear that choosing the "wrong" (the strongest) adversary won't be a good choice for the agent. Thus, to maximize the accumulated rewards, the agent must take two actions at two different timescales.
  
  I started solving it using two DQNs, one to decide the adversary to play against and one to play the game against the chosen adversary. I have two duplicate hyperparameters (`batch_size`, `target_update_freq`, etc), one for each DQN. Have you ever seen two DQNs like this? Should I train the DQNs simultaneously?
  
  The results that I am getting is not that good. The accumulated reward is decreasing, the loss isn't always decreasing...
  
  **Tags:** reinforcement-learning, deep-rl
  
  > **Comments:**
  > **username_1**: Are episodes in any way connected so that a series of episodes form a "meta-episode" with clear time sequence, consequences for picking a particular opponent in episode 1 as opposed to episode 3, and an accumulation of reward over multiple episodes that matters (e.g. winning a series of games)?  
  > **username_0**: I think they should be connected right? but I don't know how. Right now in my modeling, once I choose an adversary and play against it, I see the reward and move to the next episode by resetting the environment. Should I count the number of times I won against an adversary as a meta-episode or something?  
  > **username_1**: It depends what your goal is. Is there any larger structure in the game other than an episode? Does the choice of opponent have any *other* effect than influencing the results of an individual episode? For instance, does it affect the list of opponents possible for the next episode? If not, then perhaps there is no connection and the answer will be different. That is why I am checking in comments for clarification  
  > **username_0**: Yes, right now I am thinking of no connection. I thought of playing it with $n$ independent DQNs---one for each adversary---and then select the one with the best reward. But this does not work for large $n$.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-20. Score: 3 | Accepted answer
  
  From comments, you say there is no "outer" goal for picking an adversary other than scoring highly in an individual episode.
  
  You could *potentially* model the initial adversary choice as a partially separate Markov Decision Process (MDP), where choosing the opponent is a single-step episode with return equal to whatever reward the secondary MDP - which played the game - obtains. However, this "outer" MDP is not much of an MDP at all, it is more like a contextual bandit. In addition, the performance of the inner game-playing agent will vary both with the choice of opponent, and over time as it learns to play better against each opponent. This makes the outer MDP non-stationary. It also requires the inner MDP to know what opponent it is facing in order to correctly predict correct choices and/or future rewards.
  
  That last part - the need for any "inner" agent to be aware of the opponent it is playing against - is likely to be necessary whatever structure you chooose. That choice of opponent needs to be part of the state for this inner agent, because it will have an impact on likely future rewards. A characterisation of the opponents also needs to be part of whatever predictive models you could use for the outer agent.
  
  A more natural, and probably more useful, MDP model for your problem is to have a single MDP where the first action $a\_0$ is to select the opponent. This matches the language you use to describe the problem, and resolves your issue about trying to run a hierarchy of agents. [Hierarchical reinforcement learning](https://arxiv.org/abs/1910.04450#:%7E:text=Hierarchical%20Reinforcement%20Learning%20(HRL)%20is,to%20define%20low%2Dlevel%20rewards.) is a real thing, and very interesting for solving problems which can be broken down into meaningful sub-goals that an agent could discover autonomously, but it does not appear to apply for your problem.
  
  This leaves you with a practical problem of creating a model that can switch between choosing between two sets of radically different actions. The *select an opponent* action only occurs at the first state of the game, and the two sets of actions do not overlap at all. However, in terms of the theoretical MDP model this is not an issue at all. It is only a practical issue of how you get to fit your Q function approximator to the two radically different action types. There a a few ways around that. Here are a couple that might work for you:
  
  **One shared network**
  
  Always predict for all kinds of action choice, so the agent still makes predictions for switching opponents all the way to the end of the game. Then filter the action choices down to only those available at any time step. When $t=0$ only use the predictions for actions for selecting an opponent, for $t \ge 1$ only use predictions relating to moves in the game.
  
  **Two separate approximators**
  
  Have two function approximators in your agent, use one for predicting reward at $t=0$ that covers different opponent choices, and use the other for the rest of the game. If $n$ is small and there is no generalisation between opponents (i.e. no opponent "stats" that give some kind of clue towards the end results), then for the first approximator, you could even use a Q table.
  
  For update steps you need to know whether any particular action value was modelled in one or other of the Q functions - and this will naturally lead you to bootstrap
  
  $$\hat{q}\_{o}(s\_0, a\_0, \theta\_0) \leftarrow r\_1 + \gamma \text{max}\_{a'}\hat{q}\_{p}(s\_1, a', \theta\_1)$$
  
  where $\hat{q}\_{o}$ is your approximate model for action values of selecting opponents (and $a\_0$ must be an opponent choice) at the start of the game, and $\hat{q}\_{p}$ is the nodel you use for the rest of it (and $a'$ must be a position play in the game). I've misused $\leftarrow$ here to stand in for whatever process used to update the action value towards the new estimate - a tabular method that would be a rolling average with current estimate, in neural networks of course that is gradient descent using backpropagation.
  
  > **Comments:**
  > **username_2**: I fully agree with your first three paragraphs, but not sure about the rest. Yeah, that could work, but I feel like completely separating the problems would work better; treat the opponent selection as a MAB, and the playing as an MDP. Inside the MDP, you could augment the state representation to include a one-hot vector to represent the opponent, or even some other kind of representation if meaningful features are available that could allow for generalisation across similar opponents. Treating everything as a single MDP would just exacerbate the credit assignment problem?  
  > **username_1**: @username_2: "Treating everything as a single MDP would just exacerbate the credit assignment problem?" I don't think so. That issue is a core part of the problem definition and would either appear as "credit assignment" in a single MDP view or high variance and non-stationarity in a two MDP view. In terms of number of samples required, and the values passing between steps for value updates, the two views (single MDP with differing action spaces vs two MDPs) are practically identical.  
  > **username_0**: Say I use MAB to select opponents. Should I do (1) or (2)? Here is (1): in each round of the MAB, I select an opponent, and train the DQN (given the chosen opponent). After training, I get the best reward from the trained DQN, and update MAB probabilities and go to the next round of the MAB. Here is (2):  in each round of the MAB, I select an opponent, and play one episode of the DQN (given the chosen opponent). After the terminal state is reached, I get the reward, and update MAB probabilities and go to the next round.  
  > **username_1**: @username_0: If you are able to do this as an experiemnt, then you can test your idea. If you want some theoretical or practical advice from an expert then I think that you should ask a new question on the site. On a short think through, with (1) you will need to train N DQNs (one for each opponent) in sequence and keep them. It may be more reliable than (2) in the long term, but (2) may find your best initial choice earlier and require a lot less training as it will start to focus on the *probably best* opponents before needing to fully converge.  
  
  ---
  
  