Answers.Body: <p>What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own reward.</p>
  <p>I represent my RL agents' actions as dict, containing the RL agent ID as key and its action as value. The different agents may either use the same or a different policy and use this policy and each their own observations (also stored as dict) to compute their actions.</p>
  <pre><code>action = {}
  for agent_id, agent_obs in obs.items():
      # get the policy of the current agent
      policy_id = self.config['multiagent']['policy_mapping_fn'](agent_id)
      # use the policy and the agent's observations to compute its next action
      action[agent_id] = self.agent.compute_action(agent_obs, policy_id=policy_id)
  </code></pre>
  <p>My environment's <code>step(self, action)</code> function expects <code>action</code> to be such a dict and knows how to handle it.
  It applies all actions in the dict before calculating each agent's reward and progressing time in the environment.
  So basically, treating observations, actions, rewards (and info) as dicts solves the problem for me.</p>
  <p>For the multi-agent RL approach, I used PPO and the <a href="https://github.com/ray-project/ray" rel="nofollow noreferrer">Ray RLlib framework</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-16T09:00:37.343Z
Answers.LastActivityDate: 2021-03-16T09:00:37.343Z
Answers.UserDisplayName: stefanbschneider
Body: <p>I'm trying to design an OpenAI Gym environment in which multiple users/players perform actions over time. It's round based and each user needs to take an action before the round is evaluated and the next round starts. The action for one user can be model as a <code>gym.spaces.Discrete(5)</code> space.
  I want my RL agent to make decisions for all users. I'm wondering how to take multiple actions before progressing time and calculating the reward.</p>
  
  <p>Basically, what I want is:</p>
  
  
  
  <pre><code>obs = env.reset()
  user_actions = []
  for each user:
      user_actions.append(agent.predict(obs))
  obs, reward, done, _ = env.step(user_actions)
  </code></pre>
  
  <p>So the problem is that I don't immediately know the reward after getting an action since I need to collect all actions before evaluating the round.</p>
  
  <p>I could of course extend actions to include actions of all users in one go. But this would be problematic if I have a really large number of users or it even changes over time, right?</p>
  
  <p>I found these two (<a href="https://ai.stackexchange.com/questions/18115/openai-gym-define-action-space-when-an-agent-can-take-multiple-sub-actions-in-a">1</a>, <a href="https://stackoverflow.com/questions/55539820/how-do-apply-q-learning-to-an-openai-gym-environment-where-multiple-actions-are">2</a>) related questions, but they didn't solve my problem.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-18T11:21:00.870Z
Comments.Text: What was the format of action space you used. Was it Discrete or Multi Discrete? and how did you map that action space to your dict for passing into your step function ?
Comments.UserDisplayName: kikram
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-19T12:11:09.043Z
Comments.Text: @kikram See my question and answer below. Here, my actions were discrete and the dict, in my answer, maps agent ID to agent action.
Comments.UserDisplayName: stefanbschneider
Tags: <reinforcement-learning><open-ai><environment><gym>
Title: OpenAI Gym: Multiple actions in one step
UserDisplayName: stefanbschneider
ThreadText: # OpenAI Gym: Multiple actions in one step
  
  (Asked by: username_0 on 2020-05-05)
  
  I'm trying to design an OpenAI Gym environment in which multiple users/players perform actions over time. It's round based and each user needs to take an action before the round is evaluated and the next round starts. The action for one user can be model as a gym.spaces.Discrete(5) space.
  I want my RL agent to make decisions for all users. I'm wondering how to take multiple actions before progressing time and calculating the reward.
  Basically, what I want is:
  obs = env.reset()
  user_actions = []
  for each user:
      user_actions.append(agent.predict(obs))
  obs, reward, done, _ = env.step(user_actions)
  
  So the problem is that I don't immediately know the reward after getting an action since I need to collect all actions before evaluating the round.
  I could of course extend actions to include actions of all users in one go. But this would be problematic if I have a really large number of users or it even changes over time, right?
  I found these two (1, 2) related questions, but they didn't solve my problem.
  
  **Tags:** reinforcement-learning, open-ai, environment, gym
  > **Comments:**
  > **username_1**: What was the format of action space you used. Was it Discrete or Multi Discrete? and how did you map that action space to your dict for passing into your step function ?  
  > **username_0**: @kikram See my question and answer below. Here, my actions were discrete and the dict, in my answer, maps agent ID to agent action.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-03-16. Score: 3 | Accepted answer
  
  What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own reward.
  I represent my RL agents' actions as dict, containing the RL agent ID as key and its action as value. The different agents may either use the same or a different policy and use this policy and each their own observations (also stored as dict) to compute their actions.
  action = {}
  for agent_id, agent_obs in obs.items():
      # get the policy of the current agent
      policy_id = self.config['multiagent']['policy_mapping_fn'](agent_id)
      # use the policy and the agent's observations to compute its next action
      action[agent_id] = self.agent.compute_action(agent_obs, policy_id=policy_id)
  
  My environment's step(self, action) function expects action to be such a dict and knows how to handle it.
  It applies all actions in the dict before calculating each agent's reward and progressing time in the environment.
  So basically, treating observations, actions, rewards (and info) as dicts solves the problem for me.
  For the multi-agent RL approach, I used PPO and the Ray RLlib framework.
  
  
  ---
  
  