Answers.Body: <p>This is mostly an implementation architecture problem, and the thing is that basically you can implement anything in the traditional setting. To do so instead of having <code>Env&lt;-&gt;Agent1&lt;-&gt;Agent2</code>, you should have <code>Agent1&lt;-&gt;SuperEnv&lt;-&gt;Agent2</code> where <code>SuperEnv</code> contains <code>Env</code>, and simply uses the reward given to <code>SuperEnv</code> by <code>Agent1</code> and passes it to <code>Agent2</code>.</p>
  <p>I know this might seem a little counter-intuitive when comparing the implementation to the real-world problem setting, but the consistency of the RL structure (i.e. Environment that interacts with all the agent) is very important for your solutions to be easily understandable by others.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-02T22:57:33.397Z
Answers.Text: It would be nice if you could provide a link to some project or research paper that has done something similar to what you're suggesting.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-03T07:32:23.730Z
Answers.Text: Not aware of any project with this quite unusual setup. It is just matter of changing the environment step function essentially though (and giving the environment object an attribute that stores the reward returned by `Agent1` in order to then feed this reward at the next step to `Agent2`)
Answers.UserDisplayName: Federico Malerba
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-02T20:09:18.850Z
Answers.LastActivityDate: 2021-01-02T20:09:18.850Z
Answers.UserDisplayName: Federico Malerba
Body: <p>The traditional setting of multiagent reinforcement learning (MARL) is the mode in which there is set of agents and external environment. And the reward is given to each agent - individually or collectively - by the external environment.</p>
  <p>My question is - is there MARL model in which the reward is given by one agent to the other agent, meaning that one agent is incurring costs and other agent - revenue (or maybe even a profit?</p>
  <p>Effectively that means distributed supervision: only some agents face the environment with real reward/supervision and then this supervision is more or less effectively propgated to other agents that learn/do their own specialized tasks that are part of collective task ececuted/solved distributively in MARL.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-01T04:13:18.137Z
Comments.Text: https://arxiv.org/abs/1901.08492 seems to be some part of the answer - hierarchical supervision, mutually-supervised agent learning etc. seems to be the terms used for research suggested in my question.
Comments.UserDisplayName: TomR
Tags: <reinforcement-learning><rewards><multi-agent-systems>
Title: Is there multi-agent reinforcement learning model in which (some of the) reward is given by other agent and not by the external environment?
UserDisplayName: TomR
ThreadText: # Is there multi-agent reinforcement learning model in which (some of the) reward is given by other agent and not by the external environment?
  
  (Asked by: username_0 on 2020-12-31)
  
  The traditional setting of multiagent reinforcement learning (MARL) is the mode in which there is set of agents and external environment. And the reward is given to each agent - individually or collectively - by the external environment.
  
  My question is - is there MARL model in which the reward is given by one agent to the other agent, meaning that one agent is incurring costs and other agent - revenue (or maybe even a profit?
  
  Effectively that means distributed supervision: only some agents face the environment with real reward/supervision and then this supervision is more or less effectively propgated to other agents that learn/do their own specialized tasks that are part of collective task ececuted/solved distributively in MARL.
  
  **Tags:** reinforcement-learning, rewards, multi-agent-systems
  > **Comments:**
  > **username_0**: https://arxiv.org/abs/1901.08492 seems to be some part of the answer - hierarchical supervision, mutually-supervised agent learning etc. seems to be the terms used for research suggested in my question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-02. Score: 1
  
  This is mostly an implementation architecture problem, and the thing is that basically you can implement anything in the traditional setting. To do so instead of having `Env<->Agent1<->Agent2`, you should have `Agent1<->SuperEnv<->Agent2` where `SuperEnv` contains `Env`, and simply uses the reward given to `SuperEnv` by `Agent1` and passes it to `Agent2`.
  
  I know this might seem a little counter-intuitive when comparing the implementation to the real-world problem setting, but the consistency of the RL structure (i.e. Environment that interacts with all the agent) is very important for your solutions to be easily understandable by others.
  
  > **Comments:**
  > **username_2**: It would be nice if you could provide a link to some project or research paper that has done something similar to what you're suggesting.  
  > **username_1**: Not aware of any project with this quite unusual setup. It is just matter of changing the environment step function essentially though (and giving the environment object an attribute that stores the reward returned by `Agent1` in order to then feed this reward at the next step to `Agent2`)  
  
  ---
  
  