Body: <p>I have the following RL model that I want to train (see the diagram below). My idea is to have two agents: agent A and agent B. Agent A observes the input <code>I1</code> and decides an action <code>action1</code>, then immediately, agent B observes the input <code>(action1, I2)</code> where <code>I2=f(action1, I1)</code> is obtained with a known function <code>f(.)</code>. Next, agent B decides the action <code>action2</code>. The two actions <code>action1</code> and <code>action2</code> are used to compute the reward, which is common for the two agents. The next state of agent A will be obtained <code>I1'</code> and the process continues.</p>
  <pre><code> --------------------------- next state --------------------------------------
  |                                                                             |
  I1 -&gt; |A| -&gt; action1 ------------------------&gt;|B| -&gt; (action1, action2) -&gt; reward
  |                |                             |                
  |                |                             |
   -----------------&gt; f(I1, action1) -&gt; I2 ------
   
  </code></pre>
  <p>I want to know if is it possible to implement a DRL algorithm for this model, or more importantly, if there is a similar two-agents model that is proposed that I can use?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-19T23:53:19.343Z
Comments.Text: Is it important that there are two agents, or this just a way to explain that there are two different data structures for the two different decisions? I cannot see anything that makes this strictly a two agent problem. Is there any data that must remain hidden between the steps (e.g. for agent/step B, the agent is not allowed to know I1 (and I guess f() is not reversible if so), or some other reason why you need to view this as two agents?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-19T23:54:31.957Z
Comments.Text: @NeilSlater agree, the most important part imho is what is $f$ (and thus its properties)
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-21T15:55:13.807Z
Comments.Text: It is not important to see it as two agents. Let's say we model it as a single agent problem. What will be the input state? Only I1? How can I add I2 into the input when I don't know I2 unless I know action1 (since I know f() and I1 and to obtain I2 I need action1) ?
Comments.UserDisplayName: zdm
Tags: <reinforcement-learning><deep-rl><multi-agent-rl>
Title: Two-agent sequential RL
UserDisplayName: zdm
ThreadText: # Two-agent sequential RL
  
  (Asked by: username_0 on 2024-07-19)
  
  I have the following RL model that I want to train (see the diagram below). My idea is to have two agents: agent A and agent B. Agent A observes the input I1 and decides an action action1, then immediately, agent B observes the input (action1, I2) where I2=f(action1, I1) is obtained with a known function f(.). Next, agent B decides the action action2. The two actions action1 and action2 are used to compute the reward, which is common for the two agents. The next state of agent A will be obtained I1' and the process continues.
   --------------------------- next state --------------------------------------
  |                                                                             |
  I1 -> |A| -> action1 ------------------------>|B| -> (action1, action2) -> reward
  |                |                             |                
  |                |                             |
   -----------------> f(I1, action1) -> I2 ------
   
  
  I want to know if is it possible to implement a DRL algorithm for this model, or more importantly, if there is a similar two-agents model that is proposed that I can use?
  
  **Tags:** reinforcement-learning, deep-rl, multi-agent-rl
  > **Comments:**
  > **username_1**: Is it important that there are two agents, or this just a way to explain that there are two different data structures for the two different decisions? I cannot see anything that makes this strictly a two agent problem. Is there any data that must remain hidden between the steps (e.g. for agent/step B, the agent is not allowed to know I1 (and I guess f() is not reversible if so), or some other reason why you need to view this as two agents?  
  > **username_2**: @NeilSlater agree, the most important part imho is what is $f$ (and thus its properties)  
  > **username_0**: It is not important to see it as two agents. Let's say we model it as a single agent problem. What will be the input state? Only I1? How can I add I2 into the input when I don't know I2 unless I know action1 (since I know f() and I1 and to obtain I2 I need action1) ?  
  
  ---
  
  ## 0 answers
  
  