Answers.Body: <p>I guess it depends on what the goal is. If the goal is a general reward function, this formulation as an MPOMDP could make sense. One way to think about this, is as a way of modeling a general (centralized) POMDP with factored actions and observation spaces.</p>
  <p>However, it seems that what you are describing might be an active perception problem, where the goal is to minimize uncertainty? In that case, there could be better formulations. I personally did some work in this direction with Yash Satsangi and others. E.g., these papers might be of interest:</p>
  <p>Yash Satsangi, Shimon Whiteson, Frans A. Oliehoek, and Henri Bouma. Real-Time Resource Allocation for Tracking Systems. In Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence (UAI), August 2017.</p>
  <p>Yash Satsangi, Shimon Whiteson, Frans A. Oliehoek, and Matthijs T. J. Spaan. Exploiting submodular value functions for scaling up active perception. Autonomous Robots, 42(2):209–233, February 2018.</p>
  <p>(and there is a large literature on this topic as well - look at the references in the papers).</p>
  <p>Hope that is useful.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-24T11:04:56.683Z
Answers.LastActivityDate: 2021-09-24T11:04:56.683Z
Answers.UserDisplayName: F.A.O.
Body: <p>Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation <span class="math-container">$o_t$</span>) with a camera at each location (action <span class="math-container">$a_t$</span>). We try to find an optimal sequence of locations for a specific goal. This decision making problem can be formalized as a Partially Observable Markov Decision Process (POMDP). Here, we seek an optimal stochastic policy <span class="math-container">$\pi^{*}_{\theta}(a_t|h_t)$</span> that maps the history <span class="math-container">$h_t= \langle o_1, a_1, ..., o_{t-1},a_{t-1},o_t \rangle$</span> of actions and observations up to the current time <span class="math-container">$t$</span> to action probabilities. The history <span class="math-container">$h_t$</span> can be summarized by the hidden state of a RNN and we can use a policy gradient method, e.g. REINFORCE, to update the policy parameters <span class="math-container">$\theta$</span>.</p>
  <p>Suppose now that we want to select multiple locations, i.e. actions, simultaneously. According to my understanding, we could formalize the problem as a Mutliagent POMDP (MPOMDP) [1]. In this formalism, we would replace the single action of the previous problem by joint actions <span class="math-container">$\vec{a}_t = \langle a^1_t, ..., a^N_t \rangle$</span>, the single observation by joint observations <span class="math-container">$\vec{o}_t = \langle o^1_t, ..., o^N_t \rangle$</span> and the history by <span class="math-container">$h_t= \langle \vec{o}_1, \vec{a}_1, ..., \vec{o}_{t-1},\vec{a}_{t-1},\vec{o}_t \rangle$</span>, where <span class="math-container">$N$</span> is the number of agents. We would now try to find an optimal joint policy <span class="math-container">$\vec{\pi}^{*} = \langle \pi^{1*}, ...,\pi^{N*} \rangle$</span> consisting of sub-policies <span class="math-container">$\pi_{\theta_n}(a^n_t|h_t)$</span> that map the history <span class="math-container">$h_t$</span> to the action probability of each agent <span class="math-container">$n$</span>. This would mean that the RNN would have <span class="math-container">$N$</span> output nodes and each sub-policy <span class="math-container">$\pi^n$</span> would be parametrized by <span class="math-container">$\theta_n$</span>, a sub-set of weights of the output layer [2]. Would it be correct to assume that an optimal or near-optimal joint policy <span class="math-container">$\vec{\pi}^{*}$</span> can be obtained by simply applying the policy gradient method used above to each sub-policy <span class="math-container">$\pi^n$</span>?</p>
  <p>I would be curious to hear what you think about the MPOMDP formalism applied to the latter decision making problem or whether you would suggest something else.</p>
  <p>[1] Oliehoek, Frans A., et al. &quot;A concise introduction to decentralized POMDPs.&quot; Springer, 2016.</p>
  <p>[2] Gupta, Jayesh K., et al. &quot;Cooperative multi-agent control using deep reinforcement learning.&quot; International Conference on Autonomous Agents and Multiagent Systems. Springer, Cham, 2017.</p>
  
Tags: <reinforcement-learning><deep-rl><policy-gradients><multi-agent-systems><pomdp>
Title: Can a Reinforcement Learning problem with multiple simultaneous actions be formalized as a Multiagent Partially Observable Markov Decision Process?
UserDisplayName: Schlozma
ThreadText: # Can a Reinforcement Learning problem with multiple simultaneous actions be formalized as a Multiagent Partially Observable Markov Decision Process?
  
  (Asked by: username_0 on 2021-09-23)
  
  Consider the following decision making problem. We have a controller that selects locations from a grid of coordinates and captures an image (observation $o\_t$) with a camera at each location (action $a\_t$). We try to find an optimal sequence of locations for a specific goal. This decision making problem can be formalized as a Partially Observable Markov Decision Process (POMDP). Here, we seek an optimal stochastic policy $\pi^{\*}\_{\theta}(a\_t|h\_t)$ that maps the history $h\_t= \langle o\_1, a\_1, ..., o\_{t-1},a\_{t-1},o\_t \rangle$ of actions and observations up to the current time $t$ to action probabilities. The history $h\_t$ can be summarized by the hidden state of a RNN and we can use a policy gradient method, e.g. REINFORCE, to update the policy parameters $\theta$.
  
  Suppose now that we want to select multiple locations, i.e. actions, simultaneously. According to my understanding, we could formalize the problem as a Mutliagent POMDP (MPOMDP) [1]. In this formalism, we would replace the single action of the previous problem by joint actions $\vec{a}\_t = \langle a^1\_t, ..., a^N\_t \rangle$, the single observation by joint observations $\vec{o}\_t = \langle o^1\_t, ..., o^N\_t \rangle$ and the history by $h\_t= \langle \vec{o}\_1, \vec{a}\_1, ..., \vec{o}\_{t-1},\vec{a}\_{t-1},\vec{o}\_t \rangle$, where $N$ is the number of agents. We would now try to find an optimal joint policy $\vec{\pi}^{\*} = \langle \pi^{1\*}, ...,\pi^{N\*} \rangle$ consisting of sub-policies $\pi\_{\theta\_n}(a^n\_t|h\_t)$ that map the history $h\_t$ to the action probability of each agent $n$. This would mean that the RNN would have $N$ output nodes and each sub-policy $\pi^n$ would be parametrized by $\theta\_n$, a sub-set of weights of the output layer [2]. Would it be correct to assume that an optimal or near-optimal joint policy $\vec{\pi}^{\*}$ can be obtained by simply applying the policy gradient method used above to each sub-policy $\pi^n$?
  
  I would be curious to hear what you think about the MPOMDP formalism applied to the latter decision making problem or whether you would suggest something else.
  
  [1] Oliehoek, Frans A., et al. "A concise introduction to decentralized POMDPs." Springer, 2016.
  
  [2] Gupta, Jayesh K., et al. "Cooperative multi-agent control using deep reinforcement learning." International Conference on Autonomous Agents and Multiagent Systems. Springer, Cham, 2017.
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, multi-agent-systems, pomdp
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-24. Score: 1
  
  I guess it depends on what the goal is. If the goal is a general reward function, this formulation as an MPOMDP could make sense. One way to think about this, is as a way of modeling a general (centralized) POMDP with factored actions and observation spaces.
  
  However, it seems that what you are describing might be an active perception problem, where the goal is to minimize uncertainty? In that case, there could be better formulations. I personally did some work in this direction with Yash Satsangi and others. E.g., these papers might be of interest:
  
  Yash Satsangi, Shimon Whiteson, Frans A. Oliehoek, and Henri Bouma. Real-Time Resource Allocation for Tracking Systems. In Proceedings of the Thirty-Third Conference on Uncertainty in Artificial Intelligence (UAI), August 2017.
  
  Yash Satsangi, Shimon Whiteson, Frans A. Oliehoek, and Matthijs T. J. Spaan. Exploiting submodular value functions for scaling up active perception. Autonomous Robots, 42(2):209–233, February 2018.
  
  (and there is a large literature on this topic as well - look at the references in the papers).
  
  Hope that is useful.
  
  
  ---
  
  