Body: <p>There is a question already about applying RL to <a href="https://ai.stackexchange.com/q/11809/37627">"large scale problems"</a>, where large scale refers to the problem of a <em>relatively small number of actions</em> (that could be from a continous space) resulting in a very large number of states. </p>
  
  <p>A good exapmle of this kind of large-scale probems is modeling a motion of a boat as a point on a plane, with an action being a displacement vector <span class="math-container">$\mathbf{\delta}_b = (\delta_x, \delta_y)$</span> and there are infinitely many states, because the next state is given by the next position of the boat, in a circle surrounding the boat <span class="math-container">$\mathcal{B}(\mathbf{x}_b, \mathbf{\delta}_{b,max})$</span>, where <span class="math-container">$\mathbf{x_b}$</span> is the boat's current position, and <span class="math-container">$\mathbf{\delta}_{b,max}$</span> the maximal possible displacement. So here, the displacement as an action (move the boat) is from an infinite space because it is a 2D vector (<span class="math-container">$\delta_b \in \subset \mathbb{R}^2$</span>) and so is the state space <span class="math-container">$\mathcal{B}$</span>. Still, I just have two actions to apply to the boat in the end: move in x-directions this much, and in y-direction that much.  </p>
  
  <p>What I mean, is something even larger. Considering the example of a boat, is it possible to apply reinforcement learning on a system that has 100 000 of such boats, and what would be the methods to look into to accomplish this. I do not mean to have 100 000 agents. The agent in this scenario is observing 100 000 boats, they are its environment, and let's say the agent is distributing them in a current on the sea in such a way that they have the least amount of resistance in the water (the wake of one ship influences the resistance of its downstream neighbors). </p>
  
  <p>From <a href="https://ai.stackexchange.com/a/11821/37627">this answer</a> and from what I have read so far, I believe an approximation will be necessary for the displacements in <span class="math-container">$2D$</span> space <span class="math-container">$\mathbf{\delta}(x,y)$</span> as well as for the states and rewards, because there are so many of them. However, before digging into this, I would like to know if there are some references out there where something like this has already been tried, or if this is simply something where RL cannot be applied. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-16T16:07:47.647Z
Comments.Text: Why don't you want to use one agent to control each of the boats or a multi-agent system? Maybe you're looking for [hierarchical reinforcement learning](https://thegradient.pub/the-promise-of-hierarchical-reinforcement-learning/).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-16T17:50:27.737Z
Comments.Text: @nbro: I would like to use the single agent and I don't know enough currently to judge if multiple agents would help or not. The main problem here is that each boat (point) impacts itself and its neighbors with its motion because the reward is a function of relative positions of all boats with respect to their immediate neighbors. So if a single agent moves individually 100 000 boats, the action is computationally expensive, the state space is sampled continuously from the starting configuration, so I am interested if there are some references for this kind of a problem.
Comments.UserDisplayName: tmaric
Tags: <reinforcement-learning><reference-request>
Title: Can reinforcement learning algorithms be applied on problems involving a very large number of possible actions?
UserDisplayName: tmaric
ThreadText: # Can reinforcement learning algorithms be applied on problems involving a very large number of possible actions?
  
  (Asked by: username_0 on 2020-06-16)
  
  There is a question already about applying RL to ["large scale problems"](https://ai.stackexchange.com/q/11809/37627), where large scale refers to the problem of a *relatively small number of actions* (that could be from a continous space) resulting in a very large number of states.
  
  A good exapmle of this kind of large-scale probems is modeling a motion of a boat as a point on a plane, with an action being a displacement vector $\mathbf{\delta}\_b = (\delta\_x, \delta\_y)$ and there are infinitely many states, because the next state is given by the next position of the boat, in a circle surrounding the boat $\mathcal{B}(\mathbf{x}\_b, \mathbf{\delta}\_{b,max})$, where $\mathbf{x\_b}$ is the boat's current position, and $\mathbf{\delta}\_{b,max}$ the maximal possible displacement. So here, the displacement as an action (move the boat) is from an infinite space because it is a 2D vector ($\delta\_b \in \subset \mathbb{R}^2$) and so is the state space $\mathcal{B}$. Still, I just have two actions to apply to the boat in the end: move in x-directions this much, and in y-direction that much.
  
  What I mean, is something even larger. Considering the example of a boat, is it possible to apply reinforcement learning on a system that has 100 000 of such boats, and what would be the methods to look into to accomplish this. I do not mean to have 100 000 agents. The agent in this scenario is observing 100 000 boats, they are its environment, and let's say the agent is distributing them in a current on the sea in such a way that they have the least amount of resistance in the water (the wake of one ship influences the resistance of its downstream neighbors).
  
  From [this answer](https://ai.stackexchange.com/a/11821/37627) and from what I have read so far, I believe an approximation will be necessary for the displacements in $2D$ space $\mathbf{\delta}(x,y)$ as well as for the states and rewards, because there are so many of them. However, before digging into this, I would like to know if there are some references out there where something like this has already been tried, or if this is simply something where RL cannot be applied.
  
  **Tags:** reinforcement-learning, reference-request
  
  > **Comments:**
  > **username_1**: Why don't you want to use one agent to control each of the boats or a multi-agent system? Maybe you're looking for [hierarchical reinforcement learning](https://thegradient.pub/the-promise-of-hierarchical-reinforcement-learning/).  
  > **username_0**: @username_1: I would like to use the single agent and I don't know enough currently to judge if multiple agents would help or not. The main problem here is that each boat (point) impacts itself and its neighbors with its motion because the reward is a function of relative positions of all boats with respect to their immediate neighbors. So if a single agent moves individually 100 000 boats, the action is computationally expensive, the state space is sampled continuously from the starting configuration, so I am interested if there are some references for this kind of a problem.  
  
  ---
  
  ## 0 answers
  
  