Answers.Body: <p>In the context of reinforcement learning, the idea of modeling your goal-oriented problem as a hierarchy of multiple sub-problems is called <strong>hierarchical reinforcement learning</strong>, which gives rise to concepts such as <em>semi-Markov decision processes</em> and <em>options</em> (aka macro actions). The article <a href="https://thegradient.pub/the-promise-of-hierarchical-reinforcement-learning/" rel="nofollow noreferrer">The Promise of Hierarchical Reinforcement Learning</a> presents and describes the topic quite well, so I suggest you read it.</p>
  <p>However, this idea of solving multiple sub-tasks in order to solve a bigger task isn't limited to RL. For example, in the paper <a href="https://arxiv.org/pdf/1511.06279.pdf" rel="nofollow noreferrer">Neural Programmer-Interpreters</a> (NPIs), without referring to the traditional RL topics, a model is proposed to write programs by composing simpler ones.</p>
  <p>Another example is genetic programming with <em>automatically defined functions</em> (ADFs), where sub-programs (the ADFs) can be re-used in different parts of the program if that's beneficial according to the fitness.</p>
  <p>So, there are different ways you can design a system to solve a big task by solving multiple sub-tasks and then compose them. The approach that you choose depends on your use case. If you want to build programs, then NPIs can be a start. If you want to incorporate a time component in your system, then HRL is probably a viable approach. There are several HRL algorithms. Some of them (e.g. <a href="https://github.com/wrighteagle2d/wrighteaglebase" rel="nofollow noreferrer">MAXQ-OP</a>) have been successfully used to solve the <a href="https://www.robocup.org/" rel="nofollow noreferrer">RoboCup challenge</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-23T20:05:18.283Z
Answers.LastActivityDate: 2021-02-17T22:34:01.487Z
Answers.LastEditDate: 2021-02-17T22:34:01.487Z
Answers.UserDisplayName: nbro
Body: <p>I read some light material earlier about the possibility of building a hierarchy of agents, where the agents at the leaves solve primitive tasks while higher-level agents are optimized for orchestrating direct-descendant agents. </p>
  
  <p>According to my understanding, each agent would have different objective functions and possible moves. </p>
  
  <p>How can I design such a hierarchy of agents each of which with different goals? Are there any examples?</p>
  
Tags: <reinforcement-learning><ai-design><multi-agent-systems><hierarchical-rl>
Title: How can I design a hierarchy of agents each of which with different goals?
UserDisplayName: A.Rashad
ThreadText: # How can I design a hierarchy of agents each of which with different goals?
  
  (Asked by: username_0 on 2017-08-24)
  
  I read some light material earlier about the possibility of building a hierarchy of agents, where the agents at the leaves solve primitive tasks while higher-level agents are optimized for orchestrating direct-descendant agents.
  
  According to my understanding, each agent would have different objective functions and possible moves.
  
  How can I design such a hierarchy of agents each of which with different goals? Are there any examples?
  
  **Tags:** reinforcement-learning, ai-design, multi-agent-systems, hierarchical-rl
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-23. Score: 1 | Accepted answer
  
  In the context of reinforcement learning, the idea of modeling your goal-oriented problem as a hierarchy of multiple sub-problems is called **hierarchical reinforcement learning**, which gives rise to concepts such as *semi-Markov decision processes* and *options* (aka macro actions). The article [The Promise of Hierarchical Reinforcement Learning](https://thegradient.pub/the-promise-of-hierarchical-reinforcement-learning/) presents and describes the topic quite well, so I suggest you read it.
  
  However, this idea of solving multiple sub-tasks in order to solve a bigger task isn't limited to RL. For example, in the paper [Neural Programmer-Interpreters](https://arxiv.org/pdf/1511.06279.pdf) (NPIs), without referring to the traditional RL topics, a model is proposed to write programs by composing simpler ones.
  
  Another example is genetic programming with *automatically defined functions* (ADFs), where sub-programs (the ADFs) can be re-used in different parts of the program if that's beneficial according to the fitness.
  
  So, there are different ways you can design a system to solve a big task by solving multiple sub-tasks and then compose them. The approach that you choose depends on your use case. If you want to build programs, then NPIs can be a start. If you want to incorporate a time component in your system, then HRL is probably a viable approach. There are several HRL algorithms. Some of them (e.g. [MAXQ-OP](https://github.com/wrighteagle2d/wrighteaglebase)) have been successfully used to solve the [RoboCup challenge](https://www.robocup.org/).
  
  
  ---
  
  