Answers.Body: <p>There is a recent paper: <a href="https://arxiv.org/pdf/2001.00449.pdf" rel="nofollow noreferrer">Continuous-Discrete Reinforcement Learning for
  Hybrid Control in Robotics</a> published by DeepMind that aims to solve this problem, as stated in the abstract:</p>
  
  <blockquote>
    <p>Many real-world control problems involve both discrete decision variables – such as the choice of control modes, gear switching or digital outputs – as
    well as continuous decision variables – such as velocity setpoints, control gains or
    analogue outputs. However, when defining the corresponding optimal control or reinforcement learning problem, it is commonly approximated with fully continuous
    or fully discrete action spaces. These simplifications aim at tailoring the problem
    to a particular algorithm or solver which may only support one type of action
    space. Alternatively, expert heuristics are used to remove discrete actions from
    an otherwise continuous space. In contrast, we propose to <strong>treat hybrid problems
    in their ‘native’ form</strong> by solving them with <strong>hybrid reinforcement learning</strong>, which
    <strong>optimizes for discrete and continuous actions simultaneously</strong>. </p>
  </blockquote>
  
  <p>The idea is that they use a <em>hybrid policy</em> that uses a Gaussian distribution for the continuous decision variables and a categorical distribution for the discrete decision variables. Then, they extend the <a href="https://arxiv.org/pdf/1806.06920.pdf" rel="nofollow noreferrer">Maximum a Posteriori Policy Optimisation (MPO)</a> algorithm (also by DeepMind) to allow it to handle hybrid policies.</p>
  
  <p><a href="https://www.youtube.com/watch?v=eUqQDLQXb7I&amp;feature=youtu.be" rel="nofollow noreferrer">Here</a> is a video showing how they used the resulting <em>hybrid MPO</em> policy in a robotics task, where in addition to the continuous actions, the robot can choose a discrete action which is the control mode to be used (coarse vs. fine).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-25T11:51:43.977Z
Answers.LastActivityDate: 2020-05-25T11:51:43.977Z
Answers.UserDisplayName: user5093249
Body: <p>Typically it seems like reinforcement learning involves learning over either a discrete or a continuous action space. An example might be choosing from a set of pre-defined game actions in Gym Retro or learning the right engine force to apply in Continuous Mountain Car; some popular approaches for these problems are deep Q-learning for the former and actor-critic methods for the latter.</p>
  
  <p>What about in the case where a single action involves picking both a discrete and a continuous parameter? For example, when choosing the type (discrete), pixel grid location (discrete), and angular orientation (continuous) of a shape from a given set to place on a grid and optimize for some reward.
  Is there a well-established approach for learning a policy to make both types of decisions at once?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-22T10:48:24.203Z
Comments.Text: Hi and welcome to AI SE! So, are you looking for a setting where at each time step the agent takes multiple actions or, equivalently, an action that is composed of other sub-actions?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-22T21:28:49.327Z
Comments.Text: In the specific context I am considering, the agent will take a single action that consists of placing a shape with a given orientation and size on a grid. This involves concurrently choosing among a set of discrete shapes, anchor locations, and a continuous rotation all at once.
Comments.UserDisplayName: Alekxos
Tags: <deep-learning><reinforcement-learning><policy-gradients><actor-critic-methods>
Title: Learning policy where action involves discrete and continuous parameters
UserDisplayName: Alekxos
ThreadText: # Learning policy where action involves discrete and continuous parameters
  
  (Asked by: username_0 on 2020-05-22)
  
  Typically it seems like reinforcement learning involves learning over either a discrete or a continuous action space. An example might be choosing from a set of pre-defined game actions in Gym Retro or learning the right engine force to apply in Continuous Mountain Car; some popular approaches for these problems are deep Q-learning for the former and actor-critic methods for the latter.
  
  What about in the case where a single action involves picking both a discrete and a continuous parameter? For example, when choosing the type (discrete), pixel grid location (discrete), and angular orientation (continuous) of a shape from a given set to place on a grid and optimize for some reward.
  Is there a well-established approach for learning a policy to make both types of decisions at once?
  
  **Tags:** deep-learning, reinforcement-learning, policy-gradients, actor-critic-methods
  
  > **Comments:**
  > **username_1**: Hi and welcome to AI SE! So, are you looking for a setting where at each time step the agent takes multiple actions or, equivalently, an action that is composed of other sub-actions?  
  > **username_0**: In the specific context I am considering, the agent will take a single action that consists of placing a shape with a given orientation and size on a grid. This involves concurrently choosing among a set of discrete shapes, anchor locations, and a continuous rotation all at once.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-05-25. Score: 1
  
  There is a recent paper: [Continuous-Discrete Reinforcement Learning for
  Hybrid Control in Robotics](https://arxiv.org/pdf/2001.00449.pdf) published by DeepMind that aims to solve this problem, as stated in the abstract:
  
  > Many real-world control problems involve both discrete decision variables – such as the choice of control modes, gear switching or digital outputs – as
  > well as continuous decision variables – such as velocity setpoints, control gains or
  > analogue outputs. However, when defining the corresponding optimal control or reinforcement learning problem, it is commonly approximated with fully continuous
  > or fully discrete action spaces. These simplifications aim at tailoring the problem
  > to a particular algorithm or solver which may only support one type of action
  > space. Alternatively, expert heuristics are used to remove discrete actions from
  > an otherwise continuous space. In contrast, we propose to **treat hybrid problems
  > in their ‘native’ form** by solving them with **hybrid reinforcement learning**, which
  > **optimizes for discrete and continuous actions simultaneously**.
  
  The idea is that they use a *hybrid policy* that uses a Gaussian distribution for the continuous decision variables and a categorical distribution for the discrete decision variables. Then, they extend the [Maximum a Posteriori Policy Optimisation (MPO)](https://arxiv.org/pdf/1806.06920.pdf) algorithm (also by DeepMind) to allow it to handle hybrid policies.
  
  [Here](https://www.youtube.com/watch?v=eUqQDLQXb7I&feature=youtu.be) is a video showing how they used the resulting *hybrid MPO* policy in a robotics task, where in addition to the continuous actions, the robot can choose a discrete action which is the control mode to be used (coarse vs. fine).
  
  
  ---
  
  