Answers.Body: <h1>Q-learning for continuous state spaces</h1>
  
  <p>Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the space to a small number of states. Provided you can sample and update enough times, then a few million states is not a major problem. </p>
  
  <p>However, with large state spaces it is more common to use some form of function approximation for the action value. This is often noted <span class="math-container">$\hat{q}(s,a,\theta)$</span> to show that it is both an estimate (the circumflex over <span class="math-container">$\hat{q}$</span>) and that you are learning some function parameters (<span class="math-container">$\theta$</span>). There are broadly two popular approaches to Q-learning using function approximation:</p>
  
  <ul>
  <li><p>Linear function approximation over a processed version of the state into features. A lot of variations to generate features have been proposed and tested, including Fourier series, tile coding, radial basic functions. The advantage of these methods are that they are simple, and more robust than non-linear function approximations. Which one to choose depends on what you state space represents and how the value function is likely to vary depending on location within the state space.</p></li>
  <li><p>Neural network function approximation. This is essentially what Deep Q Networks (DQN) are. Provided you have a Markov state description, you scale it to work sensibly with neural networks, and you follow other DQN best practices (experience replay table, slow changing target network) this can work well.</p></li>
  </ul>
  
  <h1>Q-learning for continuous action spaces</h1>
  
  <p>Unless you discretise the action space, then this becomes very unwieldy.</p>
  
  <p>The problem is that, given <span class="math-container">$s,a,r,s'$</span>, Q-learning needs to evaluate the TD target:</p>
  
  <p><span class="math-container">$$Q_{target}(s,a) = r + \gamma \text{max}_{a'} \hat{q}(s',a',\theta)$$</span></p>
  
  <p>The process for evaluating the maximum becomes less efficient and less accurate the larger the space that it needs to check.</p>
  
  <p>For somewhat large action spaces, using double Q-learning can help (with two estimates of Q, one to pick the target action, the other to estimate its value, which you alternate between on different steps) - this helps avoid maximisation bias where picking an action because it has the highest value and then using that highest value in calculations leads to over-estimating value.</p>
  
  <p>For very large or continuous action spaces, it is not usually practical to check all values. The alternative to Q-learning in this case is to use a policy gradient method such as Actor-Critic which can cope with very large or continuous action spaces, and does not rely on maximising over all possible actions in order to enact or evaluate a policy.</p>
  
  <h2>Controlling a pendulum</h2>
  
  <p>For a discrete action space e.g. applying one of a choice of forces on each time step, then this can be done using a DQN approach or any other function approximation. The classic example here might be an environment like Open AI's <a href="https://gym.openai.com/envs/CartPole-v1/" rel="noreferrer">CartPole-v1</a> where the state space is continuous, but there are only two possible actions. This can be solved easily using DQN, it is something of a beginner's problem.</p>
  
  <p>Adding continuous action space ends up with something like the <a href="https://gym.openai.com/envs/Pendulum-v0/" rel="noreferrer">Pendulum-v0</a> environment. This can be solved to some degree using DQN and discretising the action space (to e.g. 9 different actions). However, it is possible to make more optimal solutions using an Actor-Critic algorithm like A3C.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-25T02:16:38.767Z
Answers.Text: How do you calculate q^(s′,a′,θ) for the training? My guess is to calculate backward from the end state of the pendulum. During the first run of the DQN, do you use one value for all q^(s′,a′,θ) or randomize them per example?
Answers.UserDisplayName: Joachim Rives
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-25T08:01:50.513Z
Answers.Text: @JoachimRives $\hat{q}(s',a',\theta)$ is the output of the neural network or other parametric estimator during training, and $\theta$ stands for the learning parameters. To calculate what the value *should* be trained to be - for DQN at least - that's the TD target, and the formula for that is shown in the answer. Wehn using DQN, it is important to recalculate the TD target every time you retrieve an example from replay memory.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-25T08:40:30.783Z
Answers.Text: Do you mean the DQN TD is calculated backward or based on the expected value of succeeding steps leading to the terminal state?
Answers.UserDisplayName: Joachim Rives
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-25T09:03:40.963Z
Answers.Text: @JoachimRives The value is calculated as in the answer. TD learning (of which Q learning is an example) does not calculate backwards from the terminal state, instead it bootstraps from existing estimates that can be in the middle of a trajectory. Other algorithms do resolve backwards from terminal states - e.g. Monte Carlo control. There are already questions on this site that explain the difference further
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-11T11:32:06.907Z
Answers.LastActivityDate: 2019-05-11T13:17:59.623Z
Answers.LastEditDate: 2019-05-11T13:17:59.623Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <blockquote>
  <p><strong>Q-Learning for continuous state space</strong></p>
  </blockquote>
  <p>Reinforcement learning algorithms (e.g Q-Learning) can be applied to both discrete and continuous spaces. If you understand how it works in discrete mode, then you can easily move to continuous mode. That's why in the literature all the introductory material focuses on discrete mode, as it's easier to model (table, grid, etc.)</p>
  <p>Supposing you have a discrete number of actions, <strong>the only difference</strong> in a continuous space is that you will be modeling the state each <span class="math-container">$X$</span> amount of time (<span class="math-container">$X$</span> being a number you can choose depending on your use case). So, basically, you end up with a discrete space, but probably with an infinite number of states. You apply then the same approach you learned for discrete mode.</p>
  <p>Let's take the example of self-driving cars, at each <span class="math-container">$X$</span>ms (e.g <span class="math-container">$X=1$</span>), you'll be computing the state of the car which are your input features (e.g direction, orientation, rotation, distance to the pavement, relative position on the lane, etc.) and take a decision of the action to take as in discrete mode. The approach is the same in other use cases, like playing games, walking robots, and so on.</p>
  <blockquote>
  <p><strong>Note (continuous action space):</strong></p>
  </blockquote>
  <p>If you have continuous actions, then in almost all use cases the best approach is to discretize your actions. I can't think of an example where discretizing your actions will lead to a considerable deficiency.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-11T11:42:36.107Z
Answers.Text: More formally, you are suggesting to [discretise](https://en.wikipedia.org/wiki/Discretization) the continuous state space. Do you have any references to works/papers that apply a discretization of the state space? I think it might be helpful to cite them, if that's the case.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-11T12:17:46.457Z
Answers.Text: If action space is continuous you could model it by outputting a mean and variance of a proper distribution and then use the outputs to sample from the distribution parameterized by those outputs (ie. applying voltages).
Answers.UserDisplayName: Hanzy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-11T12:23:38.590Z
Answers.Text: For references using discretisation you can take a look at  [the Arcade Learning Environment paper](https://arxiv.org/pdf/1207.4708.pdf) which uses frame-skipping techniques (also experimented in the well-known [Playing Atari with Deep Reinforcement Learning paper](https://arxiv.org/pdf/1312.5602v1.pdf)). Current RL papers having video as input do the process at the level of each frame thanks to computation power being available.
Answers.UserDisplayName: HLeb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-11T11:39:10.367Z
Answers.LastActivityDate: 2020-11-21T21:31:01.930Z
Answers.LastEditDate: 2020-11-21T21:31:01.930Z
Answers.UserDisplayName: HLeb
Body: <p>Many examples work with a table-based method for Q-learning. This may be suitable for a discrete state (observation) or action space, like a robot in a grid world, but is there a way to use Q-learning for continuous spaces like the control of a pendulum?</p>
  
Tags: <reinforcement-learning><q-learning><dqn><continuous-action-spaces><continuous-state-spaces>
Title: Can Q-learning be used for continuous (state or action) spaces?
UserDisplayName: Bryan McGill
ThreadText: # Can Q-learning be used for continuous (state or action) spaces?
  
  (Asked by: username_0 on 2019-05-11)
  
  Many examples work with a table-based method for Q-learning. This may be suitable for a discrete state (observation) or action space, like a robot in a grid world, but is there a way to use Q-learning for continuous spaces like the control of a pendulum?
  
  **Tags:** reinforcement-learning, q-learning, dqn, continuous-action-spaces, continuous-state-spaces
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-05-11. Score: 13
  
  # Q-learning for continuous state spaces
  
  Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the space to a small number of states. Provided you can sample and update enough times, then a few million states is not a major problem.
  
  However, with large state spaces it is more common to use some form of function approximation for the action value. This is often noted $\hat{q}(s,a,\theta)$ to show that it is both an estimate (the circumflex over $\hat{q}$) and that you are learning some function parameters ($\theta$). There are broadly two popular approaches to Q-learning using function approximation:
  
  * Linear function approximation over a processed version of the state into features. A lot of variations to generate features have been proposed and tested, including Fourier series, tile coding, radial basic functions. The advantage of these methods are that they are simple, and more robust than non-linear function approximations. Which one to choose depends on what you state space represents and how the value function is likely to vary depending on location within the state space.
  * Neural network function approximation. This is essentially what Deep Q Networks (DQN) are. Provided you have a Markov state description, you scale it to work sensibly with neural networks, and you follow other DQN best practices (experience replay table, slow changing target network) this can work well.
  
  # Q-learning for continuous action spaces
  
  Unless you discretise the action space, then this becomes very unwieldy.
  
  The problem is that, given $s,a,r,s'$, Q-learning needs to evaluate the TD target:
  
  $$Q\_{target}(s,a) = r + \gamma \text{max}\_{a'} \hat{q}(s',a',\theta)$$
  
  The process for evaluating the maximum becomes less efficient and less accurate the larger the space that it needs to check.
  
  For somewhat large action spaces, using double Q-learning can help (with two estimates of Q, one to pick the target action, the other to estimate its value, which you alternate between on different steps) - this helps avoid maximisation bias where picking an action because it has the highest value and then using that highest value in calculations leads to over-estimating value.
  
  For very large or continuous action spaces, it is not usually practical to check all values. The alternative to Q-learning in this case is to use a policy gradient method such as Actor-Critic which can cope with very large or continuous action spaces, and does not rely on maximising over all possible actions in order to enact or evaluate a policy.
  
  ## Controlling a pendulum
  
  For a discrete action space e.g. applying one of a choice of forces on each time step, then this can be done using a DQN approach or any other function approximation. The classic example here might be an environment like Open AI's [CartPole-v1](https://gym.openai.com/envs/CartPole-v1/) where the state space is continuous, but there are only two possible actions. This can be solved easily using DQN, it is something of a beginner's problem.
  
  Adding continuous action space ends up with something like the [Pendulum-v0](https://gym.openai.com/envs/Pendulum-v0/) environment. This can be solved to some degree using DQN and discretising the action space (to e.g. 9 different actions). However, it is possible to make more optimal solutions using an Actor-Critic algorithm like A3C.
  
  > **Comments:**
  > **username_2**: How do you calculate q^(s′,a′,θ) for the training? My guess is to calculate backward from the end state of the pendulum. During the first run of the DQN, do you use one value for all q^(s′,a′,θ) or randomize them per example?  
  > **username_1**: @JoachimRives $\hat{q}(s',a',\theta)$ is the output of the neural network or other parametric estimator during training, and $\theta$ stands for the learning parameters. To calculate what the value *should* be trained to be - for DQN at least - that's the TD target, and the formula for that is shown in the answer. Wehn using DQN, it is important to recalculate the TD target every time you retrieve an example from replay memory.  
  > **username_2**: Do you mean the DQN TD is calculated backward or based on the expected value of succeeding steps leading to the terminal state?  
  > **username_1**: @JoachimRives The value is calculated as in the answer. TD learning (of which Q learning is an example) does not calculate backwards from the terminal state, instead it bootstraps from existing estimates that can be in the middle of a trajectory. Other algorithms do resolve backwards from terminal states - e.g. Monte Carlo control. There are already questions on this site that explain the difference further  
  
  ---
  
  ### Answer by username_3 on 2019-05-11. Score: 2
  
  > **Q-Learning for continuous state space**
  
  Reinforcement learning algorithms (e.g Q-Learning) can be applied to both discrete and continuous spaces. If you understand how it works in discrete mode, then you can easily move to continuous mode. That's why in the literature all the introductory material focuses on discrete mode, as it's easier to model (table, grid, etc.)
  
  Supposing you have a discrete number of actions, **the only difference** in a continuous space is that you will be modeling the state each $X$ amount of time ($X$ being a number you can choose depending on your use case). So, basically, you end up with a discrete space, but probably with an infinite number of states. You apply then the same approach you learned for discrete mode.
  
  Let's take the example of self-driving cars, at each $X$ms (e.g $X=1$), you'll be computing the state of the car which are your input features (e.g direction, orientation, rotation, distance to the pavement, relative position on the lane, etc.) and take a decision of the action to take as in discrete mode. The approach is the same in other use cases, like playing games, walking robots, and so on.
  
  > **Note (continuous action space):**
  
  If you have continuous actions, then in almost all use cases the best approach is to discretize your actions. I can't think of an example where discretizing your actions will lead to a considerable deficiency.
  
  > **Comments:**
  > **username_4**: More formally, you are suggesting to [discretise](https://en.wikipedia.org/wiki/Discretization) the continuous state space. Do you have any references to works/papers that apply a discretization of the state space? I think it might be helpful to cite them, if that's the case.  
  > **username_5**: If action space is continuous you could model it by outputting a mean and variance of a proper distribution and then use the outputs to sample from the distribution parameterized by those outputs (ie. applying voltages).  
  > **username_3**: For references using discretisation you can take a look at  [the Arcade Learning Environment paper](https://arxiv.org/pdf/1207.4708.pdf) which uses frame-skipping techniques (also experimented in the well-known [Playing Atari with Deep Reinforcement Learning paper](https://arxiv.org/pdf/1312.5602v1.pdf)). Current RL papers having video as input do the process at the level of each frame thanks to computation power being available.  
  
  ---
  
  