Answers.Body: <p>Since the question may not be answered unambiguously in general, I will use the given example as a guide. As you correctly write, a large dimensionality leads to a very large solution space because of the curse of dimensions.</p>
  <p>However, whether one restricts this solution space by allowing only discrete values <span class="math-container">$[0.0, 0.1, 0.2, ..., 1.0]$</span> for the three actions or by using the resulting combinations as a list of options (one dimension) makes no difference with respect to solution space coverage.
  Depending on which inputs the model receives, however, it would be conceivable that the separate variables (option 1) could be more easily assigned and learned, e.g. if there are sensors for the state, temperature, etc. of each of the 3 heat storages as inputs. Option 2 would still require the model to implicitly learn one-dimensional coding to target specific heat storages.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-15T16:19:29.920Z
Answers.Text: Yes, in the given example, I would use the 3 actions variables separately. Of course, it depends on the problem, e.g. if you have only 2 or 3 states for each of the 3 actions a combination could be more useful. Depending on the complexity of the control problem, it could become difficult to identify and exclude all forbidden cases and be easier to use a constraint function. Especially, if the validity not only depends on the produces outputs, but also on the current state (so a output could be valid but is not valid for all cases). However, it depends ;)
Answers.UserDisplayName: dexteritas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-15T16:22:42.833Z
Answers.Text: Thanks for your comment dexteritas. What do you mean by "constraint function"? A function that is influecing the reward in a negative way if a constraint violation has occured or a super-ordinate controller that can overrule the inital actions of the RL agent?
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-15T16:32:53.447Z
Answers.Text: You have a [constrained optimization](https://www.wikiwand.com/en/Constrained_optimization) problem. There are different approaches for that. Giving a negative reward would be a *penalty function*. I think it is useful, if the agent can learn what is valid/allowed. You could use e.g. penalty function in combination with super-ordinate controller to ensure hard constraints are not violated. But that could be a separate question.
Answers.UserDisplayName: dexteritas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-15T16:54:08.937Z
Answers.Text: Thanks dexteritas for your comment. I upvoted your answer (but I will not yet accept it as I would like to see whether others also share their opinion; after some time I might also accept it). Thanks for your insights. I appreciate it.
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-26T08:22:30.287Z
Answers.Text: As nobody other than you answered my question, I have now also accepted your answer (I had upvoted it before). I appreciate it.
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-15T15:57:12.577Z
Answers.Text: Thanks dexteritas for your answer. So you would tend not to reduce the dimensionality but just use the 3 action variables separately? One big advantage of transforming them into a one dimensional action space is that you could map this one action to only valid options in a control problem and thus tell the agent some rules directly independant from the reward function (see https://ai.stackexchange.com/questions/32042/is-it-possible-to-tell-the-reinforcement-learning-agent-some-rules-directly-with). But I also understand your point. Most probably it will depend on the application.
Answers.UserDisplayName: PeterBe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-15T15:42:07.070Z
Answers.LastActivityDate: 2021-11-15T15:42:07.070Z
Answers.UserDisplayName: dexteritas
Body: <p>In supervised or unsupervised learning, it is advised to reduce the dimensionality due to the curse of dimensionality in general.</p>
  <p>Is this also generally advisable for the action space of reinforcement learning?</p>
  <p>As far as I understand (and inspired by the answer here <a href="https://ai.stackexchange.com/q/32042/2444">Is it possible to tell the Reinforcement Learning agent some rules directly without any constraints</a>), you can reduce the dimensionality of the action space always to 1, meaning that you solely have 1 action. This can be done by using a mapping (e.g. in the step function of Open AI Gym).</p>
  <p>Let's have a look at an example: We have a heating device that can heat 3 storages and we have a discrete action variable for all of them with 11 steps [0.0, 0.1, 0.2, ..., 1.0]. So we have</p>
  <ul>
  <li>action_heatStorage1: [0.0, 0.1, 0.2, ..., 1.0]</li>
  <li>action_heatStorage2: [0.0, 0.1, 0.2, ..., 1.0]</li>
  <li>action_heatStorage3: [0.0, 0.1, 0.2, ..., 1.0]</li>
  </ul>
  <p>In this case, we would have a 3-dimensional action space</p>
  <ul>
  <li>action = [action_heatStorage1, action_heatStorage2,
  action_heatStorage3]</li>
  </ul>
  <p>However, it is also possible to combine the 3 actions  into 1 action variable &quot;action_combined&quot; of the size [11 * 11 * 11=1331] by just using a mapping of this one action into the separate 3 actions. For example like this:</p>
  <ul>
  <li>action_combined = 0 --&gt; action_heatStorage1 =0, action_heatStorage2
  =0, action_heatStorage3 =0</li>
  <li>action_combined = 1 --&gt; action_heatStorage1 =0.1, action_heatStorage2
  =0, action_heatStorage3 =0</li>
  <li>action_combined = 2 --&gt; action_heatStorage1 =0.2, action_heatStorage2
  =0, action_heatStorage3 =0</li>
  </ul>
  <p>...</p>
  <ul>
  <li>action_combined = 1331 --&gt; action_heatStorage1 =1.0, action_heatStorage2
  =1.0, action_heatStorage3 =1.0</li>
  </ul>
  <p>Is it generally advisable to reduce the dimensionality of the action space (Option 2), or to use multidimensional action variables directly (Option 1)?</p>
  <p>I know that the is most probably not an answer that is valid for all problems. But, as I am relatively new to reinforcement learning, I would like to know whether in the theory of reinforcement learning there is a general recommendation to do something like this or not or whether this question can't be answered in general as it is something that totally depends on the application and should be tested for each application individually?</p>
  <p><strong>Reminder</strong>: I have already received a good answer. Still, I would like to remind you on this question to maybe hear also the opinion and experience of others regarding this topic.</p>
  
Tags: <reinforcement-learning><action-spaces><dimensionality-reduction>
Title: Is it generally advisable to have a low dimensional action space in Reinforcement Learning?
UserDisplayName: PeterBe
ThreadText: # Is it generally advisable to have a low dimensional action space in Reinforcement Learning?
  
  (Asked by: username_0 on 2021-11-11)
  
  In supervised or unsupervised learning, it is advised to reduce the dimensionality due to the curse of dimensionality in general.
  
  Is this also generally advisable for the action space of reinforcement learning?
  
  As far as I understand (and inspired by the answer here [Is it possible to tell the Reinforcement Learning agent some rules directly without any constraints](https://ai.stackexchange.com/q/32042/2444)), you can reduce the dimensionality of the action space always to 1, meaning that you solely have 1 action. This can be done by using a mapping (e.g. in the step function of Open AI Gym).
  
  Let's have a look at an example: We have a heating device that can heat 3 storages and we have a discrete action variable for all of them with 11 steps [0.0, 0.1, 0.2, ..., 1.0]. So we have
  
  * action\_heatStorage1: [0.0, 0.1, 0.2, ..., 1.0]
  * action\_heatStorage2: [0.0, 0.1, 0.2, ..., 1.0]
  * action\_heatStorage3: [0.0, 0.1, 0.2, ..., 1.0]
  
  In this case, we would have a 3-dimensional action space
  
  * action = [action\_heatStorage1, action\_heatStorage2,
    action\_heatStorage3]
  
  However, it is also possible to combine the 3 actions into 1 action variable "action\_combined" of the size [11 \* 11 \* 11=1331] by just using a mapping of this one action into the separate 3 actions. For example like this:
  
  * action\_combined = 0 --> action\_heatStorage1 =0, action\_heatStorage2
    =0, action\_heatStorage3 =0
  * action\_combined = 1 --> action\_heatStorage1 =0.1, action\_heatStorage2
    =0, action\_heatStorage3 =0
  * action\_combined = 2 --> action\_heatStorage1 =0.2, action\_heatStorage2
    =0, action\_heatStorage3 =0
  
  ...
  
  * action\_combined = 1331 --> action\_heatStorage1 =1.0, action\_heatStorage2
    =1.0, action\_heatStorage3 =1.0
  
  Is it generally advisable to reduce the dimensionality of the action space (Option 2), or to use multidimensional action variables directly (Option 1)?
  
  I know that the is most probably not an answer that is valid for all problems. But, as I am relatively new to reinforcement learning, I would like to know whether in the theory of reinforcement learning there is a general recommendation to do something like this or not or whether this question can't be answered in general as it is something that totally depends on the application and should be tested for each application individually?
  
  **Reminder**: I have already received a good answer. Still, I would like to remind you on this question to maybe hear also the opinion and experience of others regarding this topic.
  
  **Tags:** reinforcement-learning, action-spaces, dimensionality-reduction
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-15. Score: 1 | Accepted answer
  
  Since the question may not be answered unambiguously in general, I will use the given example as a guide. As you correctly write, a large dimensionality leads to a very large solution space because of the curse of dimensions.
  
  However, whether one restricts this solution space by allowing only discrete values $[0.0, 0.1, 0.2, ..., 1.0]$ for the three actions or by using the resulting combinations as a list of options (one dimension) makes no difference with respect to solution space coverage.
  Depending on which inputs the model receives, however, it would be conceivable that the separate variables (option 1) could be more easily assigned and learned, e.g. if there are sensors for the state, temperature, etc. of each of the 3 heat storages as inputs. Option 2 would still require the model to implicitly learn one-dimensional coding to target specific heat storages.
  
  > **Comments:**
  > **username_0**: Thanks dexteritas for your answer. So you would tend not to reduce the dimensionality but just use the 3 action variables separately? One big advantage of transforming them into a one dimensional action space is that you could map this one action to only valid options in a control problem and thus tell the agent some rules directly independant from the reward function (see https://ai.stackexchange.com/questions/32042/is-it-possible-to-tell-the-reinforcement-learning-agent-some-rules-directly-with). But I also understand your point. Most probably it will depend on the application.  
  > **username_1**: Yes, in the given example, I would use the 3 actions variables separately. Of course, it depends on the problem, e.g. if you have only 2 or 3 states for each of the 3 actions a combination could be more useful. Depending on the complexity of the control problem, it could become difficult to identify and exclude all forbidden cases and be easier to use a constraint function. Especially, if the validity not only depends on the produces outputs, but also on the current state (so a output could be valid but is not valid for all cases). However, it depends ;)  
  > **username_0**: Thanks for your comment dexteritas. What do you mean by "constraint function"? A function that is influecing the reward in a negative way if a constraint violation has occured or a super-ordinate controller that can overrule the inital actions of the RL agent?  
  > **username_1**: You have a [constrained optimization](https://www.wikiwand.com/en/Constrained_optimization) problem. There are different approaches for that. Giving a negative reward would be a *penalty function*. I think it is useful, if the agent can learn what is valid/allowed. You could use e.g. penalty function in combination with super-ordinate controller to ensure hard constraints are not violated. But that could be a separate question.  
  > **username_0**: Thanks dexteritas for your comment. I upvoted your answer (but I will not yet accept it as I would like to see whether others also share their opinion; after some time I might also accept it). Thanks for your insights. I appreciate it.  
  > **username_0**: As nobody other than you answered my question, I have now also accepted your answer (I had upvoted it before). I appreciate it.  
  
  ---
  
  