Answers.Body: <p>The answer is &quot;it depends&quot;. Once you have arranged the actions into order, a key trait is whether the action value function has a simple enough shape that sampling from a Gaussian policy function would give consistent expected returns, enough that learning can occur. If the underlying &quot;true&quot; value function has a lot of high frequency noise then learning would be slow. In the worst case, if the action value <span class="math-container">$Q(s,a_{n})$</span> and <span class="math-container">$Q(s,a_{n+1})$</span> is not correlated for any <span class="math-container">$n$</span>, then it will not be possible to learn with the approximation at all.</p>
  <p>You may have some sense of how similar actions <span class="math-container">$a_{n}$</span> and <span class="math-container">$a_{n+1}$</span> are. If the actions represent different ordinal choices, such as selecting an integer number of items to perform some task with such as buy/sell or transport, then in many environments there will often be a strong correlation between outcome of choosing e.g. <span class="math-container">$a_{900}$</span> and <span class="math-container">$a_{901}$</span>. If this holds in general, then that is a good indicator that you can treat <span class="math-container">$n$</span> as being continuous, and use learned parameters of a simple distribution function to find optimal policies (and in addition this could be far more efficient than using a discrete representation).</p>
  <p>It might not matter if for a small fraction of cases the difference in outcomes between <span class="math-container">$a_{n}$</span> and <span class="math-container">$a_{n+1}$</span> is large, provided that successive approximations to the optimal policy can improve towards optimal through adjusting mean <span class="math-container">$\mu(s)$</span> and standard deviation <span class="math-container">$\sigma(s)$</span>.</p>
  <p>There may still be difficult cases that cannot be learned by the approximation - for instance if a specific action <span class="math-container">$a_n$</span> is optimal for a given state <span class="math-container">$s$</span>, but <span class="math-container">$a_{n-1}$</span> and <span class="math-container">$a_{n+1}$</span> are a lot worse, then the training process of a typical policy gradient approach may never settle upon <span class="math-container">$\mu(s) = n$</span> and <span class="math-container">$\sigma(s) \approx 0$</span>, because any intermediate values between the starting policy and the optimal one will perform badly.</p>
  <p>For expanding into more dimensions, the same ideas apply to each dimension separately. You may want to use different distributions, or even have one dimension that uses a continuous model with a few parameters whilst another remains discrete with a free parameter for each choice.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-09T09:18:32.997Z
Answers.LastActivityDate: 2021-03-09T13:51:44.797Z
Answers.LastEditDate: 2021-03-09T13:51:44.797Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have a large 1D action space, e.g. dim(A)=2000-10000. Can I use continuous action space where I could learn the mean and std of the Gaussian distributions that I would use to sample action from and round the value to the nearest integer? If yes, can I extend this idea to multi-dimensional large action space?</p>
  
Tags: <reinforcement-learning><deep-rl><continuous-action-spaces><discrete-action-spaces>
Title: Can a large discrete action space be represented using Gaussian distributions?
UserDisplayName: Mika
ThreadText: # Can a large discrete action space be represented using Gaussian distributions?
  
  (Asked by: username_0 on 2021-03-09)
  
  I have a large 1D action space, e.g. dim(A)=2000-10000. Can I use continuous action space where I could learn the mean and std of the Gaussian distributions that I would use to sample action from and round the value to the nearest integer? If yes, can I extend this idea to multi-dimensional large action space?
  
  **Tags:** reinforcement-learning, deep-rl, continuous-action-spaces, discrete-action-spaces
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-09. Score: 4 | Accepted answer
  
  The answer is "it depends". Once you have arranged the actions into order, a key trait is whether the action value function has a simple enough shape that sampling from a Gaussian policy function would give consistent expected returns, enough that learning can occur. If the underlying "true" value function has a lot of high frequency noise then learning would be slow. In the worst case, if the action value $Q(s,a\_{n})$ and $Q(s,a\_{n+1})$ is not correlated for any $n$, then it will not be possible to learn with the approximation at all.
  
  You may have some sense of how similar actions $a\_{n}$ and $a\_{n+1}$ are. If the actions represent different ordinal choices, such as selecting an integer number of items to perform some task with such as buy/sell or transport, then in many environments there will often be a strong correlation between outcome of choosing e.g. $a\_{900}$ and $a\_{901}$. If this holds in general, then that is a good indicator that you can treat $n$ as being continuous, and use learned parameters of a simple distribution function to find optimal policies (and in addition this could be far more efficient than using a discrete representation).
  
  It might not matter if for a small fraction of cases the difference in outcomes between $a\_{n}$ and $a\_{n+1}$ is large, provided that successive approximations to the optimal policy can improve towards optimal through adjusting mean $\mu(s)$ and standard deviation $\sigma(s)$.
  
  There may still be difficult cases that cannot be learned by the approximation - for instance if a specific action $a\_n$ is optimal for a given state $s$, but $a\_{n-1}$ and $a\_{n+1}$ are a lot worse, then the training process of a typical policy gradient approach may never settle upon $\mu(s) = n$ and $\sigma(s) \approx 0$, because any intermediate values between the starting policy and the optimal one will perform badly.
  
  For expanding into more dimensions, the same ideas apply to each dimension separately. You may want to use different distributions, or even have one dimension that uses a continuous model with a few parameters whilst another remains discrete with a free parameter for each choice.
  
  
  ---
  
  