Body: <p>In section 5 of the paper <a href="https://arxiv.org/abs/1812.05905" rel="nofollow noreferrer">“Soft Actor Critic Algorithms and Applications”</a>, the authors propose to optimize the policy subject to the constraints that the entropy of action distribution should be greater than a specific value <span class="math-container">$H_0$</span>.</p>
  <p><span class="math-container">$
  \text{argmax}_{\pi}{\left[\sum_{t=0}^{T}{r(s_t,a_t)}\right]}\ s.t.\ \mathbb{E}\left[-\log{\pi(a_t|s_t)} \right]\geq H_0\ \forall t
  $</span></p>
  <p>This is then converted to a dual problem, and the temperature parameter <span class="math-container">$\alpha$</span> is essentially the dual variable in Lagrange function. However, I don’t know why the authors use only a single dual variable <span class="math-container">$\alpha$</span>. Since the constraint applies to all possible <span class="math-container">$t$</span>, the Lagrange function should be:
  <span class="math-container">$
  L = \sum_{t=0}^{T}{r(s_t,a_t)} + \sum_{t=0}^{T}{\alpha_t \cdot(\mathbb{E}_{a_t\sim \pi, s_t\sim p_s}{\left[-\log(\pi(a_t|s_t))\right]-H_0)}}
  $</span></p>
  <p>And there should be multiple <span class="math-container">$\alpha_t$</span> to solve. Mathematically, how could we end up in optimizing only a single <span class="math-container">$\alpha$</span> in the algorithm?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-06T00:03:28.340Z
Comments.Text: The temperature doesn’t vary per time step. Look at the original paper where they don’t introduce the entropy tuning — it is a fixed hyper parameter. When they introduce the tuning of the temperature, it is still a single scalar value we’d like to learn, it doesn’t vary by time.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-06T10:29:57.697Z
Comments.Text: @DavidIreland The two versions of SAC uses two different objective functions. The first version of SAC (https://arxiv.org/abs/1801.01290) maximize the weighted sum of reward and entropy, and there is only one alpha. The second version (https://arxiv.org/abs/1812.05905) uses another objective to tune alpha: maximize the sum of reawrd subject to contraints in action entropy at each timestep. So in the Lagrange function there should be a different alpha for each timestep. Conceptiually I know there should be one alpha, but I just don't know why using a single alpha is mathematically correct.
Comments.UserDisplayName: Cloudy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-06T10:41:27.593Z
Comments.Text: The optimisation objective that they introduce for $\alpha$ is minimised at each time step. So, there should be only one dual variable, it is just that we solve a _new_ objective (depending on $t$) for each time step. So, to relate to your question, we actually solve a different constrained optimisation problem at each time step, where each optimisation problem has a single dual variable.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><deep-rl><actor-critic-methods><soft-actor-critic><constrained-optimization>
Title: Why do Soft Actor-Critic with automatic temperature tuning use only a single dual variable?
UserDisplayName: Cloudy
ThreadText: # Why do Soft Actor-Critic with automatic temperature tuning use only a single dual variable?
  
  (Asked by: username_0 on 2023-02-05)
  
  In section 5 of the paper [“Soft Actor Critic Algorithms and Applications”](https://arxiv.org/abs/1812.05905), the authors propose to optimize the policy subject to the constraints that the entropy of action distribution should be greater than a specific value $H\_0$.
  
  $
  \text{argmax}\_{\pi}{\left[\sum\_{t=0}^{T}{r(s\_t,a\_t)}\right]}\ s.t.\ \mathbb{E}\left[-\log{\pi(a\_t|s\_t)} \right]\geq H\_0\ \forall t
  $
  
  This is then converted to a dual problem, and the temperature parameter $\alpha$ is essentially the dual variable in Lagrange function. However, I don’t know why the authors use only a single dual variable $\alpha$. Since the constraint applies to all possible $t$, the Lagrange function should be:
  $
  L = \sum\_{t=0}^{T}{r(s\_t,a\_t)} + \sum\_{t=0}^{T}{\alpha\_t \cdot(\mathbb{E}\_{a\_t\sim \pi, s\_t\sim p\_s}{\left[-\log(\pi(a\_t|s\_t))\right]-H\_0)}}
  $
  
  And there should be multiple $\alpha\_t$ to solve. Mathematically, how could we end up in optimizing only a single $\alpha$ in the algorithm?
  
  **Tags:** reinforcement-learning, deep-rl, actor-critic-methods, soft-actor-critic, constrained-optimization
  
  > **Comments:**
  > **username_1**: The temperature doesn’t vary per time step. Look at the original paper where they don’t introduce the entropy tuning — it is a fixed hyper parameter. When they introduce the tuning of the temperature, it is still a single scalar value we’d like to learn, it doesn’t vary by time.  
  > **username_0**: @DavidIreland The two versions of SAC uses two different objective functions. The first version of SAC (https://arxiv.org/abs/1801.01290) maximize the weighted sum of reward and entropy, and there is only one alpha. The second version (https://arxiv.org/abs/1812.05905) uses another objective to tune alpha: maximize the sum of reawrd subject to contraints in action entropy at each timestep. So in the Lagrange function there should be a different alpha for each timestep. Conceptiually I know there should be one alpha, but I just don't know why using a single alpha is mathematically correct.  
  > **username_1**: The optimisation objective that they introduce for $\alpha$ is minimised at each time step. So, there should be only one dual variable, it is just that we solve a _new_ objective (depending on $t$) for each time step. So, to relate to your question, we actually solve a different constrained optimisation problem at each time step, where each optimisation problem has a single dual variable.  
  
  ---
  
  ## 0 answers
  
  