Answers.Body: <p>In Soft Actor-Critic the convergence of the iterative process is based on the principle of temporal difference (TD) learning and the properties of the soft Bellman backup operator. Specifically the algo aims to minimize the TD error via the soft Bellman backup operator <span class="math-container">$T^\pi$</span> (defined in Equation 2 of your reference) between successive estimates of the above LHS Q-function and its corresponding value function. Therefore it seems here the author is referring to S&amp;B book's standard TD learning convergence result by stochastic bootstrap approximation as stated in page 124:</p>
  <blockquote>
  <p>can we still guarantee convergence to the correct answer? Happily, the answer is yes. For any fixed policy <span class="math-container">$\pi$</span>, TD(0) has been proved to converge to <span class="math-container">$v_\pi$</span>, in the mean for a constant step-size parameter if it is sufficiently small, and with probability <span class="math-container">$1$</span> if the step-size parameter decreases according to the usual stochastic approximation conditions (2.7).</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-17T06:32:06.827Z
Answers.LastActivityDate: 2024-04-17T06:32:06.827Z
Answers.UserDisplayName: cinch
Body: <p>This comes from <strong>Soft Actor-Critic:Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor</strong>.</p>
  <p>Consider the soft Bellman backup operator<span class="math-container">$\tau^{\pi}$</span> in equation and a mapping <span class="math-container">$Q^{0}: S \times A \to \mathbb{R} 
   $</span> with <span class="math-container">$\left| \mathcal{A}   \right| &lt; \infty$</span>, and define <span class="math-container">$Q^{k+1} = \tau^{\pi}Q^{K}$</span>. Then the sequence <span class="math-container">$Q^{k}$</span> will converge to the soft Q-value of <span class="math-container">$\pi$</span> as <span class="math-container">$k\to\infty$</span>.</p>
  <p><span class="math-container">$$
  \tau^{\pi}Q(s_{t}, a_{t})\triangleq r(s_{t}, a_{t}) + \gamma\mathbb{E}_{s_{t+1} \sim p} [V(s_{t+1})]
  $$</span></p>
  <p>where</p>
  <p><span class="math-container">$$
  V(s_{t}) = \mathbb{E}_{a_{t}\sim \pi}[Q(s_{t}, a_{t}) - \log\pi(a_{t}|s_{t})]
  $$</span></p>
  <p><a href="https://i.sstatic.net/65lLy4HB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/65lLy4HB.png" alt="enter image description here" /></a></p>
  <p>The proof is confusing me.
  What's the standard convergence result for policy evaluation and why this is converged?</p>
  
Tags: <reinforcement-learning>
Title: Why soft Bellman backup operator is converged in tabular settings?
UserDisplayName: yeebo xie
ThreadText: # Why soft Bellman backup operator is converged in tabular settings?
  
  (Asked by: username_0 on 2024-04-15)
  
  This comes from **Soft Actor-Critic:Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor**.
  
  Consider the soft Bellman backup operator$\tau^{\pi}$ in equation and a mapping $Q^{0}: S \times A \to \mathbb{R}
  $ with $\left| \mathcal{A} \right| < \infty$, and define $Q^{k+1} = \tau^{\pi}Q^{K}$. Then the sequence $Q^{k}$ will converge to the soft Q-value of $\pi$ as $k\to\infty$.
  
  $$
  \tau^{\pi}Q(s\_{t}, a\_{t})\triangleq r(s\_{t}, a\_{t}) + \gamma\mathbb{E}\_{s\_{t+1} \sim p} [V(s\_{t+1})]
  $$
  
  where
  
  $$
  V(s\_{t}) = \mathbb{E}\_{a\_{t}\sim \pi}[Q(s\_{t}, a\_{t}) - \log\pi(a\_{t}|s\_{t})]
  $$
  
  [![enter image description here](https://i.sstatic.net/65lLy4HB.png)](https://i.sstatic.net/65lLy4HB.png)
  
  The proof is confusing me.
  What's the standard convergence result for policy evaluation and why this is converged?
  
  **Tags:** reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-04-17. Score: 0
  
  In Soft Actor-Critic the convergence of the iterative process is based on the principle of temporal difference (TD) learning and the properties of the soft Bellman backup operator. Specifically the algo aims to minimize the TD error via the soft Bellman backup operator $T^\pi$ (defined in Equation 2 of your reference) between successive estimates of the above LHS Q-function and its corresponding value function. Therefore it seems here the author is referring to S&B book's standard TD learning convergence result by stochastic bootstrap approximation as stated in page 124:
  
  > can we still guarantee convergence to the correct answer? Happily, the answer is yes. For any fixed policy $\pi$, TD(0) has been proved to converge to $v\_\pi$, in the mean for a constant step-size parameter if it is sufficiently small, and with probability $1$ if the step-size parameter decreases according to the usual stochastic approximation conditions (2.7).
  
  
  ---
  
  