Answers.Body: <p><strong>TLDR;</strong></p>
  <p><em>The output of Q-value function <strong>will</strong> eventually <strong>saturate</strong>. Can't say when, but it will surely do.</em></p>
  <h2>Detailed Answer</h2>
  <blockquote>
  <p>if you don't have TD error</p>
  </blockquote>
  <p>It meant, if I don't have logs of the error.</p>
  <blockquote>
  <p><span class="math-container">$Q(s, a)$</span> vs Episodes Graph</p>
  </blockquote>
  <p>To understand, how <span class="math-container">$Q(s, a)$</span> behaves as the episodes increase.</p>
  <p>I was under a <em>wrong</em> impression that, the <span class="math-container">$Q(s, a)$</span> will saturate around the values given by the reward function.</p>
  <p>As evident from Loss vs Training Episodes Curve, we can see that loss (TD Error) is almost saturated. <a href="https://i.sstatic.net/YTHGL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YTHGL.png" alt="enter image description here" /></a></p>
  <p>However, <span class="math-container">$Q(s, a)$</span> vs training episodes curve is not saturated yet.
  <a href="https://i.sstatic.net/AOnIH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AOnIH.png" alt="enter image description here" /></a></p>
  <p>The only explanation for the above two graphs could be given as follows:
  The target estimate (<span class="math-container">$r + \gamma Q(s', a')$</span>) and <span class="math-container">$Q(s, a)$</span> are almost similar due to which, the error is quite low. But, <span class="math-container">$Q(s, a)$</span> is still nowhere near optimal value <span class="math-container">$Q^{*}(s, a)$</span>.</p>
  <p>Hence, I gave it another shot and made it run for twice training episodes i.e. 20000, and below are the results.</p>
  <p>A nearly saturated loss
  <a href="https://i.sstatic.net/5icLQ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5icLQ.png" alt="enter image description here" /></a></p>
  <p>and a nearly saturating <span class="math-container">$Q(s, a)$</span>.
  <a href="https://i.sstatic.net/pLF04.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pLF04.png" alt="enter image description here" /></a>.</p>
  <p>Note that, the value of <span class="math-container">$Q(s, a)$</span> saturating is around 250 - 300 (will run it for more iterations) and it is nowhere around the reward values <span class="math-container">$ \in [-100, 35]$</span>.</p>
  <p>Hence, the <span class="math-container">$Q(s, a)$</span> vs episodes will saturate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-25T16:36:29.877Z
Answers.LastActivityDate: 2021-01-25T16:36:29.877Z
Answers.UserDisplayName: Anubhav Sachan
Body: <p>I'm employing the Actor-Critic algorithm. The critic network approximates the action-value function, i.e. <span class="math-container">$Q(s, a)$</span>, which determines how <em>good</em> a particular state is, when provided with an action.</p>
  <p><span class="math-container">$Q(s, a)$</span> is approximated using the backpropagation of the temporal difference error (TD error). We can understand that <span class="math-container">$Q(s, a)$</span> has been approximated properly when TD error is minimized, i.e. when it is saturated at lower values.</p>
  <p>My question is, when exactly can you say that <span class="math-container">$Q(s, a)$</span> is approximated properly, if you don't have TD error, i.e. if you have to plot the graph between <span class="math-container">$Q(s, a)$</span> vs episodes, then what would be the optimal behaviour?</p>
  <p>Will it be increasing exponential with saturation value around reward values, or increasing exponential with saturation (around any value)?</p>
  <p>Follow up: What can be the possible mistake, if the output of Q-value function is around 5x the rewards, and not saturating?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T12:46:11.980Z
Comments.Text: Which actor-critic algorithm have you implemented exactly? From which paper or book? I suppose you're using a neural network to approximate $Q(s, a)$, given your description. Moreover, when you say "if you have to plot the graph between Q(s, a) vs episodes", what graph are you exactly talking about? Can you show an example of such a graph that you have plotted? Also, why don't you have the TD error?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T16:06:10.820Z
Comments.Text: I'm using the standard actor critic, where critic network is optimized using TD Error. I wanted to plot the values of $Q(s, a)$ (my network has 1 output neuron) vs training episodes, to know what are the possible values around which $Q(s, a)$ get saturated. However, my question is answered by little more experimentation and research. I'll be adding the answer soon.
Comments.UserDisplayName: Anubhav Sachan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T18:51:57.190Z
Comments.Text: So, are you using function approximation or not? In any case, I would suggest that you put the title _in the form of_ a question, your **specific** question. Right now, the title is not very clear. Also, what you wanted to plot, according to your answer, was the **mean** Q(s, a) value, but that alone doesn't mean much. The Q-value is a function of the state and action, so if you average the Q-values across all states and actions, that's probably meaningless.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T19:03:54.880Z
Comments.Text: Yes, I'm using function approximation. I'll take care of the title as well. @nbro, can elaborate on why averaging Q-values across all the states and actions is probably meaningless? My state is basically the channel coefficient H, and action is the amount of power transmitted in that H.
Comments.UserDisplayName: Anubhav Sachan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T19:08:03.677Z
Comments.Text: Because, for example, in one state $s_1$, the average value across all possible actions in that state may be 1, while the average value across all actions in state $s_2$ may be 1000. However, maybe, in both cases, the relative value of the actions in the specific state is the same, i.e. the optimal action is the same, the second best optimal action is also the same and so on. Of course, the values of the states depend on 1. how you initialize the Q-network, 2. how you explore the environment, 3. the learning update.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T19:24:12.470Z
Comments.Text: If the action space for a particular state $s_i$ is about a mean $\mu_{i}$, and $\mu_{i+1}$ is comparable to the previous $\mu_{i}$ (which is my case), and $\mu_{i+1} - \mu_{i} \in [0, 5]$, will taking mean of $Q(s_i, a)$ over all $i$'s still be useless?
Comments.UserDisplayName: Anubhav Sachan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T21:05:54.933Z
Comments.Text: I didn't yet read your answer, but what did you want to get from the mean of the Q-values across all states and actions? What do you think that represents? Wait, are you computing the mean across all states for a fixed $a$? If this is this last case, it probably makes some sense (but this depends on what you wanted to achieve with this).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-26T07:50:16.147Z
Comments.Text: The mean of Q-values across all the states and actions will denote the stability of the algorithm as used in [DQN Paper](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf). The figure 2c and 2d. There is another question posing similar concerns over [averaging all the q-values](https://stats.stackexchange.com/questions/355247/how-to-calculate-average-action-value-q-for-deep-q-learning).
Comments.UserDisplayName: Anubhav Sachan
Tags: <reinforcement-learning><q-learning><actor-critic-methods>
Title: Relationship between Rewards and Q Value (Graph between Q(s, a) vs episodes)
UserDisplayName: Anubhav Sachan
ThreadText: # Relationship between Rewards and Q Value (Graph between Q(s, a) vs episodes)
  
  (Asked by: username_0 on 2021-01-25)
  
  I'm employing the Actor-Critic algorithm. The critic network approximates the action-value function, i.e. $Q(s, a)$, which determines how *good* a particular state is, when provided with an action.
  
  $Q(s, a)$ is approximated using the backpropagation of the temporal difference error (TD error). We can understand that $Q(s, a)$ has been approximated properly when TD error is minimized, i.e. when it is saturated at lower values.
  
  My question is, when exactly can you say that $Q(s, a)$ is approximated properly, if you don't have TD error, i.e. if you have to plot the graph between $Q(s, a)$ vs episodes, then what would be the optimal behaviour?
  
  Will it be increasing exponential with saturation value around reward values, or increasing exponential with saturation (around any value)?
  
  Follow up: What can be the possible mistake, if the output of Q-value function is around 5x the rewards, and not saturating?
  
  **Tags:** reinforcement-learning, q-learning, actor-critic-methods
  > **Comments:**
  > **username_1**: Which actor-critic algorithm have you implemented exactly? From which paper or book? I suppose you're using a neural network to approximate $Q(s, a)$, given your description. Moreover, when you say "if you have to plot the graph between Q(s, a) vs episodes", what graph are you exactly talking about? Can you show an example of such a graph that you have plotted? Also, why don't you have the TD error?  
  > **username_0**: I'm using the standard actor critic, where critic network is optimized using TD Error. I wanted to plot the values of $Q(s, a)$ (my network has 1 output neuron) vs training episodes, to know what are the possible values around which $Q(s, a)$ get saturated. However, my question is answered by little more experimentation and research. I'll be adding the answer soon.  
  > **username_1**: So, are you using function approximation or not? In any case, I would suggest that you put the title _in the form of_ a question, your **specific** question. Right now, the title is not very clear. Also, what you wanted to plot, according to your answer, was the **mean** Q(s, a) value, but that alone doesn't mean much. The Q-value is a function of the state and action, so if you average the Q-values across all states and actions, that's probably meaningless.  
  > **username_0**: Yes, I'm using function approximation. I'll take care of the title as well. @nbro, can elaborate on why averaging Q-values across all the states and actions is probably meaningless? My state is basically the channel coefficient H, and action is the amount of power transmitted in that H.  
  > **username_1**: Because, for example, in one state $s_1$, the average value across all possible actions in that state may be 1, while the average value across all actions in state $s_2$ may be 1000. However, maybe, in both cases, the relative value of the actions in the specific state is the same, i.e. the optimal action is the same, the second best optimal action is also the same and so on. Of course, the values of the states depend on 1. how you initialize the Q-network, 2. how you explore the environment, 3. the learning update.  
  > **username_0**: If the action space for a particular state $s_i$ is about a mean $\mu_{i}$, and $\mu_{i+1}$ is comparable to the previous $\mu_{i}$ (which is my case), and $\mu_{i+1} - \mu_{i} \in [0, 5]$, will taking mean of $Q(s_i, a)$ over all $i$'s still be useless?  
  > **username_1**: I didn't yet read your answer, but what did you want to get from the mean of the Q-values across all states and actions? What do you think that represents? Wait, are you computing the mean across all states for a fixed $a$? If this is this last case, it probably makes some sense (but this depends on what you wanted to achieve with this).  
  > **username_0**: The mean of Q-values across all the states and actions will denote the stability of the algorithm as used in [DQN Paper](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf). The figure 2c and 2d. There is another question posing similar concerns over [averaging all the q-values](https://stats.stackexchange.com/questions/355247/how-to-calculate-average-action-value-q-for-deep-q-learning).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-01-25. Score: 1 | Accepted answer
  
  **TLDR;**
  
  *The output of Q-value function **will** eventually **saturate**. Can't say when, but it will surely do.*
  
  ## Detailed Answer
  
  > if you don't have TD error
  
  It meant, if I don't have logs of the error.
  
  > $Q(s, a)$ vs Episodes Graph
  
  To understand, how $Q(s, a)$ behaves as the episodes increase.
  
  I was under a *wrong* impression that, the $Q(s, a)$ will saturate around the values given by the reward function.
  
  As evident from Loss vs Training Episodes Curve, we can see that loss (TD Error) is almost saturated. [![enter image description here](https://i.sstatic.net/YTHGL.png)](https://i.sstatic.net/YTHGL.png)
  
  However, $Q(s, a)$ vs training episodes curve is not saturated yet.
  [![enter image description here](https://i.sstatic.net/AOnIH.png)](https://i.sstatic.net/AOnIH.png)
  
  The only explanation for the above two graphs could be given as follows:
  The target estimate ($r + \gamma Q(s', a')$) and $Q(s, a)$ are almost similar due to which, the error is quite low. But, $Q(s, a)$ is still nowhere near optimal value $Q^{\*}(s, a)$.
  
  Hence, I gave it another shot and made it run for twice training episodes i.e. 20000, and below are the results.
  
  A nearly saturated loss
  [![enter image description here](https://i.sstatic.net/5icLQ.png)](https://i.sstatic.net/5icLQ.png)
  
  and a nearly saturating $Q(s, a)$.
  [![enter image description here](https://i.sstatic.net/pLF04.png)](https://i.sstatic.net/pLF04.png).
  
  Note that, the value of $Q(s, a)$ saturating is around 250 - 300 (will run it for more iterations) and it is nowhere around the reward values $ \in [-100, 35]$.
  
  Hence, the $Q(s, a)$ vs episodes will saturate.
  
  
  ---
  
  