Answers.Body: <p>In the Sutton and Barto book <span class="math-container">$q(s,a)$</span> is used to denote the <em>true</em> expected value of taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span>, whereas capital <span class="math-container">$Q(s,a)$</span> is used to denote an <em>estimate</em> of <span class="math-container">$q(s,a)$</span>. However, there is likely to be a lot of inconsistency in the literature as each author has their own preference on how to denote things. I would encourage you to consider whether the value you are reading is to denote an estimate or the true value. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-10T10:17:58.523Z
Answers.LastActivityDate: 2020-06-10T10:17:58.523Z
Answers.UserDisplayName: David
Answers.Body: <h1>Ordinary variables vs Random Variables</h1>
  
  <p>The difference is whether you're talking about a ordinary variable or a random variable.</p>
  
  <p>For instance, the q-function (lowercase) is an expectation value (i.e. not a random variable), conditioned on a specific state-action pair:
  <span class="math-container">$$
  q(s,a)\ =\ \mathbb{E}_t\left\{ R_t+\gamma R_{t+1} + \gamma^2R_{t+2}+\dots\,\Big|\, S_t=s, A_t=a \right\}
  $$</span>
  Then, in some case, some authors may abuse notation slightly by feeding in a random variable into the q-function, e.g. <span class="math-container">$q(S_t,a)$</span>, <span class="math-container">$q(s,A_t)$</span> or even <span class="math-container">$q(S_t,A_t)$</span>, thereby <em>undoing</em> some or all of the conditioning in the definition of the q-function as an expectation value.</p>
  
  <p>Feeding a random variable into a function like the q-function results in an output that is a random variable in its own right. It is for this reason that some authors choose to give the function itself an uppercase letter as well.</p>
  
  <p>My advice would be to think to yourself, <em>is this a random variable?</em> For the rest, I would interpret upper/lowercase as no more than a hint to the reader.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-10T09:50:22.637Z
Answers.Text: to add to your answer, capital Q is typically used when it is an estimate of the true q-value function. this follows from what you are saying as if it is an estimate then it is a stochastic approximation i.e. a random variable.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-10T10:15:05.327Z
Answers.Text: It's also worth mentioning that the notation in the first edition of Barto and Sutton's wasn't very consistent, but it should have improved in the second edition.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T02:55:39.763Z
Answers.Text: @Kris, could you maybe clarify what would passing a random variable into a function q mean?
Answers.UserDisplayName: d56
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-10T04:45:57.627Z
Answers.LastActivityDate: 2020-06-10T04:45:57.627Z
Answers.UserDisplayName: Kris
Body: <p>Why are the state-value and action-value functions are sometimes written in small letters and other times in capitals? For instance, why in the Q-learning algorithm (<a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">page 131 of Barto and Sutton's book</a> but not only), we the capitals are used <span class="math-container">$Q(S, A)$</span>, while the Bellman equation it is <span class="math-container">$q(s,a)$</span>? </p>
  
Tags: <reinforcement-learning><value-functions><notation>
Title: Why are the value functions sometimes written with capital letters and other times with lower-case letters?
UserDisplayName: d56
ThreadText: # Why are the value functions sometimes written with capital letters and other times with lower-case letters?
  
  (Asked by: username_0 on 2020-06-10)
  
  Why are the state-value and action-value functions are sometimes written in small letters and other times in capitals? For instance, why in the Q-learning algorithm ([page 131 of Barto and Sutton's book](http://incompleteideas.net/book/the-book-2nd.html) but not only), we the capitals are used $Q(S, A)$, while the Bellman equation it is $q(s,a)$?
  
  **Tags:** reinforcement-learning, value-functions, notation
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-06-10. Score: 4
  
  In the Sutton and Barto book $q(s,a)$ is used to denote the *true* expected value of taking action $a$ in state $s$, whereas capital $Q(s,a)$ is used to denote an *estimate* of $q(s,a)$. However, there is likely to be a lot of inconsistency in the literature as each author has their own preference on how to denote things. I would encourage you to consider whether the value you are reading is to denote an estimate or the true value.
  
  
  ---
  
  ### Answer by username_2 on 2020-06-10. Score: 1
  
  # Ordinary variables vs Random Variables
  
  The difference is whether you're talking about a ordinary variable or a random variable.
  
  For instance, the q-function (lowercase) is an expectation value (i.e. not a random variable), conditioned on a specific state-action pair:
  $$
  q(s,a)\ =\ \mathbb{E}\_t\left\{ R\_t+\gamma R\_{t+1} + \gamma^2R\_{t+2}+\dots\,\Big|\, S\_t=s, A\_t=a \right\}
  $$
  Then, in some case, some authors may abuse notation slightly by feeding in a random variable into the q-function, e.g. $q(S\_t,a)$, $q(s,A\_t)$ or even $q(S\_t,A\_t)$, thereby *undoing* some or all of the conditioning in the definition of the q-function as an expectation value.
  
  Feeding a random variable into a function like the q-function results in an output that is a random variable in its own right. It is for this reason that some authors choose to give the function itself an uppercase letter as well.
  
  My advice would be to think to yourself, *is this a random variable?* For the rest, I would interpret upper/lowercase as no more than a hint to the reader.
  
  > **Comments:**
  > **username_1**: to add to your answer, capital Q is typically used when it is an estimate of the true q-value function. this follows from what you are saying as if it is an estimate then it is a stochastic approximation i.e. a random variable.  
  > **username_3**: It's also worth mentioning that the notation in the first edition of Barto and Sutton's wasn't very consistent, but it should have improved in the second edition.  
  > **username_0**: @Kris, could you maybe clarify what would passing a random variable into a function q mean?  
  
  ---
  
  