Answers.Body: <p>The square brackets are part of the <a href="https://www.statlect.com/fundamentals-of-probability/expected-value" rel="nofollow noreferrer"><strong>expectation</strong></a> <a href="https://en.wikipedia.org/wiki/Operator_(mathematics)" rel="nofollow noreferrer">operator</a> (i.e. a function of a random variable, which in this case is <span class="math-container">$G_t$</span>). This is common notation for the expectation. So, it's not <span class="math-container">$\left[ X \right]$</span>, but rather <span class="math-container">$\mathbb{E}\left[ \cdot \right]$</span> or <span class="math-container">$\mathbb{E}\left[ X \right]$</span>, for some random variable <span class="math-container">$X$</span>. This notation is similar to the notation of a function <span class="math-container">$f(\cdot)$</span> or <span class="math-container">$f(x)$</span>, but we use square brackets because expectations are taken with respect to random variables, which are actually functions (if this is too confusing, just ignore these details for now).</p>
  <p>The <span class="math-container">$\mid$</span> is also common notation and means that we condition on knowing <span class="math-container">$S_t = s$</span> (an event). If you are not familiar with conditional expectations and probability distributions, you can take a look at them e.g. <a href="https://www.statlect.com/fundamentals-of-probability/conditional-expectation" rel="nofollow noreferrer">here</a>.</p>
  <p>So, you can read your formula as</p>
  <blockquote>
  <p>the conditional expectation of <span class="math-container">$G_t$</span> (the return, i.e. the <em>sum of future rewards</em>), given that we know that the state at time <span class="math-container">$t$</span> is <span class="math-container">$s$</span> (i.e. the condition).</p>
  </blockquote>
  <p>This is indeed the definition of the state value function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-11T06:24:26.253Z
Answers.Text: Thanks for your patience, I have a similar question. $\sum_{r}\sum_{s'}p(s',r|s,a)[r+\gamma V\pi(s')]$, the $p(..|..)$ is probability, what's the meaning of the follow square bracket part $[r+\gamma V\pi(s')]$? Is it means the $p(..|..)$ multiply by $[r+\gamma V\pi(s')]$?
Answers.UserDisplayName: huang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-12T06:10:07.503Z
Answers.Text: I found that this is really a multiplication operation. This notation inconsistency confuses me very much.
Answers.UserDisplayName: huang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-12T09:03:53.277Z
Answers.Text: @JoeHuang Yes. Sorry for not answering immediately. Here, $[ \cdot]$ would be like $( \cdot )$, so, yes, it's just a multiplication. They probably used square brackets because, if you turn this equation into an expectation, you will see that it will be equivalent to an expectation of a random variable that looks like what is inside the square brackets. See my blog post [here](https://nbro.gitlab.io/blogging/2022/02/16/bellman-equations/). It should clarify what I've just said.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-06T17:17:27.393Z
Answers.LastActivityDate: 2022-03-06T17:17:27.393Z
Answers.UserDisplayName: nbro
Body: <p>Here is the formula of state-value function in Reinforcement Learning.</p>
  <p><a href="https://i.sstatic.net/B5gxF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/B5gxF.png" alt="enter image description here" /></a></p>
  <p>What do the square brackets <span class="math-container">$[ ]$</span> and <span class="math-container">$\mid$</span> mean in <span class="math-container">$[G_t \mid S_t=s]$</span>? Why use square brackets? Why use <span class="math-container">$\mid$</span>?</p>
  <p>Why do mathematical formulas use all kinds of ambiguous symbols? Not as unique as the symbols of programming language?</p>
  
Tags: <reinforcement-learning><math><notation>
Title: What do the square brackets $[ ]$ and $\mid$ mean in $[G_t \mid S_t=s]$?
UserDisplayName: huang
ThreadText: # What do the square brackets $[ ]$ and $\mid$ mean in $[G_t \mid S_t=s]$?
  
  (Asked by: username_0 on 2022-03-06)
  
  Here is the formula of state-value function in Reinforcement Learning.
  
  [![enter image description here](https://i.sstatic.net/B5gxF.png)](https://i.sstatic.net/B5gxF.png)
  
  What do the square brackets $[ ]$ and $\mid$ mean in $[G\_t \mid S\_t=s]$? Why use square brackets? Why use $\mid$?
  
  Why do mathematical formulas use all kinds of ambiguous symbols? Not as unique as the symbols of programming language?
  
  **Tags:** reinforcement-learning, math, notation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-06. Score: 3 | Accepted answer
  
  The square brackets are part of the [**expectation**](https://www.statlect.com/fundamentals-of-probability/expected-value) [operator](https://en.wikipedia.org/wiki/Operator_(mathematics)) (i.e. a function of a random variable, which in this case is $G\_t$). This is common notation for the expectation. So, it's not $\left[ X \right]$, but rather $\mathbb{E}\left[ \cdot \right]$ or $\mathbb{E}\left[ X \right]$, for some random variable $X$. This notation is similar to the notation of a function $f(\cdot)$ or $f(x)$, but we use square brackets because expectations are taken with respect to random variables, which are actually functions (if this is too confusing, just ignore these details for now).
  
  The $\mid$ is also common notation and means that we condition on knowing $S\_t = s$ (an event). If you are not familiar with conditional expectations and probability distributions, you can take a look at them e.g. [here](https://www.statlect.com/fundamentals-of-probability/conditional-expectation).
  
  So, you can read your formula as
  
  > the conditional expectation of $G\_t$ (the return, i.e. the *sum of future rewards*), given that we know that the state at time $t$ is $s$ (i.e. the condition).
  
  This is indeed the definition of the state value function.
  
  > **Comments:**
  > **username_0**: Thanks for your patience, I have a similar question. $\sum_{r}\sum_{s'}p(s',r|s,a)[r+\gamma V\pi(s')]$, the $p(..|..)$ is probability, what's the meaning of the follow square bracket part $[r+\gamma V\pi(s')]$? Is it means the $p(..|..)$ multiply by $[r+\gamma V\pi(s')]$?  
  > **username_0**: I found that this is really a multiplication operation. This notation inconsistency confuses me very much.  
  > **username_1**: @JoeHuang Yes. Sorry for not answering immediately. Here, $[ \cdot]$ would be like $( \cdot )$, so, yes, it's just a multiplication. They probably used square brackets because, if you turn this equation into an expectation, you will see that it will be equivalent to an expectation of a random variable that looks like what is inside the square brackets. See my blog post [here](https://username_1.gitlab.io/blogging/2022/02/16/bellman-equations/). It should clarify what I've just said.  
  
  ---
  
  