Answers.Body: <p>This answer assumes that you only have a problem with this notation from the article:</p>
  <blockquote>
  <p><span class="math-container">$r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$</span></p>
  </blockquote>
  <p>This is a standard notation, used in many disciplines, for defining a function and its input and output <em>domains</em>. It is a bit like the method signature for the function - it does not fully define it, but does enough to show how it can interact with other expressions.</p>
  <p>All functions can be thought of as maps between the input domain and output domain. You provide an input value, and it returns an output value. The values can be arbitrary mathematical objects. To show what kind of objects the inputs and outputs are allowed to be, the <a href="https://www.mathsisfun.com/sets/symbols.html" rel="nofollow noreferrer">notation for sets</a> is used.</p>
  <p>Importantly the symbol <span class="math-container">$\mathbb{R}$</span> at the end does not refer to the set of possible rewards in the environment (although it is a reward function, and that will be its output), but the set of all real numbers, because a reward is always a real number*.</p>
  <p>As a concrete example, if you had the function <span class="math-container">$f(x) = x^2 - 2x + 7$</span> defined for a real number <span class="math-container">$x$</span>, then its equivalent notation might be <span class="math-container">$f : \mathbb{R} \rightarrow \mathbb{R}$</span>. If you allowed <span class="math-container">$x$</span> to be complex then it would be <span class="math-container">$f : \mathbb{C} \rightarrow \mathbb{C}$</span>, because <span class="math-container">$\mathbb{C}$</span> is the standard symbol for the set of all complex numbers.</p>
  <p>So now we can break down the notation <span class="math-container">$r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$</span></p>
  <blockquote>
  <p><span class="math-container">$r$</span></p>
  </blockquote>
  <p>The function is called <span class="math-container">$r$</span></p>
  <blockquote>
  <p><span class="math-container">$:$</span></p>
  </blockquote>
  <p>It has an input domain of . . .</p>
  <blockquote>
  <p><span class="math-container">$\mathcal{S} \times \mathcal{A}$</span></p>
  </blockquote>
  <p>The cartesian product of the set of all possible states <span class="math-container">$\mathcal{S}$</span> and the set of possible actions <span class="math-container">$\mathcal{A}$</span>.</p>
  <p>That is much the same as saying the function has a signature <span class="math-container">$r(s, a)$</span> where <span class="math-container">$s \in \mathcal{S}$</span> and <span class="math-container">$a \in \mathcal{A}$</span></p>
  <blockquote>
  <p><span class="math-container">$\rightarrow$</span></p>
  </blockquote>
  <p>It has an output domain of . . .</p>
  <blockquote>
  <p><span class="math-container">$\mathbb{R}$</span></p>
  </blockquote>
  <p>any single real number.</p>
  <hr />
  <p>* This choice (of declaring the more general <span class="math-container">$\mathbb{R}$</span> instead of specific <span class="math-container">$\mathcal{R}$</span>) is made partly because operators like <span class="math-container">$+$</span> and <span class="math-container">$\times$</span> are well defined for real numbers. This is a useful thing to assert about the behaviour of the reward function output when defining how value functions work for instance. Of course you could be more specific, defining <span class="math-container">$\mathcal{R}$</span> as some subset of <span class="math-container">$\mathbb{R}$</span>, that would be correct and more precise definition, but it is not needed for general theory in reinforcement learning. The less precise definition is fine for nearly all purposes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-14T13:14:46.827Z
Answers.LastActivityDate: 2020-09-14T16:02:58.903Z
Answers.LastEditDate: 2020-09-14T16:02:58.903Z
Answers.UserDisplayName: Neil Slater
Body: <p>Taken from section 2.1 in the article:</p>
  <blockquote>
  <p>We consider the standard reinforcement learning formalism consisting of an agent interacting with an environment. To simplify the exposition we assume that the environment is fully observable. An environment is described by a set of states <span class="math-container">$S$</span>, a set of actions <span class="math-container">$A$</span>, a distribution of initial states <span class="math-container">$p(s_0)$</span>, a reward function <strong><span class="math-container">$r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$</span></strong>, transition probabilities <span class="math-container">$p(s_{t+1} \mid s_t, a_t)$</span>, and a discount factor <span class="math-container">$\gamma \in [0, 1]$</span>.*</p>
  </blockquote>
  <p>How should one interpret the maths behind it?</p>
  
Tags: <reinforcement-learning><math><papers><hindsight-experience-replay>
Title: What does $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ mean in the article Hindsight Experience Replay, section 2.1?
UserDisplayName: WinnieThePooh
ThreadText: # What does $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ mean in the article Hindsight Experience Replay, section 2.1?
  
  (Asked by: username_0 on 2020-09-14)
  
  Taken from section 2.1 in the article:
  
  > We consider the standard reinforcement learning formalism consisting of an agent interacting with an environment. To simplify the exposition we assume that the environment is fully observable. An environment is described by a set of states $S$, a set of actions $A$, a distribution of initial states $p(s\_0)$, a reward function **$r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$**, transition probabilities $p(s\_{t+1} \mid s\_t, a\_t)$, and a discount factor $\gamma \in [0, 1]$.\*
  
  How should one interpret the maths behind it?
  
  **Tags:** reinforcement-learning, math, papers, hindsight-experience-replay
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-14. Score: 3 | Accepted answer
  
  This answer assumes that you only have a problem with this notation from the article:
  
  > $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$
  
  This is a standard notation, used in many disciplines, for defining a function and its input and output *domains*. It is a bit like the method signature for the function - it does not fully define it, but does enough to show how it can interact with other expressions.
  
  All functions can be thought of as maps between the input domain and output domain. You provide an input value, and it returns an output value. The values can be arbitrary mathematical objects. To show what kind of objects the inputs and outputs are allowed to be, the [notation for sets](https://www.mathsisfun.com/sets/symbols.html) is used.
  
  Importantly the symbol $\mathbb{R}$ at the end does not refer to the set of possible rewards in the environment (although it is a reward function, and that will be its output), but the set of all real numbers, because a reward is always a real number\*.
  
  As a concrete example, if you had the function $f(x) = x^2 - 2x + 7$ defined for a real number $x$, then its equivalent notation might be $f : \mathbb{R} \rightarrow \mathbb{R}$. If you allowed $x$ to be complex then it would be $f : \mathbb{C} \rightarrow \mathbb{C}$, because $\mathbb{C}$ is the standard symbol for the set of all complex numbers.
  
  So now we can break down the notation $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$
  
  > $r$
  
  The function is called $r$
  
  > $:$
  
  It has an input domain of . . .
  
  > $\mathcal{S} \times \mathcal{A}$
  
  The cartesian product of the set of all possible states $\mathcal{S}$ and the set of possible actions $\mathcal{A}$.
  
  That is much the same as saying the function has a signature $r(s, a)$ where $s \in \mathcal{S}$ and $a \in \mathcal{A}$
  
  > $\rightarrow$
  
  It has an output domain of . . .
  
  > $\mathbb{R}$
  
  any single real number.
  
  ---
  
  \* This choice (of declaring the more general $\mathbb{R}$ instead of specific $\mathcal{R}$) is made partly because operators like $+$ and $\times$ are well defined for real numbers. This is a useful thing to assert about the behaviour of the reward function output when defining how value functions work for instance. Of course you could be more specific, defining $\mathcal{R}$ as some subset of $\mathbb{R}$, that would be correct and more precise definition, but it is not needed for general theory in reinforcement learning. The less precise definition is fine for nearly all purposes.
  
  
  ---
  
  