Answers.Body: <p>A <a href="https://math.stackexchange.com/a/1678283">parameterized function</a> is a function that is defined by a <strong>set of parameters</strong>. If you change the parameters, you also change the actual function. For example, let's define this linear function</p>
  <p><span class="math-container">$$f: \mathcal{X} \rightarrow \mathcal{Y}$$</span> as follows</p>
  <p><span class="math-container">$$f(x; a, b) = ax + b \in \mathcal{Y}.$$</span></p>
  <p>Here, <span class="math-container">$x \in \mathcal{X}$</span> is the input to <span class="math-container">$f$</span>, while <span class="math-container">$a \in \mathbb{R}$</span> and <span class="math-container">$b \in \mathbb{R}$</span> are the parameters. Clearly, if you change <span class="math-container">$a$</span> and <span class="math-container">$b$</span>, you actual get different functions. If that's not obvious, you can graph them with a tool (e.g. <a href="https://www.desmos.com/calculator" rel="noreferrer">this one</a>). So, a parametrized function can actually be thought of a set of functions or a model. For more details about how functions and models are related in AI, see <a href="https://ai.stackexchange.com/a/25460/2444">this answer</a>.</p>
  <p>Now, back to the specific context. <span class="math-container">$v_\pi$</span> and <span class="math-container">$q_\pi$</span> are value functions, so they give you the value of a state or state-action pair, respectively, i.e. the expected return (reward in the long run). The goal in RL is really to learn these functions or a policy. Now, Sutton &amp; Barto mean that we can use a model like <span class="math-container">$f$</span> above and adjust the parameters <span class="math-container">$a$</span> and <span class="math-container">$b$</span> in order to learn the <em>optimal</em> value functions. Now, I will not define what an optimal value function is. It's defined in the book. Once you have a good grasp of the notion of a value function and optimality, you can also check <a href="https://nbro.gitlab.io/blogging/2022/02/16/bellman-equations/" rel="noreferrer">this post</a> (written by me) for more mathematical details on Bellman equations and value functions.</p>
  <p>Now, why would you want to use a function like <span class="math-container">$f$</span> to approximate these value functions? The same book tells you why in the sentence before - for finite MDPs, we could use an array to represent these functions, but if there are many states, actions, etc., it becomes hard to learn these functions in this way. If we use parameterized functions, we do not need to maintain so many entries (one for each state or state-action pair) and we hope that the function can generalise to unseen or rarely seen inputs.</p>
  <p>Finally, in practice, people have been using neural networks to learn value functions. The difference between neural networks and <span class="math-container">$f$</span> above is that neural networks are <a href="https://www.mathsisfun.com/definitions/nonlinear-equation.html" rel="noreferrer"><em>non-linear</em></a>, more powerful, but they also provide fewer guarantees.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-28T18:12:42.707Z
Answers.LastActivityDate: 2022-12-28T18:18:49.503Z
Answers.LastEditDate: 2022-12-28T18:18:49.503Z
Answers.UserDisplayName: nbro
Body: <p>From &quot;Reinforcement Learning: An introduction (2nd ed.)&quot; by Richard S. Sutton and Andrew G. Barto, <a href="https://www.incompleteideas.net/book/RLbook2020.pdf#page=81" rel="nofollow noreferrer">on page 59</a></p>
  <blockquote>
  <p>Instead, the agent would have to maintain <span class="math-container">$v_\pi$</span> and <span class="math-container">$q_\pi$</span> as parameterized functions (with fewer parameters than states) and adjust the parameters to better match the observed returns.</p>
  </blockquote>
  <p>What do they mean by &quot;parametrized functions&quot; here? The part of &quot;adjust parameters to better match observed returns&quot; also sounds odd to me.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-28T17:52:30.243Z
Comments.Text: I won't post a full answer since this comment might not fit the requirements for a good answer. A parameterized function is just where you model a real world function, g(x), as f(x,theta) and then find the theta that makes f agree with g. A neural network has inputs as well as unit weights and biases.
Comments.UserDisplayName: JacKeown
Tags: <reinforcement-learning><terminology><sutton-barto>
Title: Sutton & Barto: what are parametrized functions?
UserDisplayName: SomeoneUnknown
ThreadText: # Sutton & Barto: what are parametrized functions?
  
  (Asked by: username_0 on 2022-12-28)
  
  From "Reinforcement Learning: An introduction (2nd ed.)" by Richard S. Sutton and Andrew G. Barto, [on page 59](https://www.incompleteideas.net/book/RLbook2020.pdf#page=81)
  
  > Instead, the agent would have to maintain $v\_\pi$ and $q\_\pi$ as parameterized functions (with fewer parameters than states) and adjust the parameters to better match the observed returns.
  
  What do they mean by "parametrized functions" here? The part of "adjust parameters to better match observed returns" also sounds odd to me.
  
  **Tags:** reinforcement-learning, terminology, sutton-barto
  
  > **Comments:**
  > **username_1**: I won't post a full answer since this comment might not fit the requirements for a good answer. A parameterized function is just where you model a real world function, g(x), as f(x,theta) and then find the theta that makes f agree with g. A neural network has inputs as well as unit weights and biases.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-12-28. Score: 6 | Accepted answer
  
  A [parameterized function](https://math.stackexchange.com/a/1678283) is a function that is defined by a **set of parameters**. If you change the parameters, you also change the actual function. For example, let's define this linear function
  
  $$f: \mathcal{X} \rightarrow \mathcal{Y}$$ as follows
  
  $$f(x; a, b) = ax + b \in \mathcal{Y}.$$
  
  Here, $x \in \mathcal{X}$ is the input to $f$, while $a \in \mathbb{R}$ and $b \in \mathbb{R}$ are the parameters. Clearly, if you change $a$ and $b$, you actual get different functions. If that's not obvious, you can graph them with a tool (e.g. [this one](https://www.desmos.com/calculator)). So, a parametrized function can actually be thought of a set of functions or a model. For more details about how functions and models are related in AI, see [this answer](https://ai.stackexchange.com/a/25460/2444).
  
  Now, back to the specific context. $v\_\pi$ and $q\_\pi$ are value functions, so they give you the value of a state or state-action pair, respectively, i.e. the expected return (reward in the long run). The goal in RL is really to learn these functions or a policy. Now, Sutton & Barto mean that we can use a model like $f$ above and adjust the parameters $a$ and $b$ in order to learn the *optimal* value functions. Now, I will not define what an optimal value function is. It's defined in the book. Once you have a good grasp of the notion of a value function and optimality, you can also check [this post](https://username_2.gitlab.io/blogging/2022/02/16/bellman-equations/) (written by me) for more mathematical details on Bellman equations and value functions.
  
  Now, why would you want to use a function like $f$ to approximate these value functions? The same book tells you why in the sentence before - for finite MDPs, we could use an array to represent these functions, but if there are many states, actions, etc., it becomes hard to learn these functions in this way. If we use parameterized functions, we do not need to maintain so many entries (one for each state or state-action pair) and we hope that the function can generalise to unseen or rarely seen inputs.
  
  Finally, in practice, people have been using neural networks to learn value functions. The difference between neural networks and $f$ above is that neural networks are [*non-linear*](https://www.mathsisfun.com/definitions/nonlinear-equation.html), more powerful, but they also provide fewer guarantees.
  
  
  ---
  
  