Answers.Body: <p>Here is a mathematical definition for state aggregation, which I think covers all the necessary elements to make something an aggregation:</p>
  <ul>
  <li>Define a collection of state groups <span class="math-container">$\mathscr{G} = \{\mathcal{G}_1, \mathcal{G}_2, \mathcal{G}_3 ...  \}$</span></li>
  <li><span class="math-container">$\forall \mathcal{G} \in \mathscr{G}: \mathcal{G} \subset \mathcal{S}$</span> (each group is a subset of all states)</li>
  <li><span class="math-container">$\mathcal{S} \subseteq \bigcup_{\mathcal{G} \in \mathscr{G}} \mathcal{G}$</span> (the groups cover all the states)</li>
  <li><span class="math-container">$\forall \mathcal{G} \in \mathscr{G}: \forall \mathcal{H} \in \mathscr{G}, \mathcal{H} \neq \mathcal{G}:\mathcal{G} \cap \mathcal{H} = \phi$</span> (all groups are disjoint - this is usefully relaxed for systems described later in S&amp;B such as tile coding)</li>
  <li><span class="math-container">$|\mathscr{G}| \lt |\mathcal{S}|$</span> (number of groups is less than number of states - that is what makes it an aggregation as opposed to some other mapping)</li>
  </ul>
  <p>The text</p>
  <blockquote>
  <p>is 1 for <span class="math-container">$S_t$</span>’s group’s component</p>
  </blockquote>
  <p>Is referring to using the subscript <span class="math-container">$j$</span> in <span class="math-container">$\mathcal{G}_j$</span> for the group that contains <span class="math-container">$S_t$</span>, or alternatively some function <span class="math-container">$g(s): \mathcal{S} \rightarrow \mathbb{Z}$</span> that returns a group id or index for each state. Using this function can map the state value <span class="math-container">$s$</span> into a one-hot encoded vector <span class="math-container">$\mathbf{x}$</span> which is then used in the linear regression estimator <span class="math-container">$\hat{v}(s, \mathbf{w}) = \mathbf{w}^T\mathbf{x}$</span>.</p>
  <p>Worth noting that the above defines an <em>arbitrary</em> state aggregation. Usually you would try to do better than that and group states together because there is some reason to expect them to have similar state values in general.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-30T21:53:54.867Z
Answers.Text: @ Neil Slater, I think that is what I need:  https://ai.stackexchange.com/questions/12206/state-aggregation-methods
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-30T21:55:45.970Z
Answers.Text: @ Neil Slater,   I was looking for this definition:  \begin{equation}
  v(\mathbf w) = x_1w_1 + x_2w_2 + ... + x_{10}w_{10}
  \end{equation}
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-30T22:09:30.913Z
Answers.Text: @ Neil Slater,  for your answer to be a full answer, we need the definition of $\hat{v}(\bf{w})$ to see clearly why  $\nabla \hat{v}(S_t,w_t)$, is 1 for $S_t$’s group’s component and 0 for the other components. If you have time, it may be better to include the definition of  $\hat{v}(S_t,w_t)$. I am sure that this is not clear for many readers of Sutton-Barto's  RL book.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-31T06:07:04.427Z
Answers.Text: @DSPInfitinity: That's not part of the definition for state aggregation, which is about how you get $\mathbf{x}$ from $s$ i.e. the part where you aggregate the states and form a feature vector. In fact it seems you were not really asking about state aggregation, but about the definition of linear regression in RL?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-31T06:36:32.943Z
Answers.Text: I added a note about the linear estimator.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-30T21:14:43.410Z
Answers.LastActivityDate: 2022-03-31T06:36:21.987Z
Answers.LastEditDate: 2022-03-31T06:36:21.987Z
Answers.UserDisplayName: Neil Slater
Body: <p>Sutton-Barto's RL book (page 203):</p>
  <blockquote>
  <p>State aggregation is a simple form of generalizing function approximation in which
  states are grouped together, with one estimated value (one component of the weight
  vector w) for each group. The value of a state is estimated as its group’s component,
  and when the state is updated, that component alone is updated. State aggregation
  is a special case of SGD (9.7) in which the gradient, <span class="math-container">$\nabla \hat{v}(S_t,w_t)$</span>, is 1 for <span class="math-container">$S_t$</span>’s group’s component and 0 for the other components.</p>
  </blockquote>
  <p>Nothing is clear for me here. How is state aggregation defined mathematically here?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-29T23:21:58.883Z
Comments.Text: @nro,  it is done.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T13:37:07.627Z
Comments.Text: Do you understand this part "State aggregation is a simple form of generalizing function approximation in which states are grouped together"? Do you know what function approximation is? Can you edit your post to tell us what you understand about this definition? Note that it's a _definition_. So it's not something that you can derive from other things.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T13:40:11.420Z
Comments.Text: @nbro, I hate your comments and what you are doing is to prevent people receiving help from others. You should not be in this group. You are simply UNNECESSARY!
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T13:45:07.957Z
Comments.Text: I'm trying to help, not just you, but people that would like to answer your question(s). That's what I've been doing here for a long time. I'm trying to understand what you understand or not about the definition. They are providing a definition and you're asking for another definition. This does not make sense to me, unless you didn't understand that this is a definition. Having said this, you should show some respect to someone that helped to shape this community and to make it a valuable place where people can come and receive good answers to their questions.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T13:47:01.643Z
Comments.Text: @nro, I have asked so far more than 20 questions and you never answered any: what you did was to be negative and undermine. Please leave the group!
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T13:49:42.073Z
Comments.Text: We are volunteers here. If you ask a question, it doesn't mean it will be answered. Even if I knew the answer or I could try to give an answer, I might choose not to. People could not have the time or will to answer your questions. I would recommend that you do as I said, tell us what you understand about that definition and why you're asking for another definition. Then have you read [this](https://ai.stackexchange.com/q/12206/2444)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T13:50:35.917Z
Comments.Text: @nro, on top of  these you do not have any RL knowledge. My question is quite clear: they provided a definition by "words" which is very vague and I asked the mathematical definition to clearly understand what they mean. We are not stupid and you are not genius! Let me repeat again: please leave the group!
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T13:52:08.110Z
Comments.Text: I have "some" RL knowledge. You see confirm that [here](https://ai.stackexchange.com/tags/reinforcement-learning/topusers).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T17:05:45.013Z
Comments.Text: What nbro is doing is helping to refine the question so that it is more likely to get a useful answer. That is a standard part of workflow on the site, and many of us do it. You may view it as some barrier to getting help (because it requires you to do work and define your problem more precisely), but if you engage with it and attempt to clarify the details that are being asked about, hopefully you will find it is not. This is unfortunately a common "clash" between new members asking questions and longer-term experts, and is why many Stack Exchange sites get a reputation for being unfriendly.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T19:42:32.650Z
Comments.Text: @Neil Slater, my question is indeed very clear: "How is state aggregation defined mathematically here?" I am asking its mathematical definition and really there is nothing to clarify.
Comments.UserDisplayName: DSPinfinity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-30T21:20:10.893Z
Comments.Text: I have answered with a mathematical definition, using notation (as well as I can, I am not a notation expert). If you want a more text-based answer or guide to what is going on, then we need to do the clarification that nbro is asking for. Some people from a maths background may appreciate my attempt, and perhaps you are one of them. However, that's unusual outside of math-heavy Stack Exchanges (where my attempts at notation would probably be laughed at)
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><sutton-barto>
Title: How is state aggregation defined mathematically here?
UserDisplayName: DSPinfinity
ThreadText: # How is state aggregation defined mathematically here?
  
  (Asked by: username_0 on 2022-03-29)
  
  Sutton-Barto's RL book (page 203):
  
  > State aggregation is a simple form of generalizing function approximation in which
  > states are grouped together, with one estimated value (one component of the weight
  > vector w) for each group. The value of a state is estimated as its group’s component,
  > and when the state is updated, that component alone is updated. State aggregation
  > is a special case of SGD (9.7) in which the gradient, $\nabla \hat{v}(S\_t,w\_t)$, is 1 for $S\_t$’s group’s component and 0 for the other components.
  
  Nothing is clear for me here. How is state aggregation defined mathematically here?
  
  **Tags:** reinforcement-learning, sutton-barto
  
  > **Comments:**
  > **username_0**: @nro,  it is done.  
  > **username_1**: Do you understand this part "State aggregation is a simple form of generalizing function approximation in which states are grouped together"? Do you know what function approximation is? Can you edit your post to tell us what you understand about this definition? Note that it's a _definition_. So it's not something that you can derive from other things.  
  > **username_0**: @username_1, I hate your comments and what you are doing is to prevent people receiving help from others. You should not be in this group. You are simply UNNECESSARY!  
  > **username_1**: I'm trying to help, not just you, but people that would like to answer your question(s). That's what I've been doing here for a long time. I'm trying to understand what you understand or not about the definition. They are providing a definition and you're asking for another definition. This does not make sense to me, unless you didn't understand that this is a definition. Having said this, you should show some respect to someone that helped to shape this community and to make it a valuable place where people can come and receive good answers to their questions.  
  > **username_0**: @nro, I have asked so far more than 20 questions and you never answered any: what you did was to be negative and undermine. Please leave the group!  
  > **username_1**: We are volunteers here. If you ask a question, it doesn't mean it will be answered. Even if I knew the answer or I could try to give an answer, I might choose not to. People could not have the time or will to answer your questions. I would recommend that you do as I said, tell us what you understand about that definition and why you're asking for another definition. Then have you read [this](https://ai.stackexchange.com/q/12206/2444)?  
  > **username_0**: @nro, on top of  these you do not have any RL knowledge. My question is quite clear: they provided a definition by "words" which is very vague and I asked the mathematical definition to clearly understand what they mean. We are not stupid and you are not genius! Let me repeat again: please leave the group!  
  > **username_1**: I have "some" RL knowledge. You see confirm that [here](https://ai.stackexchange.com/tags/reinforcement-learning/topusers).  
  > **username_2**: What username_1 is doing is helping to refine the question so that it is more likely to get a useful answer. That is a standard part of workflow on the site, and many of us do it. You may view it as some barrier to getting help (because it requires you to do work and define your problem more precisely), but if you engage with it and attempt to clarify the details that are being asked about, hopefully you will find it is not. This is unfortunately a common "clash" between new members asking questions and longer-term experts, and is why many Stack Exchange sites get a reputation for being unfriendly.  
  > **username_0**: @username_2, my question is indeed very clear: "How is state aggregation defined mathematically here?" I am asking its mathematical definition and really there is nothing to clarify.  
  > **username_2**: I have answered with a mathematical definition, using notation (as well as I can, I am not a notation expert). If you want a more text-based answer or guide to what is going on, then we need to do the clarification that username_1 is asking for. Some people from a maths background may appreciate my attempt, and perhaps you are one of them. However, that's unusual outside of math-heavy Stack Exchanges (where my attempts at notation would probably be laughed at)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-03-30. Score: 1 | Accepted answer
  
  Here is a mathematical definition for state aggregation, which I think covers all the necessary elements to make something an aggregation:
  
  * Define a collection of state groups $\mathscr{G} = \{\mathcal{G}\_1, \mathcal{G}\_2, \mathcal{G}\_3 ... \}$
  * $\forall \mathcal{G} \in \mathscr{G}: \mathcal{G} \subset \mathcal{S}$ (each group is a subset of all states)
  * $\mathcal{S} \subseteq \bigcup\_{\mathcal{G} \in \mathscr{G}} \mathcal{G}$ (the groups cover all the states)
  * $\forall \mathcal{G} \in \mathscr{G}: \forall \mathcal{H} \in \mathscr{G}, \mathcal{H} \neq \mathcal{G}:\mathcal{G} \cap \mathcal{H} = \phi$ (all groups are disjoint - this is usefully relaxed for systems described later in S&B such as tile coding)
  * $|\mathscr{G}| \lt |\mathcal{S}|$ (number of groups is less than number of states - that is what makes it an aggregation as opposed to some other mapping)
  
  The text
  
  > is 1 for $S\_t$’s group’s component
  
  Is referring to using the subscript $j$ in $\mathcal{G}\_j$ for the group that contains $S\_t$, or alternatively some function $g(s): \mathcal{S} \rightarrow \mathbb{Z}$ that returns a group id or index for each state. Using this function can map the state value $s$ into a one-hot encoded vector $\mathbf{x}$ which is then used in the linear regression estimator $\hat{v}(s, \mathbf{w}) = \mathbf{w}^T\mathbf{x}$.
  
  Worth noting that the above defines an *arbitrary* state aggregation. Usually you would try to do better than that and group states together because there is some reason to expect them to have similar state values in general.
  
  > **Comments:**
  > **username_0**: @ username_2 username_2, I think that is what I need:  https://ai.stackexchange.com/questions/12206/state-aggregation-methods  
  > **username_0**: @ username_2 username_2,   I was looking for this definition:  \begin{equation} v(\mathbf w) = x_1w_1 + x_2w_2 + ... + x_{10}w_{10} \end{equation}  
  > **username_0**: @ username_2 username_2,  for your answer to be a full answer, we need the definition of $\hat{v}(\bf{w})$ to see clearly why  $\nabla \hat{v}(S_t,w_t)$, is 1 for $S_t$’s group’s component and 0 for the other components. If you have time, it may be better to include the definition of  $\hat{v}(S_t,w_t)$. I am sure that this is not clear for many readers of Sutton-Barto's  RL book.  
  > **username_2**: @DSPInfitinity: That's not part of the definition for state aggregation, which is about how you get $\mathbf{x}$ from $s$ i.e. the part where you aggregate the states and form a feature vector. In fact it seems you were not really asking about state aggregation, but about the definition of linear regression in RL?  
  > **username_2**: I added a note about the linear estimator.  
  
  ---
  
  