Answers.Body: <p>Overall your understanding of dimensions here is correct. The function <span class="math-container">$a_k(\mathbf{x})$</span> measures the <em>exponential</em> of the <em>negative squared</em> Euclidean distance between the input state <span class="math-container">$x$</span> and the center <span class="math-container">$c_k$</span>, and then weighted by <span class="math-container">$s_k$</span> of the <span class="math-container">$k$</span>-th basis function. Then the normalized scalar <span class="math-container">$b_k(\mathbf{x})$</span> is a value between <span class="math-container">$0$</span> and <span class="math-container">$1$</span>, representing the contribution or activation of the <span class="math-container">$k$</span>-th basis function in a normalized way.</p>
  <p>This type of function is commonly used in radial basis function (RBF) networks, where Gaussian-like functions centered at specific points are combined to form a basis for approximating a target function. The <span class="math-container">$b_k(\mathbf{x})$</span> functions are often interpreted as activation functions for the input state or stacked with action as joint input representation, determining how much influence each basis function has at a given input representation.</p>
  <p>Further clarification based on your comment question, indeed here's a little messy and the authors were not clear enough assuming readers already have much background in <a href="https://en.wikipedia.org/wiki/Hamilton%E2%80%93Jacobi%E2%80%93Bellman_equation" rel="nofollow noreferrer">HJB equations</a> of classical physics and optimal control theory, you can further review the linked reference of HJB equation. As clearly defined in (32) here they're using linear function approximation thus the summations in both (33) &amp; (35) express the usual inner product of the parameter vector and the basis vector <span class="math-container">$[b_1,...,b_k]$</span>. But notice value function is scalar thus the parameters <span class="math-container">$ω_k$</span> in (32) is interpreted as usual which has no upper right superscript if you take minute notice, however, the parameters in (33) &amp; (35) have superscript <span class="math-container">$A,M$</span>, respectively. So as you rightly concerned to make dimension match with their LHS, their parameters should be construed as matrix multiplication with their respective basis vector <span class="math-container">$[b_1(x),...,b_k(x)],[b_1(x,u),...,b_k(x,u)]$</span> instead of simply inner product producing a scaler.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-05T14:52:19.317Z
Answers.Text: Thank you @cinch for your reply and clarification! I am still confused however, on how the scalar value in (33) could then be added to the vector valued $\mathbf{n}(t)$? If the answer is that $\omega_k^A$ is a vector, then I have included an edit at the bottom where I would like to then ask what that expression would mean? If that is not the answer then don't worry about that. Thank you!
Answers.UserDisplayName: Derick Diana
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-06T05:54:59.907Z
Answers.Text: As clearly defined in (32) here's linear function approximation thus the summations in both (33) & (35) express the usual inner product of the parameter vector and the basis vector $[b_1,...,b_k]$. But notice value function is scalar thus the parameters $\omega_k$ in (32) has no upper right superscript interpreted as usual. But the parameters in (33) & (35) have superscript $A,M$, respectively. So as you rightly concerned to make dimension match with LHS, their parameters should be treated as matrix multiplication with their respective basis vector $[b_1(x),...,b_k(x)],[b_1(x,u),...,b_k(x,u)]$
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-04T23:46:50.253Z
Answers.LastActivityDate: 2024-03-13T21:19:20.060Z
Answers.LastEditDate: 2024-03-13T21:19:20.060Z
Answers.UserDisplayName: cinch
Body: <p>I am working on implementing the learning shown in this paper:</p>
  <p><a href="https://homes.cs.washington.edu/%7Etodorov/courses/amath579/reading/Continuous.pdf" rel="nofollow noreferrer">https://homes.cs.washington.edu/~todorov/courses/amath579/reading/Continuous.pdf</a></p>
  <p>In the paper, the authors devise a continuous time learning extension of the actor critic method in reinforcement learning. My question is simply about the dimensions of some objects in equations shown in the paper.</p>
  <p>I will summarize the relevant equations for my problem. Let the state of the system be described by a state vector <span class="math-container">$\mathbf{x}(t)\in\mathbb{R}^n$</span> and the actions undertaken by an agent interacting with the system be described by <span class="math-container">$\mathbf{u}(t)\in\mathbb{R}^m$</span>. Then the natural generalisation of the evolution of the system to continuous time is:</p>
  <p><span class="math-container">$$\dot{\mathbf{x}}(t)=\mathbf{f}(\mathbf{x}(t),\mathbf{u}(t)) \tag{1} $$</span></p>
  <p>A quick note is that I am simply copying the paper's notation (which is part of the problem) and I am also numbering the equations according to where they appear in the paper. From this we also see that <span class="math-container">$f\in\mathbb{R}^n$</span>. I mention these equations just to set the scene and dimensions of the various objects. They then do some derivations, and end up saying that they will describe the actor as:</p>
  <p><span class="math-container">$$\mathbf{u}(t) = s\left( \mathbf{A}(\mathbf{x}(t);\mathbf{w}^A) + \sigma \mathbf{n}(t)\right) \tag{19}$$</span></p>
  <p>Here, <span class="math-container">$s()$</span> is an activation function, <span class="math-container">$\mathbf{w}^A = \{ \omega_i^A \}$</span> is a parameter vector, <span class="math-container">$\mathbf{n}(t)\in\mathbb{R}^m$</span> and <span class="math-container">$\mathbf{A}(\mathbf{x}(t);\mathbf{w}^A)\in\mathbb{R}^m$</span>. The last two make sense since <span class="math-container">$\mathbf{u}(t)\in\mathbb{R}^m$</span> and I would guess then that the function <span class="math-container">$s()$</span> is applied elementwise. We have that <span class="math-container">$\mathbf{n}(t)$</span> is some noise to encourage exploration. Now <span class="math-container">$ \mathbf{A}(\mathbf{x}(t);\mathbf{w}^A)$</span> is defined by further refining the above expression in the appendix to</p>
  <p><span class="math-container">$$\mathbf{u}(t) = \mathbf{u}^{\text{max}}s\left( \sum_k \omega_k^{A} b_k(\mathbf{x}(t))+ \sigma \mathbf{n}(t)\right) \tag{33}$$</span></p>
  <p>Here is where the problem comes in. We would expect the the summation above outputs an <span class="math-container">$m$</span> dimensional vector as it is added to the presumably <span class="math-container">$m$</span> dimensionl <span class="math-container">$\mathbf{n}(t)$</span>, then <span class="math-container">$s()$</span> (a sigmoid function) is applied pointwise so that each of the <span class="math-container">$m$</span> elements is limited to a max value of 1, and then <span class="math-container">$\mathbf{u}^{\text{max}}$</span> pointwise multiplies each of those elements to choose its new maximum possible value. However, the <span class="math-container">$b_k(\mathbf{x}(t))$</span> above apparently form a Normalized Gaussian Network and are defined as follows:</p>
  <p><span class="math-container">$$b_k(\mathbf{x}) = \frac{a_k(\mathbf{x})}{\sum_{l=1}^K a_l(\mathbf{x})}$$</span></p>
  <p>with</p>
  <p><span class="math-container">$$a_k(\mathbf{x}) = e^{-||\mathbf{s}_k^T(\mathbf{x}-\mathbf{c}_k)||^2} \tag{32}$$</span>.</p>
  <p>So each <span class="math-container">$\mathbf{c}_k$</span> is the center in state space of the <span class="math-container">$k$</span>'th basis vector from which the current position in state space is subtracted. This is dotted with <span class="math-container">$\mathbf{s}_k$</span> to account for units and the whole thing is put through a norm and exponentiated. So I believe the function <span class="math-container">$a_k(\mathbf{x})$</span> in some sense measures the distance of the current state <span class="math-container">$\mathbf{x}$</span> to <span class="math-container">$k$</span>'th basis vector. Then <span class="math-container">$b_k(\mathbf{x})$</span> does the same thing but normalized over all distances.</p>
  <p>But here is my question: surely <span class="math-container">$b_k(\mathbf{x})$</span> is a scalar due to its form as a ratio of <span class="math-container">$a_k$</span> functions (and the form of <span class="math-container">$a_k$</span>)? In that case, it does not output an <span class="math-container">$m$</span> dimensional vector.</p>
  <p>In my code (which does seem to work) I have <span class="math-container">$m=1$</span> so I haven't encountered this issue so far but I need to understand to build more complicated models.</p>
  <p>An answer to how the above makes sense may be all I need, but if you think the solution is merely that we replace the scalar with an <span class="math-container">$m$</span> dimensional vector whose every entry is the scalar value, then we get another issue later on. When we define a Normalized Gaussian Network <span class="math-container">$\hat{\mathbf{f}}$</span> to try to learn <span class="math-container">$\dot{\mathbf{x}}(t)$</span> as follows:</p>
  <p><span class="math-container">$$\dot{\mathbf{x}}(t)\sim \hat{\mathbf{f}}(\mathbf{x}(t),\mathbf{u}(t)) = \sum_k \omega_k^M b_k(\mathbf{x}(t),\mathbf{u}(t)) \tag{35}$$</span></p>
  <p>They say very very little about this, but I would assume they mean that the function <span class="math-container">$b_k$</span> takes as input the vectors <span class="math-container">$\mathbf{x}(t)$</span> and <span class="math-container">$\mathbf{u}(t)$</span> &quot;stacked&quot; in some sense, and gives you the distance to basis vectors in the larger space of states of the system and actions taken by the agent. The issue is then with the way the <span class="math-container">$\omega_k^M$</span> are updated, which obeys the following equation:</p>
  <p><span class="math-container">$$\dot{\omega_k^M}(t)=\eta^M\left(\dot{\mathbf{x}}(t)-\hat{\mathbf{f}}(\mathbf{x}(t),\mathbf{u}(t))\right)b_k\left(\mathbf{x}(t),\mathbf{u}(t)\right)$$</span></p>
  <p>Now you will note that the <strong>weights</strong> <span class="math-container">$\omega_k^M$</span> are vectors, since the difference <span class="math-container">$\dot{\mathbf{x}}(t)-\hat{\mathbf{f}}(\mathbf{x}(t),\mathbf{u}(t))$</span> is a difference of elements of <span class="math-container">$\mathbb{R}^n$</span> and we are assuming the <span class="math-container">$b_k$</span> are scalars. I do not know if this is correct and there is very little explanation given in this part of the paper.</p>
  <p>Any help understanding the dimensions of these objects would be appreciated.</p>
  <p>Edit 1: For some additional information that may be helpful for the deduction of the dimensions involved, the weights <span class="math-container">$\omega_k^A$</span> in equation (33) are updated by the following equation:</p>
  <p><span class="math-container">$$\dot{\omega}_i^A = \eta^A \delta(t) \mathbf{n}(t) \frac{\partial \mathbf{A}(\mathbf{x}(t);\mathbf{w}^A)}{\partial \omega_i^A}\tag{20}$$</span></p>
  <p>where <span class="math-container">$\eta^A$</span> and <span class="math-container">$\delta(t)$</span> are definitely scalars.</p>
  <p>As a further note, I pointed out above (and they specifically state in the paper) that <span class="math-container">$\mathbf{w}^A$</span> is a &quot;parameter vector&quot; so I would be surprised if <span class="math-container">$\omega_k^A$</span> in (33) is a vector too. However, they say almost nothing about (35), so there I'm less sure.</p>
  
Tags: <reinforcement-learning><machine-learning><papers><actor-critic-methods><notation>
Title: Notation used in paper on Continuous Time Reinforcement Learning
UserDisplayName: Derick Diana
ThreadText: # Notation used in paper on Continuous Time Reinforcement Learning
  
  (Asked by: username_0 on 2024-03-03)
  
  I am working on implementing the learning shown in this paper:
  
  [https://homes.cs.washington.edu/~todorov/courses/amath579/reading/Continuous.pdf](https://homes.cs.washington.edu/%7Etodorov/courses/amath579/reading/Continuous.pdf)
  
  In the paper, the authors devise a continuous time learning extension of the actor critic method in reinforcement learning. My question is simply about the dimensions of some objects in equations shown in the paper.
  
  I will summarize the relevant equations for my problem. Let the state of the system be described by a state vector $\mathbf{x}(t)\in\mathbb{R}^n$ and the actions undertaken by an agent interacting with the system be described by $\mathbf{u}(t)\in\mathbb{R}^m$. Then the natural generalisation of the evolution of the system to continuous time is:
  
  $$\dot{\mathbf{x}}(t)=\mathbf{f}(\mathbf{x}(t),\mathbf{u}(t)) \tag{1} $$
  
  A quick note is that I am simply copying the paper's notation (which is part of the problem) and I am also numbering the equations according to where they appear in the paper. From this we also see that $f\in\mathbb{R}^n$. I mention these equations just to set the scene and dimensions of the various objects. They then do some derivations, and end up saying that they will describe the actor as:
  
  $$\mathbf{u}(t) = s\left( \mathbf{A}(\mathbf{x}(t);\mathbf{w}^A) + \sigma \mathbf{n}(t)\right) \tag{19}$$
  
  Here, $s()$ is an activation function, $\mathbf{w}^A = \{ \omega\_i^A \}$ is a parameter vector, $\mathbf{n}(t)\in\mathbb{R}^m$ and $\mathbf{A}(\mathbf{x}(t);\mathbf{w}^A)\in\mathbb{R}^m$. The last two make sense since $\mathbf{u}(t)\in\mathbb{R}^m$ and I would guess then that the function $s()$ is applied elementwise. We have that $\mathbf{n}(t)$ is some noise to encourage exploration. Now $ \mathbf{A}(\mathbf{x}(t);\mathbf{w}^A)$ is defined by further refining the above expression in the appendix to
  
  $$\mathbf{u}(t) = \mathbf{u}^{\text{max}}s\left( \sum\_k \omega\_k^{A} b\_k(\mathbf{x}(t))+ \sigma \mathbf{n}(t)\right) \tag{33}$$
  
  Here is where the problem comes in. We would expect the the summation above outputs an $m$ dimensional vector as it is added to the presumably $m$ dimensionl $\mathbf{n}(t)$, then $s()$ (a sigmoid function) is applied pointwise so that each of the $m$ elements is limited to a max value of 1, and then $\mathbf{u}^{\text{max}}$ pointwise multiplies each of those elements to choose its new maximum possible value. However, the $b\_k(\mathbf{x}(t))$ above apparently form a Normalized Gaussian Network and are defined as follows:
  
  $$b\_k(\mathbf{x}) = \frac{a\_k(\mathbf{x})}{\sum\_{l=1}^K a\_l(\mathbf{x})}$$
  
  with
  
  $$a\_k(\mathbf{x}) = e^{-||\mathbf{s}\_k^T(\mathbf{x}-\mathbf{c}\_k)||^2} \tag{32}$$.
  
  So each $\mathbf{c}\_k$ is the center in state space of the $k$'th basis vector from which the current position in state space is subtracted. This is dotted with $\mathbf{s}\_k$ to account for units and the whole thing is put through a norm and exponentiated. So I believe the function $a\_k(\mathbf{x})$ in some sense measures the distance of the current state $\mathbf{x}$ to $k$'th basis vector. Then $b\_k(\mathbf{x})$ does the same thing but normalized over all distances.
  
  But here is my question: surely $b\_k(\mathbf{x})$ is a scalar due to its form as a ratio of $a\_k$ functions (and the form of $a\_k$)? In that case, it does not output an $m$ dimensional vector.
  
  In my code (which does seem to work) I have $m=1$ so I haven't encountered this issue so far but I need to understand to build more complicated models.
  
  An answer to how the above makes sense may be all I need, but if you think the solution is merely that we replace the scalar with an $m$ dimensional vector whose every entry is the scalar value, then we get another issue later on. When we define a Normalized Gaussian Network $\hat{\mathbf{f}}$ to try to learn $\dot{\mathbf{x}}(t)$ as follows:
  
  $$\dot{\mathbf{x}}(t)\sim \hat{\mathbf{f}}(\mathbf{x}(t),\mathbf{u}(t)) = \sum\_k \omega\_k^M b\_k(\mathbf{x}(t),\mathbf{u}(t)) \tag{35}$$
  
  They say very very little about this, but I would assume they mean that the function $b\_k$ takes as input the vectors $\mathbf{x}(t)$ and $\mathbf{u}(t)$ "stacked" in some sense, and gives you the distance to basis vectors in the larger space of states of the system and actions taken by the agent. The issue is then with the way the $\omega\_k^M$ are updated, which obeys the following equation:
  
  $$\dot{\omega\_k^M}(t)=\eta^M\left(\dot{\mathbf{x}}(t)-\hat{\mathbf{f}}(\mathbf{x}(t),\mathbf{u}(t))\right)b\_k\left(\mathbf{x}(t),\mathbf{u}(t)\right)$$
  
  Now you will note that the **weights** $\omega\_k^M$ are vectors, since the difference $\dot{\mathbf{x}}(t)-\hat{\mathbf{f}}(\mathbf{x}(t),\mathbf{u}(t))$ is a difference of elements of $\mathbb{R}^n$ and we are assuming the $b\_k$ are scalars. I do not know if this is correct and there is very little explanation given in this part of the paper.
  
  Any help understanding the dimensions of these objects would be appreciated.
  
  Edit 1: For some additional information that may be helpful for the deduction of the dimensions involved, the weights $\omega\_k^A$ in equation (33) are updated by the following equation:
  
  $$\dot{\omega}\_i^A = \eta^A \delta(t) \mathbf{n}(t) \frac{\partial \mathbf{A}(\mathbf{x}(t);\mathbf{w}^A)}{\partial \omega\_i^A}\tag{20}$$
  
  where $\eta^A$ and $\delta(t)$ are definitely scalars.
  
  As a further note, I pointed out above (and they specifically state in the paper) that $\mathbf{w}^A$ is a "parameter vector" so I would be surprised if $\omega\_k^A$ in (33) is a vector too. However, they say almost nothing about (35), so there I'm less sure.
  
  **Tags:** reinforcement-learning, machine-learning, papers, actor-critic-methods, notation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-03-04. Score: 1
  
  Overall your understanding of dimensions here is correct. The function $a\_k(\mathbf{x})$ measures the *exponential* of the *negative squared* Euclidean distance between the input state $x$ and the center $c\_k$, and then weighted by $s\_k$ of the $k$-th basis function. Then the normalized scalar $b\_k(\mathbf{x})$ is a value between $0$ and $1$, representing the contribution or activation of the $k$-th basis function in a normalized way.
  
  This type of function is commonly used in radial basis function (RBF) networks, where Gaussian-like functions centered at specific points are combined to form a basis for approximating a target function. The $b\_k(\mathbf{x})$ functions are often interpreted as activation functions for the input state or stacked with action as joint input representation, determining how much influence each basis function has at a given input representation.
  
  Further clarification based on your comment question, indeed here's a little messy and the authors were not clear enough assuming readers already have much background in [HJB equations](https://en.wikipedia.org/wiki/Hamilton%E2%80%93Jacobi%E2%80%93Bellman_equation) of classical physics and optimal control theory, you can further review the linked reference of HJB equation. As clearly defined in (32) here they're using linear function approximation thus the summations in both (33) & (35) express the usual inner product of the parameter vector and the basis vector $[b\_1,...,b\_k]$. But notice value function is scalar thus the parameters $ω\_k$ in (32) is interpreted as usual which has no upper right superscript if you take minute notice, however, the parameters in (33) & (35) have superscript $A,M$, respectively. So as you rightly concerned to make dimension match with their LHS, their parameters should be construed as matrix multiplication with their respective basis vector $[b\_1(x),...,b\_k(x)],[b\_1(x,u),...,b\_k(x,u)]$ instead of simply inner product producing a scaler.
  
  > **Comments:**
  > **username_0**: Thank you @username_1 for your reply and clarification! I am still confused however, on how the scalar value in (33) could then be added to the vector valued $\mathbf{n}(t)$? If the answer is that $\omega_k^A$ is a vector, then I have included an edit at the bottom where I would like to then ask what that expression would mean? If that is not the answer then don't worry about that. Thank you!  
  > **username_1**: As clearly defined in (32) here's linear function approximation thus the summations in both (33) & (35) express the usual inner product of the parameter vector and the basis vector $[b_1,...,b_k]$. But notice value function is scalar thus the parameters $\omega_k$ in (32) has no upper right superscript interpreted as usual. But the parameters in (33) & (35) have superscript $A,M$, respectively. So as you rightly concerned to make dimension match with LHS, their parameters should be treated as matrix multiplication with their respective basis vector $[b_1(x),...,b_k(x)],[b_1(x,u),...,b_k(x,u)]$  
  
  ---
  
  