Answers.Body: <p>I add some thoughts of mine following Sutton and Barto, chapter 13, also to keep track of my work. This effort should go in the direction of answering point 2 of the question.</p>
  <p><strong>Policy gradient theorem</strong></p>
  <p>We consider the setting RL0 as in the OP, with the additional notation:</p>
  <ol>
  <li>Each episod goes to the terminal state immediately after the first action.</li>
  <li>Each state <span class="math-container">$s$</span> can be denoted as <span class="math-container">$x$</span>, a member of the input state <span class="math-container">$X$</span>, more suggestively remembering the notation in a supervised setting.</li>
  <li>The action is the (deterministic) prediction of the parametrized model (e.g. a neural network), so that <span class="math-container">$\pi(a,s)$</span> can be rewritten as <span class="math-container">$\pi(y,x)=\delta(y-Y^{pred}(x,\theta))$</span>, where <span class="math-container">$a$</span> has been substituted with <span class="math-container">$y$</span> to match notation with a supervised setting.</li>
  <li>The reward is the squared error: <span class="math-container">$R=G=|Y^{pred}(x)-Y^{ex}(x)|^2$</span></li>
  </ol>
  <p>Such a setting looks very similar to a supervised one. Indeed the policy gradient theorem reads:</p>
  <p><span class="math-container">$$\partial_{\theta} J=\int dy |Y^{pred}(x,\theta)-Y^{ex}(x)|^2\partial_{\theta}(\delta(Y^{pred}(x,\theta)-y)) [1]$$</span></p>
  <p>After some maninpulations and integration by parts one can rewrite the policy gradient theorem as above in the form:</p>
  <p><span class="math-container">$$\partial_{\theta} J=\partial_{\theta} |Y^{pred}(x,\theta)-Y^{ex}(x)|^2 [2]$$</span></p>
  <p>which is the form one would use in a supervised form for the update. So up to the policy gradient theorem RL0 and ANN would be equal in this setting.</p>
  <p><strong>REINFORCE update</strong></p>
  <p>If we look at the REINFORCE update rule in this setting it reads:</p>
  <p><span class="math-container">$\delta \theta=|Y^{pred}(x,\theta)-Y^{ex}(x)|^2 \partial_{\theta}(\delta(Y^{pred}(x,\theta)-y))/\delta(Y^{pred}(x,\theta)-y)$</span></p>
  <p>To get a better understanding of the ratio of <span class="math-container">$\delta$</span> functions we rewrite:</p>
  <p><span class="math-container">$\partial_{\theta}(\delta(Y^{pred}(x,\theta)-y))/\delta(Y^{pred}(x,\theta)-y)=\delta'(Y^{pred}(x,\theta)-y)/\delta(Y^{pred}(x,\theta)-y)\partial_{\theta}Y^{pred}(x,\theta)$</span></p>
  <p>So we would like to understand if the expression <span class="math-container">$\delta'(x)/\delta(x)$</span> can be given a finite meaning. Unluckily if we use a regularization of the delta funcion <span class="math-container">$\delta(x)=\frac{1}{\epsilon}e^{-\pi x^2/\epsilon^2}$</span> we see that this expression diverges when x is not null.</p>
  <p><strong>INTERPRETATION (for the moment):</strong></p>
  <p>It looks that the manipulations starting from the policy gradient theorem and leading to the reinforce update is where RL0 and ANN start to diverge substantially. I think the main issue there is that when in the derivation of the REINFORCE algorithm one multiplies and divides by <span class="math-container">$\pi(a|s)$</span>, this operation provides larger estimators when <span class="math-container">$\pi(a|s)$</span> is low (because the <span class="math-container">$\pi$</span> at the denominator is the one surviving, the other gets replaces by sampling). As a consequence the REINFORCE update breaks when the densities are deterministic, i.e. when no sampling is necessary.</p>
  <p>Comments welcome.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-13T08:45:19.247Z
Answers.LastActivityDate: 2023-06-16T11:27:10.580Z
Answers.LastEditDate: 2023-06-16T11:27:10.580Z
Answers.UserDisplayName: Thomas
Body: <p>I am on a learning phase (still to enter the details but I would like first to get a bird's eye overview). I would like to understand the difference between:</p>
  <p>(1) a reinforcement learning framework, with <span class="math-container">$\gamma=0$</span>, policy gradient optimization, and where the environment presents a new state at every iteration sampled from an input distribution. These conditions define a setup that we call RL0</p>
  <p>(2) a standard neural network supervised classification framework. Call this second setup ANN.</p>
  <p>To me the setups seem equal if we make the correspondence:</p>
  <ul>
  <li>ANN weights &lt;-&gt; RL0 policy parameters</li>
  <li>ANN inputs &lt;-&gt; RL0 states</li>
  <li>ANN predicted values &lt;-&gt; RL0 deterministic action given the state</li>
  <li>ANN loss function for one sample &lt;-&gt; RL0 reward given the state and action</li>
  </ul>
  <p>My questions:</p>
  <ol>
  <li><p>Is it really so? In other words can an ANN used in a supervised framework can be seen as a special case of reinforcement learning with a parametrized policy and discounted rewards with <span class="math-container">$\gamma=0$</span> ? Or am I missing some fundamental difference?</p>
  </li>
  <li><p>Does this mean that every RL technique applied in this setting should fall in some supervised learning category ? How would any RL technique applied to solve a supervised problem compare with a standard supervised framework?</p>
  </li>
  </ol>
  <p>NB: question <a href="https://ai.stackexchange.com/questions/14167/can-supervised-learning-be-recast-as-reinforcement-learning-problem">Can supervised learning be recast as reinforcement learning problem?</a> is very similar and answers point 1, but still left point 2 not very clear to me.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-09T14:01:41.730Z
Comments.Text: Hi. Yes the question is very similar indeed. I started also reading the article you linked. Indeed it looks that one can rewrite a supervised learning problem in a RL framework.
Comments.UserDisplayName: Thomas
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-09T14:04:39.793Z
Comments.Text: So question 1 is solved. I have some follow up on question 2. You write in your answer to the other question "However, note that the learned policy might not be able to generalise to observations not present in the training dataset. Moreover, although it is possible to solve an SL problem as an RL problem, in practice, this may not be the most appropriate approach (i.e. it may be inefficient)."
Comments.UserDisplayName: Thomas
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-09T14:06:56.450Z
Comments.Text: This seems to suggest that the answer to question 2 is "no", because you (and also Barto in his chapter) suggest that using RL to answer a SL learning problem would be suboptimal. The follow up question would be than "what would be the differences in tackling a SL problem as a RL one and why this would be not an equivalent choice but a suboptimal one"?
Comments.UserDisplayName: Thomas
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-16T07:52:31.633Z
Comments.Text: An effort to answer more quantitatively the second question is in the answer I wrote. Feedbacks welcome.
Comments.UserDisplayName: Thomas
Tags: <reinforcement-learning><supervised-learning>
Title: Reinforcement learning with $\gamma=0$ versus supervised framework
UserDisplayName: Thomas
ThreadText: # Reinforcement learning with $\gamma=0$ versus supervised framework
  
  (Asked by: username_0 on 2023-06-07)
  
  I am on a learning phase (still to enter the details but I would like first to get a bird's eye overview). I would like to understand the difference between:
  
  (1) a reinforcement learning framework, with $\gamma=0$, policy gradient optimization, and where the environment presents a new state at every iteration sampled from an input distribution. These conditions define a setup that we call RL0
  
  (2) a standard neural network supervised classification framework. Call this second setup ANN.
  
  To me the setups seem equal if we make the correspondence:
  
  * ANN weights <-> RL0 policy parameters
  * ANN inputs <-> RL0 states
  * ANN predicted values <-> RL0 deterministic action given the state
  * ANN loss function for one sample <-> RL0 reward given the state and action
  
  My questions:
  
  1. Is it really so? In other words can an ANN used in a supervised framework can be seen as a special case of reinforcement learning with a parametrized policy and discounted rewards with $\gamma=0$ ? Or am I missing some fundamental difference?
  2. Does this mean that every RL technique applied in this setting should fall in some supervised learning category ? How would any RL technique applied to solve a supervised problem compare with a standard supervised framework?
  
  NB: question [Can supervised learning be recast as reinforcement learning problem?](https://ai.stackexchange.com/questions/14167/can-supervised-learning-be-recast-as-reinforcement-learning-problem) is very similar and answers point 1, but still left point 2 not very clear to me.
  
  **Tags:** reinforcement-learning, supervised-learning
  > **Comments:**
  > **username_0**: Hi. Yes the question is very similar indeed. I started also reading the article you linked. Indeed it looks that one can rewrite a supervised learning problem in a RL framework.  
  > **username_0**: So question 1 is solved. I have some follow up on question 2. You write in your answer to the other question "However, note that the learned policy might not be able to generalise to observations not present in the training dataset. Moreover, although it is possible to solve an SL problem as an RL problem, in practice, this may not be the most appropriate approach (i.e. it may be inefficient)."  
  > **username_0**: This seems to suggest that the answer to question 2 is "no", because you (and also Barto in his chapter) suggest that using RL to answer a SL learning problem would be suboptimal. The follow up question would be than "what would be the differences in tackling a SL problem as a RL one and why this would be not an equivalent choice but a suboptimal one"?  
  > **username_0**: An effort to answer more quantitatively the second question is in the answer I wrote. Feedbacks welcome.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-06-13. Score: 0
  
  I add some thoughts of mine following Sutton and Barto, chapter 13, also to keep track of my work. This effort should go in the direction of answering point 2 of the question.
  
  **Policy gradient theorem**
  
  We consider the setting RL0 as in the OP, with the additional notation:
  
  1. Each episod goes to the terminal state immediately after the first action.
  2. Each state $s$ can be denoted as $x$, a member of the input state $X$, more suggestively remembering the notation in a supervised setting.
  3. The action is the (deterministic) prediction of the parametrized model (e.g. a neural network), so that $\pi(a,s)$ can be rewritten as $\pi(y,x)=\delta(y-Y^{pred}(x,\theta))$, where $a$ has been substituted with $y$ to match notation with a supervised setting.
  4. The reward is the squared error: $R=G=|Y^{pred}(x)-Y^{ex}(x)|^2$
  
  Such a setting looks very similar to a supervised one. Indeed the policy gradient theorem reads:
  
  $$\partial\_{\theta} J=\int dy |Y^{pred}(x,\theta)-Y^{ex}(x)|^2\partial\_{\theta}(\delta(Y^{pred}(x,\theta)-y)) [1]$$
  
  After some maninpulations and integration by parts one can rewrite the policy gradient theorem as above in the form:
  
  $$\partial\_{\theta} J=\partial\_{\theta} |Y^{pred}(x,\theta)-Y^{ex}(x)|^2 [2]$$
  
  which is the form one would use in a supervised form for the update. So up to the policy gradient theorem RL0 and ANN would be equal in this setting.
  
  **REINFORCE update**
  
  If we look at the REINFORCE update rule in this setting it reads:
  
  $\delta \theta=|Y^{pred}(x,\theta)-Y^{ex}(x)|^2 \partial\_{\theta}(\delta(Y^{pred}(x,\theta)-y))/\delta(Y^{pred}(x,\theta)-y)$
  
  To get a better understanding of the ratio of $\delta$ functions we rewrite:
  
  $\partial\_{\theta}(\delta(Y^{pred}(x,\theta)-y))/\delta(Y^{pred}(x,\theta)-y)=\delta'(Y^{pred}(x,\theta)-y)/\delta(Y^{pred}(x,\theta)-y)\partial\_{\theta}Y^{pred}(x,\theta)$
  
  So we would like to understand if the expression $\delta'(x)/\delta(x)$ can be given a finite meaning. Unluckily if we use a regularization of the delta funcion $\delta(x)=\frac{1}{\epsilon}e^{-\pi x^2/\epsilon^2}$ we see that this expression diverges when x is not null.
  
  **INTERPRETATION (for the moment):**
  
  It looks that the manipulations starting from the policy gradient theorem and leading to the reinforce update is where RL0 and ANN start to diverge substantially. I think the main issue there is that when in the derivation of the REINFORCE algorithm one multiplies and divides by $\pi(a|s)$, this operation provides larger estimators when $\pi(a|s)$ is low (because the $\pi$ at the denominator is the one surviving, the other gets replaces by sampling). As a consequence the REINFORCE update breaks when the densities are deterministic, i.e. when no sampling is necessary.
  
  Comments welcome.
  
  
  ---
  
  