Answers.Body: <p>From your description, it seems that you are implementing a version of an algorithm called REINFORCE. This algorithm belongs to a family called <em>Policy Gradient methods</em>, which directly optimizes the policy network <span class="math-container">$\pi(a_t|s_t)$</span> from rewards without ever worrying about estimating a value function. This type of algorithm is usually pretty slow and presents high variance. </p>
  
  <p>The methods that you recognize as the ones using two neural networks correspond to a family called <em>Actor-Critic methods</em>. This type of algorithm uses the trajectories of rewards to estimate a <em>value network</em> <span class="math-container">$q(s_t,a_t)$</span> (called the <em>critic</em>), and, contrary to the previous family of methods, it uses the value network to train the <em>policy network</em> <span class="math-container">$\pi(a_t|s_t)$</span> (called the <em>actor</em>), instead of directly using the trajectory of rewards. This indirect dependence usually makes variance smaller and also learning faster. I recommend you have a look at chapter 13 of the book <a href="http://incompleteideas.net/book/RLbook2018.pdf" rel="nofollow noreferrer">An Introduction to Reinforcement Learning</a>.</p>
  
  <p>So, to answer your first question: it seems you are missing the family of Actor-Critic methods. I recommend you learn about them since they are very powerful (e.g., read about DDPG or SAC).</p>
  
  <p>About your second question, the standard method to "reward" a policy network is not by training it. Usually, you have a <em>reward function</em> <span class="math-container">$r(s_t,a_t)$</span> that depends on your current state <span class="math-container">$s_t$</span> and action <span class="math-container">$a_t$</span> and you modify the parameters <span class="math-container">$\theta$</span> of your network in such a way that the probability of an action <span class="math-container">$\pi(a_t|s_t)$</span> increases if the reward is positive or decreases if it is negative. More specifically, you perform stochastic gradient ascend steps like this one: </p>
  
  <p><span class="math-container">$$\theta_t\leftarrow\theta_t+\alpha\mathbb E\left[\sum_{k=t}^{T+t} \gamma^{k-t}r(s_k,a_k)\right]\nabla \log\pi(a_t|s_t,\theta_t)$$</span></p>
  
  <p>What this formula says is that if in the time-step <span class="math-container">$t$</span> you take an action <span class="math-container">$a_t$</span> in the state <span class="math-container">$s_t$</span>, wait <span class="math-container">$T$</span> steps and collect the rewards from <span class="math-container">$r(s_t,a_t)$</span> to <span class="math-container">$r(s_{t+T},a_{t+T})$</span>, then, you should modify your parameters in the direction that the policy increases the most (i.e., <span class="math-container">$\nabla \log\pi(a_t|s_t,\theta_t)$</span>) if the expected return <span class="math-container">$E\left[\sum_{k=t}^{T+t} \gamma^{k-t}r(s_k,a_k)\right]$</span> is positive, or in the direction that the policy decreases the most if that expected return is negative.      </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-02T21:01:08.030Z
Answers.Text: So, @NeilSlater are you saying that there exists a target distribution that we know beforehand? I considered that, but I found it very weird that one could know that target distribution, which is precisely the solution of the RL problem. If you take your reward as 1 for "good" results and 0 for "bad" ones, you can see that the only difference is that in REINFORCE the loss function is the loglikelihood of a specific action, while in the Cross Entropy Method it is the dot product between the loglikelihood and that target distribution.
Answers.UserDisplayName: Diego Gomez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-02T22:42:17.260Z
Answers.Text: @NeilSlater, but according to you exactly what is the loss function being used to train the network? If it is the Cross Entropy with the action itself (a vector with 0s in all positions except for the position corresponding to the action taken, say the i-th position, and a 1 in this  i-th position), then as I already explained in the previous comment, it is exactly the same as REINFORCE using a reward function that has outcomes 0 and 1. This is the case since the dot product between the loglikelihood and a vector of 0s and a 1 in the i-th entry is precisely log pi(a_t|s_t).
Answers.UserDisplayName: Diego Gomez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-03T18:48:43.780Z
Answers.Text: I still do not understand what would be the loss function. As far as I understand, it is some kind of distance between your output distribution and the target distribution that you build with your actions. Well, a typical loss function when your output is a distribution and your target is also a distribution is the Cross Entropy. Whether you derive CEM or not from the policy gradient theorem is not relevant. I'm telling you (just look at the formula) that CEM turns out to be the same as REINFORCE in the especial case that you choose a specific binary reward function and the Cross Entropy loss.
Answers.UserDisplayName: Diego Gomez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-03T18:54:59.367Z
Answers.Text: I see your point in calling my characterization "innacurate" and I will edit those parts. I could argue that CEM is a funny version of REINFORCE in the context of RL, though XD. However, I think it is more appropriate to the principal question to introduce the Actor-Critic methods as opposed to the Policy Gradient methods, not the particular CEM algorithm that turns out to be equivalent to a specific version of REINFORCE
Answers.UserDisplayName: Diego Gomez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-03T20:21:22.433Z
Answers.Text: To make CEM weight updates look the same as basic REINFORCE, you need to filter down to episodes where the only and final reward is +1 (which is not a great use for CEM - it works much better when returns can be differentiated), and also set discount factor to 1. Thanks for the edit anyway - it doesn't go quite as far as I'd like by identifying the algorithm the OP appears to be using, but at least it doesn't tell the OP that they are doing something "funny" :-)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-02T20:01:19.283Z
Answers.LastActivityDate: 2019-11-03T18:50:06.097Z
Answers.LastEditDate: 2019-11-03T18:50:06.097Z
Answers.UserDisplayName: Diego Gomez
Body: <p>I’m using a simple neural network to solve a reinforcement learning problem.</p>
  
  <p>The configuration is:</p>
  
  <p>X-inputs: The current state
  Y-outputs: The possible actions</p>
  
  <p>Whenever the network yields a “good” solution, i “reward” the network by training it a number of times.</p>
  
  <p>Whenever the network yields a “bad” or “neutral” solution, i ignore it.</p>
  
  <p>This seems to be working somewhat, but from what i read, everyone else (in broad terms) seems to be using a 2 neural network configuration for similar tasks. (Policy network and value network)</p>
  
  <p>Am i missing something? - and are there any obvious caveats of the “single network” method i am using?</p>
  
  <p>Supplemental question: Are there other methods of “rewarding” a network, aside from simply training it?</p>
  
  <p>Thanks,</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-31T09:33:00.127Z
Comments.Text: Is the output in your case a softmax over action choices? If that's the case, you are using a well-known RL method, and an answer can give its strengths and weaknesses compared to other methods.
Comments.UserDisplayName: Neil Slater
Tags: <neural-networks><machine-learning><reinforcement-learning><rewards>
Title: Neural network for reinforcement learning
UserDisplayName: skywalkerdk
ThreadText: # Neural network for reinforcement learning
  
  (Asked by: username_0 on 2019-10-31)
  
  I’m using a simple neural network to solve a reinforcement learning problem.
  
  The configuration is:
  
  X-inputs: The current state
  Y-outputs: The possible actions
  
  Whenever the network yields a “good” solution, i “reward” the network by training it a number of times.
  
  Whenever the network yields a “bad” or “neutral” solution, i ignore it.
  
  This seems to be working somewhat, but from what i read, everyone else (in broad terms) seems to be using a 2 neural network configuration for similar tasks. (Policy network and value network)
  
  Am i missing something? - and are there any obvious caveats of the “single network” method i am using?
  
  Supplemental question: Are there other methods of “rewarding” a network, aside from simply training it?
  
  Thanks,
  
  **Tags:** neural-networks, machine-learning, reinforcement-learning, rewards
  
  > **Comments:**
  > **username_1**: Is the output in your case a softmax over action choices? If that's the case, you are using a well-known RL method, and an answer can give its strengths and weaknesses compared to other methods.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-11-02. Score: 1
  
  From your description, it seems that you are implementing a version of an algorithm called REINFORCE. This algorithm belongs to a family called *Policy Gradient methods*, which directly optimizes the policy network $\pi(a\_t|s\_t)$ from rewards without ever worrying about estimating a value function. This type of algorithm is usually pretty slow and presents high variance.
  
  The methods that you recognize as the ones using two neural networks correspond to a family called *Actor-Critic methods*. This type of algorithm uses the trajectories of rewards to estimate a *value network* $q(s\_t,a\_t)$ (called the *critic*), and, contrary to the previous family of methods, it uses the value network to train the *policy network* $\pi(a\_t|s\_t)$ (called the *actor*), instead of directly using the trajectory of rewards. This indirect dependence usually makes variance smaller and also learning faster. I recommend you have a look at chapter 13 of the book [An Introduction to Reinforcement Learning](http://incompleteideas.net/book/RLbook2018.pdf).
  
  So, to answer your first question: it seems you are missing the family of Actor-Critic methods. I recommend you learn about them since they are very powerful (e.g., read about DDPG or SAC).
  
  About your second question, the standard method to "reward" a policy network is not by training it. Usually, you have a *reward function* $r(s\_t,a\_t)$ that depends on your current state $s\_t$ and action $a\_t$ and you modify the parameters $\theta$ of your network in such a way that the probability of an action $\pi(a\_t|s\_t)$ increases if the reward is positive or decreases if it is negative. More specifically, you perform stochastic gradient ascend steps like this one:
  
  $$\theta\_t\leftarrow\theta\_t+\alpha\mathbb E\left[\sum\_{k=t}^{T+t} \gamma^{k-t}r(s\_k,a\_k)\right]\nabla \log\pi(a\_t|s\_t,\theta\_t)$$
  
  What this formula says is that if in the time-step $t$ you take an action $a\_t$ in the state $s\_t$, wait $T$ steps and collect the rewards from $r(s\_t,a\_t)$ to $r(s\_{t+T},a\_{t+T})$, then, you should modify your parameters in the direction that the policy increases the most (i.e., $\nabla \log\pi(a\_t|s\_t,\theta\_t)$) if the expected return $E\left[\sum\_{k=t}^{T+t} \gamma^{k-t}r(s\_k,a\_k)\right]$ is positive, or in the direction that the policy decreases the most if that expected return is negative.
  
  > **Comments:**
  > **username_2**: So, @username_1 are you saying that there exists a target distribution that we know beforehand? I considered that, but I found it very weird that one could know that target distribution, which is precisely the solution of the RL problem. If you take your reward as 1 for "good" results and 0 for "bad" ones, you can see that the only difference is that in REINFORCE the loss function is the loglikelihood of a specific action, while in the Cross Entropy Method it is the dot product between the loglikelihood and that target distribution.  
  > **username_2**: @username_1, but according to you exactly what is the loss function being used to train the network? If it is the Cross Entropy with the action itself (a vector with 0s in all positions except for the position corresponding to the action taken, say the i-th position, and a 1 in this  i-th position), then as I already explained in the previous comment, it is exactly the same as REINFORCE using a reward function that has outcomes 0 and 1. This is the case since the dot product between the loglikelihood and a vector of 0s and a 1 in the i-th entry is precisely log pi(a_t|s_t).  
  > **username_2**: I still do not understand what would be the loss function. As far as I understand, it is some kind of distance between your output distribution and the target distribution that you build with your actions. Well, a typical loss function when your output is a distribution and your target is also a distribution is the Cross Entropy. Whether you derive CEM or not from the policy gradient theorem is not relevant. I'm telling you (just look at the formula) that CEM turns out to be the same as REINFORCE in the especial case that you choose a specific binary reward function and the Cross Entropy loss.  
  > **username_2**: I see your point in calling my characterization "innacurate" and I will edit those parts. I could argue that CEM is a funny version of REINFORCE in the context of RL, though XD. However, I think it is more appropriate to the principal question to introduce the Actor-Critic methods as opposed to the Policy Gradient methods, not the particular CEM algorithm that turns out to be equivalent to a specific version of REINFORCE  
  > **username_1**: To make CEM weight updates look the same as basic REINFORCE, you need to filter down to episodes where the only and final reward is +1 (which is not a great use for CEM - it works much better when returns can be differentiated), and also set discount factor to 1. Thanks for the edit anyway - it doesn't go quite as far as I'd like by identifying the algorithm the OP appears to be using, but at least it doesn't tell the OP that they are doing something "funny" :-)  
  
  ---
  
  