Answers.Body: <p>You could save yourself some time and use a contextual bandit solver in the first place if you are secure in your knowledge that the player's actions have no impact on the game state. There is a gradient-based method based on policy functions, if you <em>need</em> to use policy-space search, as opposed to tracking expected values.</p>
  <p>Any other policy search should work too. The good thing about genetic algoirthms is that they are robust to the rest of the problem framing. The bad thing about them is that they scale poorly with problem complexity. If there are a lot of states and actions to learn, or you have designed a neural network with many parameters, it may learn slowly. I would recommend something like NEAT if you really must solve this using neural networks and genetic algorithms, as it is well established as a training approach for neural networks in policies for games and simulations.</p>
  <p>If some states are are only subtly different in their features, but result in large changes to the optimal action choice, this will be a challenge to any policy search using function approximation. I would suspect gradient-based contextual bandit solvers to be better in that case, but you would need to do the experiment to be certain.</p>
  <blockquote>
  <p>Do the optimal weights be learned that make the agent reach the target reward after some generations of training?</p>
  </blockquote>
  <p>Given enough time, and that the GA approach used is expressive enough to search the space near optimal weights, then yes.</p>
  <p>&quot;Enough time&quot; could be longer than you are willing to run the experiment.</p>
  <p>&quot;Expressive enough&quot; is difficult to quantify, and with neural networks there is not an easy way to tell by inspecting code or weights or any of the design.</p>
  <p>So, sadly, one re-wording of this answer is &quot;It will work if it works. There is no reason why it cannot work given details in the question.&quot;</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-21T07:25:44.407Z
Answers.LastActivityDate: 2022-04-21T07:34:24.623Z
Answers.LastEditDate: 2022-04-21T07:34:24.623Z
Answers.UserDisplayName: Neil Slater
Body: <p>There is a game in which the state comes one after the other without depending on the agent's action. The agent gets a reward for its actions at the end of the game. The goal of the agent is to reach a target reward when the game ends. The agent fits a policy network (returns distribution of actions to be taken at that particular state) and trains it using evolutionary algorithms with a fitness function as</p>
  <p><span class="math-container">$$
  (\text{reward in the game} - \text{target reward})^2
  $$</span></p>
  <p>Do the optimal weights be learned that make the agent reach the target reward after some generations of training?</p>
  <p>Also, can the policy search work, even if the game can not be modeled as an MDP (as there is no state to state dependency by actions in this case. The game is a kind of contextual bandit)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-23T08:42:07.440Z
Comments.Text: Can you please rephrase this question "Do the optimal weights be learned that make the agent reach the target reward after some generations of training?". To me, at least, it's unclear. Maybe there's a typo. It seems that Neil interpreted it as asking whether you can learn the weights with evolutionary algorithms with enough generations. Also, keep in mind that **a post should contain only one question**, otherwise, the post may be closed as "needs focus".
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><policy-gradients><markov-decision-process><evolutionary-algorithms><contextual-bandits>
Title: Does the policy search work if there is no state to state dependency through actions?
UserDisplayName: veerendra
ThreadText: # Does the policy search work if there is no state to state dependency through actions?
  
  (Asked by: username_0 on 2022-04-21)
  
  There is a game in which the state comes one after the other without depending on the agent's action. The agent gets a reward for its actions at the end of the game. The goal of the agent is to reach a target reward when the game ends. The agent fits a policy network (returns distribution of actions to be taken at that particular state) and trains it using evolutionary algorithms with a fitness function as
  
  $$
  (\text{reward in the game} - \text{target reward})^2
  $$
  
  Do the optimal weights be learned that make the agent reach the target reward after some generations of training?
  
  Also, can the policy search work, even if the game can not be modeled as an MDP (as there is no state to state dependency by actions in this case. The game is a kind of contextual bandit)?
  
  **Tags:** reinforcement-learning, policy-gradients, markov-decision-process, evolutionary-algorithms, contextual-bandits
  
  > **Comments:**
  > **username_1**: Can you please rephrase this question "Do the optimal weights be learned that make the agent reach the target reward after some generations of training?". To me, at least, it's unclear. Maybe there's a typo. It seems that username_2 interpreted it as asking whether you can learn the weights with evolutionary algorithms with enough generations. Also, keep in mind that **a post should contain only one question**, otherwise, the post may be closed as "needs focus".  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-04-21. Score: 0
  
  You could save yourself some time and use a contextual bandit solver in the first place if you are secure in your knowledge that the player's actions have no impact on the game state. There is a gradient-based method based on policy functions, if you *need* to use policy-space search, as opposed to tracking expected values.
  
  Any other policy search should work too. The good thing about genetic algoirthms is that they are robust to the rest of the problem framing. The bad thing about them is that they scale poorly with problem complexity. If there are a lot of states and actions to learn, or you have designed a neural network with many parameters, it may learn slowly. I would recommend something like NEAT if you really must solve this using neural networks and genetic algorithms, as it is well established as a training approach for neural networks in policies for games and simulations.
  
  If some states are are only subtly different in their features, but result in large changes to the optimal action choice, this will be a challenge to any policy search using function approximation. I would suspect gradient-based contextual bandit solvers to be better in that case, but you would need to do the experiment to be certain.
  
  > Do the optimal weights be learned that make the agent reach the target reward after some generations of training?
  
  Given enough time, and that the GA approach used is expressive enough to search the space near optimal weights, then yes.
  
  "Enough time" could be longer than you are willing to run the experiment.
  
  "Expressive enough" is difficult to quantify, and with neural networks there is not an easy way to tell by inspecting code or weights or any of the design.
  
  So, sadly, one re-wording of this answer is "It will work if it works. There is no reason why it cannot work given details in the question."
  
  
  ---
  
  