Answers.Body: <p>You are describing an environment which requires a full Markov Decision Process (MDP) to model it and reinforcement learning (RL) algorithms to solve it. You will not be able to adapt k-armed bandit algorithms, without effectively re-inventing MDPs.</p>
  <p>The two key details that make this full RL, and not a bandit problem, are:</p>
  <ul>
  <li><p>Decisions are sequential, with options and outcomes that depend on previous decisions.</p>
  </li>
  <li><p>Action choices make changes to variables (which in MDP would be part of the state description) that impact outcomes of future actions.</p>
  </li>
  </ul>
  <p>If you allow the agent to access the state including effects of previous actions encoded in a way that it has enough data to correctly predict rewards, then you have a normal MDP and most RL methods should be applicable.</p>
  <p>If you do not allow the agent to use a convenient history of past actions (and/or their effects) as part of the state, then you will have constructed a partially observable MDP (POMDP) and may need a more advanced approach to solve it. For instance, using an RNN (most likely an LSTM or a GRU architecture) to process state sequence and predict action values could learn about the hidden sequence.</p>
  <p>In terms of implementing a simulation of your environment, you will need to model it as a stateful system, and will have to include a concept of forward step in the sequence which modifies the state variables (regardless of whether these variables are made available to the agent in any observations). This would include the location information, and any other factors that change the allowed actions or outcome. As well as a step function, you will probably want a state reset function that puts the system into a starting state, or one of a range of possible starting states.</p>
  <p>If your environment is episodic (a sequence can end), then you will need a way to flag that so that the learning agent can react to the end of an episode and request a new starting state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-03T17:19:12.157Z
Answers.Text: Thank you Neil for taking the time. I already implemented it as a stateful machine and I change potential actions based on locations. There is also the end of states and as you said I already put  the agent in a starting state when it comes to somehow end. My question is related to the reward function. I know you mentioned some techniques like RNN(LSTM or GRU), but it would be helpful if you can clarify that part or maybe share some relevant links so that I may try to understand ?
Answers.UserDisplayName: Ferda-Ozdemir-Sonmez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-03T19:06:35.163Z
Answers.Text: @Ferda-Ozdemir-Sonmez: OK, the reason for my answer here is that you are calling your system a bandit problem, but it is not one. I am not clear what your problem with the reward function is?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-03T19:14:39.133Z
Answers.Text: @Ferda-Ozdemir-Sonmez - maybe this Q&A could help: https://ai.stackexchange.com/questions/22851/what-are-some-best-practices-when-trying-to-design-a-reward-function/22854#22854 ?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-03T16:37:49.860Z
Answers.LastActivityDate: 2022-07-03T16:45:05.993Z
Answers.LastEditDate: 2022-07-03T16:45:05.993Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Neil Slater has a solid answer. In general you could have a bandit algorithm in which the reward <span class="math-container">$r \sim f(. | a)$</span>, or an mdp where reward <span class="math-container">$r \sim f(. |s,a,s')$</span>. Here <span class="math-container">$a$</span> is the current action, <span class="math-container">$s$</span> is the current state, and <span class="math-container">$s'$</span> is the next state. How you encode <span class="math-container">$f$</span> if a bandit or mdp is up to you whether deterministic or based on assuming a parametric family. Of course, an MDP is not the most general setup either, but describing a setting with a highly action/state dependent history is not simple to do for mathematical analysis (and coding too).</p>
  <p>Hence if you want something more complex than what an mdp assumes, that's up to you to code/describe, or to initialize something more complex like an rnn as Neil Slater says and use that as your reward. Without nice data to train it though, an rnn reward function may not give very sensical rewards.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-03T19:19:15.423Z
Answers.LastActivityDate: 2022-07-03T19:19:15.423Z
Answers.UserDisplayName: sma
Body: <p>I want to design a multi-arm bandit system for a multi-step, multi-location system. Locations are dynamic, so I can not design the system based on them. In each location, the alternative actions that can be taken would be different. When you take correct actions, taken in correct locations, then some rewards would be earned. Some other alternative rewards can be incorporated in the system for the activities taken before reaching the correct state.</p>
  <p>I know this may not be very clear. What I want to ask is &quot;Is there a way to form a reward function so that it would take into account the order of the actions or the correctness of the order of the actions?&quot;.</p>
  <p>Previously, I have implemented some other multi-arm bandit problems, but they were more straightforward. I need some ideas to help me to implement this new type of problem from some experts.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-03T20:17:11.453Z
Comments.Text: Make sure my edits did not change the meaning of your post (it's a good idea to have a descriptive title, which is actually your question - that's what I tried to do here). Honestly, it's not fully clear to me what you're asking here or where the confusion lies, though. You have different action sets for different locations, but apart from that your post is quite unclear. Are you familiar with MDPs? Please, provide a description of your actual problem, that would be more helpful, IMHO.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><reward-functions><multi-armed-bandits><reward-design>
Title: Is there a way to form a reward function so that it would take into account the order of the actions?
UserDisplayName: Ferda-Ozdemir-Sonmez
ThreadText: # Is there a way to form a reward function so that it would take into account the order of the actions?
  
  (Asked by: username_0 on 2022-07-03)
  
  I want to design a multi-arm bandit system for a multi-step, multi-location system. Locations are dynamic, so I can not design the system based on them. In each location, the alternative actions that can be taken would be different. When you take correct actions, taken in correct locations, then some rewards would be earned. Some other alternative rewards can be incorporated in the system for the activities taken before reaching the correct state.
  
  I know this may not be very clear. What I want to ask is "Is there a way to form a reward function so that it would take into account the order of the actions or the correctness of the order of the actions?".
  
  Previously, I have implemented some other multi-arm bandit problems, but they were more straightforward. I need some ideas to help me to implement this new type of problem from some experts.
  
  **Tags:** reinforcement-learning, reward-functions, multi-armed-bandits, reward-design
  > **Comments:**
  > **username_1**: Make sure my edits did not change the meaning of your post (it's a good idea to have a descriptive title, which is actually your question - that's what I tried to do here). Honestly, it's not fully clear to me what you're asking here or where the confusion lies, though. You have different action sets for different locations, but apart from that your post is quite unclear. Are you familiar with MDPs? Please, provide a description of your actual problem, that would be more helpful, IMHO.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-07-03. Score: 1 | Accepted answer
  
  You are describing an environment which requires a full Markov Decision Process (MDP) to model it and reinforcement learning (RL) algorithms to solve it. You will not be able to adapt k-armed bandit algorithms, without effectively re-inventing MDPs.
  
  The two key details that make this full RL, and not a bandit problem, are:
  
  * Decisions are sequential, with options and outcomes that depend on previous decisions.
  * Action choices make changes to variables (which in MDP would be part of the state description) that impact outcomes of future actions.
  
  If you allow the agent to access the state including effects of previous actions encoded in a way that it has enough data to correctly predict rewards, then you have a normal MDP and most RL methods should be applicable.
  
  If you do not allow the agent to use a convenient history of past actions (and/or their effects) as part of the state, then you will have constructed a partially observable MDP (POMDP) and may need a more advanced approach to solve it. For instance, using an RNN (most likely an LSTM or a GRU architecture) to process state sequence and predict action values could learn about the hidden sequence.
  
  In terms of implementing a simulation of your environment, you will need to model it as a stateful system, and will have to include a concept of forward step in the sequence which modifies the state variables (regardless of whether these variables are made available to the agent in any observations). This would include the location information, and any other factors that change the allowed actions or outcome. As well as a step function, you will probably want a state reset function that puts the system into a starting state, or one of a range of possible starting states.
  
  If your environment is episodic (a sequence can end), then you will need a way to flag that so that the learning agent can react to the end of an episode and request a new starting state.
  
  > **Comments:**
  > **username_0**: Thank you Neil for taking the time. I already implemented it as a stateful machine and I change potential actions based on locations. There is also the end of states and as you said I already put  the agent in a starting state when it comes to somehow end. My question is related to the reward function. I know you mentioned some techniques like RNN(LSTM or GRU), but it would be helpful if you can clarify that part or maybe share some relevant links so that I may try to understand ?  
  > **username_2**: @Ferda-Ozdemir-Sonmez: OK, the reason for my answer here is that you are calling your system a bandit problem, but it is not one. I am not clear what your problem with the reward function is?  
  > **username_2**: @Ferda-Ozdemir-Sonmez - maybe this Q&A could help: https://ai.stackexchange.com/questions/22851/what-are-some-best-practices-when-trying-to-design-a-reward-function/22854#22854 ?  
  
  ---
  
  ### Answer by username_3 on 2022-07-03. Score: 0
  
  Neil Slater has a solid answer. In general you could have a bandit algorithm in which the reward $r \sim f(. | a)$, or an mdp where reward $r \sim f(. |s,a,s')$. Here $a$ is the current action, $s$ is the current state, and $s'$ is the next state. How you encode $f$ if a bandit or mdp is up to you whether deterministic or based on assuming a parametric family. Of course, an MDP is not the most general setup either, but describing a setting with a highly action/state dependent history is not simple to do for mathematical analysis (and coding too).
  
  Hence if you want something more complex than what an mdp assumes, that's up to you to code/describe, or to initialize something more complex like an rnn as Neil Slater says and use that as your reward. Without nice data to train it though, an rnn reward function may not give very sensical rewards.
  
  
  ---
  
  