Answers.Body: <p>You don't have a full reinforcement learning problem, but appear to have a <a href="https://en.wikipedia.org/wiki/Multi-armed_bandit" rel="nofollow noreferrer">context-free k-armed bandit problem</a>:</p>
  <ul>
  <li><p>The start state at time <span class="math-container">$t$</span> is essentially irrelevant to the problem. It does not impact available actions, reward or next state.</p>
  </li>
  <li><p>The next state at time <span class="math-container">$t+1$</span> is only of interest because it determines the reward.</p>
  </li>
  <li><p>All actions are effectively independent events, unaffected by prior history of the system.</p>
  </li>
  </ul>
  <p>As far as the agent is concerned, you can ignore the state. It may be occurring mechanically within the environment, but the agent can be optimised by observing reward values following each action. It does not need to observe the state, because there is nothing to learn from it, and there is no point in having a policy function with state as an input argument.</p>
  <p>If your action space is small, you can use any one of a number of optimisers for k-armed bandits.</p>
  <p>If your action space is large, you may need to use a gradient bandit of some kind (very similar to policy gradient methods used in RL, except there is no input layer, since using the state value as input to the function would be counter-productive).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-19T20:07:59.780Z
Answers.LastActivityDate: 2021-10-19T20:07:59.780Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am trying to implement reinforcement learning into my real-world problem.
  One thing making me hesitant to apply RL is that this real-world problem of mine is unique in a way how every state is independent of one another. The action taken by the agent at timestep <em>t</em> is the only thing that affects the state at the next timestep. (For example, in the cycle of &quot;<em>state-action-reward-next state</em>&quot;, the &quot;<em>next state</em>&quot; is solely dependent on the &quot;<em>action</em>&quot; but not the &quot;<em>state</em>&quot;.)</p>
  <p>I am wondering if the RL could still be able to learn through this scenario. If not, what other methods could be an option?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-16T08:21:08.770Z
Comments.Text: Welcome to AI stack exchange. A clarifying question: Does the available choice of action depend on the current state in any way? Also, what does the reward depend on? Some variations of your setup, e.g. where all actions are always available, should lead to optimisation of always choosing a single best action.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-19T19:51:31.877Z
Comments.Text: @NeilSlater 1. action does not depend on the current state 2. The reward depends on the result/consequence of the action (which is the next state). Observation of this specific state will give the reward at that timestep. 3. (not too sure what to answer for this one) All actions within the action space is always available, hopefully, the policy gets optimized to choosing a best action.
Comments.UserDisplayName: Ykwk
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-19T19:54:46.037Z
Comments.Text: @Ykwk That's what I needed to know - that there is one single best action, always available. I'm guessing the action space is non-trivial and/or very expensive to sample?
Comments.UserDisplayName: Neil Slater
Tags: <machine-learning><reinforcement-learning><markov-decision-process>
Title: Can RL still learn in a scenario where current state and the next state are independant?
UserDisplayName: Ykwk
ThreadText: # Can RL still learn in a scenario where current state and the next state are independant?
  
  (Asked by: username_0 on 2021-10-15)
  
  I am trying to implement reinforcement learning into my real-world problem.
  One thing making me hesitant to apply RL is that this real-world problem of mine is unique in a way how every state is independent of one another. The action taken by the agent at timestep *t* is the only thing that affects the state at the next timestep. (For example, in the cycle of "*state-action-reward-next state*", the "*next state*" is solely dependent on the "*action*" but not the "*state*".)
  
  I am wondering if the RL could still be able to learn through this scenario. If not, what other methods could be an option?
  
  **Tags:** machine-learning, reinforcement-learning, markov-decision-process
  > **Comments:**
  > **username_1**: Welcome to AI stack exchange. A clarifying question: Does the available choice of action depend on the current state in any way? Also, what does the reward depend on? Some variations of your setup, e.g. where all actions are always available, should lead to optimisation of always choosing a single best action.  
  > **username_0**: @NeilSlater 1. action does not depend on the current state 2. The reward depends on the result/consequence of the action (which is the next state). Observation of this specific state will give the reward at that timestep. 3. (not too sure what to answer for this one) All actions within the action space is always available, hopefully, the policy gets optimized to choosing a best action.  
  > **username_1**: @Ykwk That's what I needed to know - that there is one single best action, always available. I'm guessing the action space is non-trivial and/or very expensive to sample?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-10-19. Score: 1 | Accepted answer
  
  You don't have a full reinforcement learning problem, but appear to have a [context-free k-armed bandit problem](https://en.wikipedia.org/wiki/Multi-armed_bandit):
  
  * The start state at time $t$ is essentially irrelevant to the problem. It does not impact available actions, reward or next state.
  * The next state at time $t+1$ is only of interest because it determines the reward.
  * All actions are effectively independent events, unaffected by prior history of the system.
  
  As far as the agent is concerned, you can ignore the state. It may be occurring mechanically within the environment, but the agent can be optimised by observing reward values following each action. It does not need to observe the state, because there is nothing to learn from it, and there is no point in having a policy function with state as an input argument.
  
  If your action space is small, you can use any one of a number of optimisers for k-armed bandits.
  
  If your action space is large, you may need to use a gradient bandit of some kind (very similar to policy gradient methods used in RL, except there is no input layer, since using the state value as input to the function would be counter-productive).
  
  
  ---
  
  