Body: <p>I am working on an RL problem where the time when the agent obtains the reward for taking action <span class="math-container">$a$</span> in time step <span class="math-container">$t$</span> is stochastic. In fact, there is no immediate reward for taking action <span class="math-container">$a$</span> in time step <span class="math-container">$t$</span>, and, for example, the agent may obtain the reward in time step <span class="math-container">$t+k$</span> (where <span class="math-container">$k&gt;1$</span>). I was wondering if this kind of reward function is categorized as a non-Markovian reward function, and which RL method works better (to approximate/find the optimum policy) in this environment?</p>
  <p>PS: It differs from the sparse reward problems. In my problem, there is a non-zero reward associated with every action taken. However, the agent does not receive any reward immediately. In fact, once the agent takes an action, like <span class="math-container">$a$</span>, at a time step like <span class="math-container">$t$</span>, the agent does not have any control over when he/she will receive the reward associated with that action. The time when the reward is received is stochastic.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-10T22:42:31.373Z
Comments.Text: The Markov property is typically defined for states, so I'm curious to know what you mean by "Markovian reward function". So, what definition of "Markovian" do you have in mind for a reward function? Moreover, when you say "realizes the reward", do you mean "get/obtain the reward"? Please, edit your post to clarify these two points. To me, it seems that you're only talking about _sparse reward functions_. However, you first need to answer these 2 questions in order to properly give an answer.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-11T08:21:29.180Z
Comments.Text: Are you also user `xuan`? If so, you have attempted to edit the question under a different account to that used to ask the question. You can ask moderators to merge accounts if you have done this in error. Running two accounts might get you in trouble, if for example you use one account to upvote posts made by the other.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-12T10:07:41.403Z
Comments.Text: This question was also asked on [Stack Overflow](https://stackoverflow.com/q/65659915/3924118).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-12T10:24:50.120Z
Comments.Text: @xuan Do you care about explaining what you mean by non-Markovian in this case?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><terminology><reference-request><reward-functions><markov-property>
Title: Is my reward function non-Markovian?
UserDisplayName: shirin elahi
ThreadText: # Is my reward function non-Markovian?
  
  (Asked by: username_0 on 2021-01-10)
  
  I am working on an RL problem where the time when the agent obtains the reward for taking action $a$ in time step $t$ is stochastic. In fact, there is no immediate reward for taking action $a$ in time step $t$, and, for example, the agent may obtain the reward in time step $t+k$ (where $k>1$). I was wondering if this kind of reward function is categorized as a non-Markovian reward function, and which RL method works better (to approximate/find the optimum policy) in this environment?
  
  PS: It differs from the sparse reward problems. In my problem, there is a non-zero reward associated with every action taken. However, the agent does not receive any reward immediately. In fact, once the agent takes an action, like $a$, at a time step like $t$, the agent does not have any control over when he/she will receive the reward associated with that action. The time when the reward is received is stochastic.
  
  **Tags:** reinforcement-learning, terminology, reference-request, reward-functions, markov-property
  
  > **Comments:**
  > **username_1**: The Markov property is typically defined for states, so I'm curious to know what you mean by "Markovian reward function". So, what definition of "Markovian" do you have in mind for a reward function? Moreover, when you say "realizes the reward", do you mean "get/obtain the reward"? Please, edit your post to clarify these two points. To me, it seems that you're only talking about _sparse reward functions_. However, you first need to answer these 2 questions in order to properly give an answer.  
  > **username_2**: Are you also user `xuan`? If so, you have attempted to edit the question under a different account to that used to ask the question. You can ask moderators to merge accounts if you have done this in error. Running two accounts might get you in trouble, if for example you use one account to upvote posts made by the other.  
  > **username_1**: This question was also asked on [Stack Overflow](https://stackoverflow.com/q/65659915/3924118).  
  > **username_1**: @xuan Do you care about explaining what you mean by non-Markovian in this case?  
  
  ---
  
  ## 0 answers
  
  