Answers.Body: <blockquote>
    <p>Are there ways to more precisely approximate how good a single action really is considering its short and long term effects?</p>
  </blockquote>
  
  <p>To understand the short-term effects of an action, just take each of the available actions from the current state and observe the reward for each of them. The action that gives you the highest immediate reward is the best action. However, note that the reward function may change or could be stochastic. In those cases, you may need to estimate the best action e.g. by executing it multiple times.</p>
  
  <p>If you want to know the action that gives you the highest amount of reward in the long run (i.e. that gives you the highest return), then you can use one of the available RL algorithms, which were invented exactly to solve this problem. Basically, you're asking us what is the best RL algorithm. It depends on the problem, as usual.</p>
  
  <p>If you want to know the effects of an action in e.g. <span class="math-container">$n$</span> steps ahead, then you can probably formulate this problem as a truncated version of the typical reinforcement learning problem. In practice, you probably can achieve this by changing the <em>discount factor</em> so that the next <span class="math-container">$n$</span> rewards are more valuable (or are the only ones considered) than the rewards after <span class="math-container">$n$</span> steps. If you aren't familiar with discount factors, I encourage you to have a look at this concept from a reference book.</p>
  
  <p>Note that, in this answer, I am just trying to give you the idea and intuition behind a possible answer to your question (also because your question isn't really suited to provide more detailed or rigorous answers).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-18T14:37:43.463Z
Answers.Text: Thank you very much for the detailed and very insightful answer, @nbro!
Answers.UserDisplayName: Ray Walker
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-18T13:34:29.333Z
Answers.LastActivityDate: 2020-04-18T22:11:35.953Z
Answers.LastEditDate: 2020-04-18T22:11:35.953Z
Answers.UserDisplayName: nbro
Body: <p>Even when we get a valuable reward signal after every single action, this <em>immediate reward</em> only approximates the <strong>short term</strong> goodness of the action. </p>
  
  <p>To consider the <strong>long term</strong> effect of an action, we can use the return of an episode, the action value function <span class="math-container">$Q(s,a)$</span> or the advantage <span class="math-container">$A(s,a) = Q(s,a) - V(s)$</span>. However, these measures do not rate the action in isolation but take all the following actions until the end of an episode into account. </p>
  
  <p>Are there ways to more precisely approximate how good a single action really is considering its short and long term effects? </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-15T17:26:09.800Z
Comments.Text: You say "And the following actions are sampled from a policy and depend on the state transitions.". The way you sample the actions does not necessarily depend on the state transitions. For example, if you use $\epsilon$-greedy policy to sample an action from the estimate of the value function. Maybe I am misunderstanding your sentence or maybe you didn't express yourself well.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-15T21:43:41.230Z
Comments.Text: "To encounter the long term effect of an action" does not make sense. I don't think you meant to use the word "encounter"? But I am not quite sure what you are trying to say from the context - it looks like you are saying that the Q value is dependent on the current policy, and you would like some measure of how good an action is *independently* of the policy?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-17T13:11:45.827Z
Comments.Text: I've provisionally closed this post until you clarify it by addressing the issues in the comments above. Please, edit your post to clarify it and then vote to re-open it, and I will re-open it.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-18T13:17:07.623Z
Comments.Text: @nbro, I've removed this part from the question to reduce the chance of confusion. I meant that the actions taken in an episode depend on the policy and the state transitions probabilities. The policy chooses an action based on visited states while the latter determines which states are visited.
Comments.UserDisplayName: Ray Walker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-18T13:22:23.570Z
Comments.Text: @RayWalker What do you mean by "better ways"?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-18T13:24:15.770Z
Comments.Text: @NeilSlater, yes, I've meant "consider the long term effect" and not "encounter". And yes, you understood it correctly and formulated it very well: I'm asking for a way to rate a single action in isolation and thus a **policy independent rating of an action**.
Comments.UserDisplayName: Ray Walker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-18T13:26:57.307Z
Comments.Text: @RayWalker What does it mean to rate an action independently of any policy? Does that make any sense? Or in which way do you think it makes sense? The goal in reinforcement learning is to maximise reward or, actually, return. If you want to maximise only one-step ahead reward, then just choose the action that gives you the highest reward. That's it. That's the best action. If you want to maximise return, then you will have to know something about the future.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-18T13:28:21.743Z
Comments.Text: @nbro, thank you for your useful questions! I've edited the post and explained "better" to be more precise. Regarding the **independence from a policy**: When we rate an action using the return or the Q function, we estimate the value of this action based on the immediate reward and the rewards the agents get from future actions which are sampled from the policy.
Comments.UserDisplayName: Ray Walker
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-18T13:45:10.223Z
Comments.Text: @RayWalker There's no way to estimate the return if you don't have the MDP (i.e. transition + reward function) or you don't explore the environment (and to explore the environment you need a policy), that is, if you have no knowledge of the environment and you don't move, how can you understand something about the world?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-18T14:20:35.460Z
Comments.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/106875/discussion-between-ray-walker-and-nbro).
Comments.UserDisplayName: Ray Walker
Tags: <reinforcement-learning><deep-rl><rewards>
Title: What is the best measurement for how good an action of a reinforcement learning agent really is?
UserDisplayName: Ray Walker
ThreadText: # What is the best measurement for how good an action of a reinforcement learning agent really is?
  
  (Asked by: username_0 on 2020-04-15)
  
  Even when we get a valuable reward signal after every single action, this *immediate reward* only approximates the **short term** goodness of the action.
  
  To consider the **long term** effect of an action, we can use the return of an episode, the action value function $Q(s,a)$ or the advantage $A(s,a) = Q(s,a) - V(s)$. However, these measures do not rate the action in isolation but take all the following actions until the end of an episode into account.
  
  Are there ways to more precisely approximate how good a single action really is considering its short and long term effects?
  
  **Tags:** reinforcement-learning, deep-rl, rewards
  
  > **Comments:**
  > **username_1**: You say "And the following actions are sampled from a policy and depend on the state transitions.". The way you sample the actions does not necessarily depend on the state transitions. For example, if you use $\epsilon$-greedy policy to sample an action from the estimate of the value function. Maybe I am misunderstanding your sentence or maybe you didn't express yourself well.  
  > **username_2**: "To encounter the long term effect of an action" does not make sense. I don't think you meant to use the word "encounter"? But I am not quite sure what you are trying to say from the context - it looks like you are saying that the Q value is dependent on the current policy, and you would like some measure of how good an action is *independently* of the policy?  
  > **username_1**: I've provisionally closed this post until you clarify it by addressing the issues in the comments above. Please, edit your post to clarify it and then vote to re-open it, and I will re-open it.  
  > **username_0**: @username_1, I've removed this part from the question to reduce the chance of confusion. I meant that the actions taken in an episode depend on the policy and the state transitions probabilities. The policy chooses an action based on visited states while the latter determines which states are visited.  
  > **username_1**: @username_0 What do you mean by "better ways"?  
  > **username_0**: @username_2, yes, I've meant "consider the long term effect" and not "encounter". And yes, you understood it correctly and formulated it very well: I'm asking for a way to rate a single action in isolation and thus a **policy independent rating of an action**.  
  > **username_1**: @username_0 What does it mean to rate an action independently of any policy? Does that make any sense? Or in which way do you think it makes sense? The goal in reinforcement learning is to maximise reward or, actually, return. If you want to maximise only one-step ahead reward, then just choose the action that gives you the highest reward. That's it. That's the best action. If you want to maximise return, then you will have to know something about the future.  
  > **username_0**: @username_1, thank you for your useful questions! I've edited the post and explained "better" to be more precise. Regarding the **independence from a policy**: When we rate an action using the return or the Q function, we estimate the value of this action based on the immediate reward and the rewards the agents get from future actions which are sampled from the policy.  
  > **username_1**: @username_0 There's no way to estimate the return if you don't have the MDP (i.e. transition + reward function) or you don't explore the environment (and to explore the environment you need a policy), that is, if you have no knowledge of the environment and you don't move, how can you understand something about the world?  
  > **username_0**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/106875/discussion-between-ray-username_0-and-username_1).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-18. Score: 1 | Accepted answer
  
  > Are there ways to more precisely approximate how good a single action really is considering its short and long term effects?
  
  To understand the short-term effects of an action, just take each of the available actions from the current state and observe the reward for each of them. The action that gives you the highest immediate reward is the best action. However, note that the reward function may change or could be stochastic. In those cases, you may need to estimate the best action e.g. by executing it multiple times.
  
  If you want to know the action that gives you the highest amount of reward in the long run (i.e. that gives you the highest return), then you can use one of the available RL algorithms, which were invented exactly to solve this problem. Basically, you're asking us what is the best RL algorithm. It depends on the problem, as usual.
  
  If you want to know the effects of an action in e.g. $n$ steps ahead, then you can probably formulate this problem as a truncated version of the typical reinforcement learning problem. In practice, you probably can achieve this by changing the *discount factor* so that the next $n$ rewards are more valuable (or are the only ones considered) than the rewards after $n$ steps. If you aren't familiar with discount factors, I encourage you to have a look at this concept from a reference book.
  
  Note that, in this answer, I am just trying to give you the idea and intuition behind a possible answer to your question (also because your question isn't really suited to provide more detailed or rigorous answers).
  
  > **Comments:**
  > **username_0**: Thank you very much for the detailed and very insightful answer, @username_1!  
  
  ---
  
  