Body: <p>This question is similar to <a href="https://ai.stackexchange.com/questions/36639/policy-and-value-function-as-a-function-of-discount">this question</a>, however it has a different question.</p>
  <p>I'm learning MDP's and I'd like to know if I'm doing these exercises correctly:</p>
  <p>Consider the following MDP:</p>
  <p><a href="https://i.sstatic.net/R6jDc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/R6jDc.png" alt="enter image description here" /></a></p>
  <p>Suppose a behavioural scientist was doing an experiment where they gave rewards to an animal. Suppose <span class="math-container">$X=5$</span> (e.g., 5 food pellets at that location). It turns out that, after repeatedly exploring the grid, the animal seems to prefer going up to the rewards of <span class="math-container">$+2$</span>.</p>
  <p><strong>Question 1:</strong> In this setting, prove that no scalar discount <span class="math-container">$\gamma\in[0,1]$</span> exists for which the optimal policy is to go to <span class="math-container">$+2$</span>.</p>
  <p><strong>Question 2:</strong> Consider a Monte Carlo return <span class="math-container">$$G_t=R_{t+1}+f(R_{t+1}+f(R_{t+2}+f(...))).$$</span></p>
  <p>Standard discounting can be seen as applying a linear transformation <span class="math-container">$f(x) = \gamma x$</span>, by multiplying the remaining return after each step by a factor <span class="math-container">$\gamma$</span>. Consider the following alternative where instead of multiplying with a factor <span class="math-container">$\gamma$</span>, we raise the value to power: <span class="math-container">$f(x)=x^{\gamma}$</span>. Does this mathematical model better explain
  the observed behaviour, in the sense that a <span class="math-container">$γ$</span> exists for which the optimal policy
  goes to <span class="math-container">$+2$</span>? If so, give such a value for <span class="math-container">$γ$</span>. If not, prove why not.</p>
  <p>My answer to question 1: For the optimal policy to go to <span class="math-container">$2$</span>, we need the return for going to <span class="math-container">$+2$</span> to be greater than both the return of going to <span class="math-container">$+1$</span> and <span class="math-container">$+5$</span>, i.e., mathematically</p>
  <p><span class="math-container">\begin{align}
  2\gamma &gt; \gamma^25 \cap 2\gamma&gt;1 \\
  \frac{2}{5} &gt; \gamma \cap \gamma &gt; \frac{1}{2}
  \end{align}</span></p>
  <p>Since <span class="math-container">$(\frac{1}{2},\infty) \cap (-\infty, \frac{2}{5}) = \emptyset$</span>, this means that there is no such <span class="math-container">$\gamma$</span> for which the optimal policy is <span class="math-container">$+2$</span>.</p>
  <p>Answer to question 2:</p>
  <p>Again the return of <span class="math-container">$+2$</span> should be both greater than <span class="math-container">$+1$</span> and <span class="math-container">$+5$</span>:</p>
  <p><span class="math-container">\begin{align}
  2^{\gamma} &gt; (5^{\gamma})^{\gamma} \cap 2^{\gamma}&gt;1 \\
  \gamma \ln 2 &gt; \gamma^2 \ln 5 \cap \gamma &gt; 0 \\
  \frac{\ln 2}{\ln 5} &gt; \gamma \cap \gamma &gt; 0 \\
  0.43 &gt; \gamma \cap \gamma &gt; 0
  \end{align}</span></p>
  <p>Hence for <span class="math-container">$0&lt; \gamma &lt; 0.43$</span>, the optimal policy will be to go to <span class="math-container">$+2$</span>. Is this correct?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-07T16:12:54.393Z
Comments.Text: Why are people voting to close this as opinion-based? So, now mathematics is an opinion?! What the...!
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-09T08:47:09.903Z
Comments.Text: @nbro The 3 questions look very similar so I was confused thinking they are duplicates
Comments.UserDisplayName: Aray Karjauv
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-09T18:35:15.647Z
Comments.Text: @ArayKarjauv Ha, ok, don't worry then! ;)
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-29T00:21:09.777Z
Comments.Text: Both look correct
Comments.UserDisplayName: David
Tags: <reinforcement-learning><value-functions><policies><discount-factor>
Title: Policy and Discount Factor
UserDisplayName: user
ThreadText: # Policy and Discount Factor
  
  (Asked by: username_0 on 2022-07-06)
  
  This question is similar to [this question](https://ai.stackexchange.com/questions/36639/policy-and-value-function-as-a-function-of-discount), however it has a different question.
  
  I'm learning MDP's and I'd like to know if I'm doing these exercises correctly:
  
  Consider the following MDP:
  
  [![enter image description here](https://i.sstatic.net/R6jDc.png)](https://i.sstatic.net/R6jDc.png)
  
  Suppose a behavioural scientist was doing an experiment where they gave rewards to an animal. Suppose $X=5$ (e.g., 5 food pellets at that location). It turns out that, after repeatedly exploring the grid, the animal seems to prefer going up to the rewards of $+2$.
  
  **Question 1:** In this setting, prove that no scalar discount $\gamma\in[0,1]$ exists for which the optimal policy is to go to $+2$.
  
  **Question 2:** Consider a Monte Carlo return $$G\_t=R\_{t+1}+f(R\_{t+1}+f(R\_{t+2}+f(...))).$$
  
  Standard discounting can be seen as applying a linear transformation $f(x) = \gamma x$, by multiplying the remaining return after each step by a factor $\gamma$. Consider the following alternative where instead of multiplying with a factor $\gamma$, we raise the value to power: $f(x)=x^{\gamma}$. Does this mathematical model better explain
  the observed behaviour, in the sense that a $γ$ exists for which the optimal policy
  goes to $+2$? If so, give such a value for $γ$. If not, prove why not.
  
  My answer to question 1: For the optimal policy to go to $2$, we need the return for going to $+2$ to be greater than both the return of going to $+1$ and $+5$, i.e., mathematically
  
  \begin{align}
  2\gamma > \gamma^25 \cap 2\gamma>1 \\
  \frac{2}{5} > \gamma \cap \gamma > \frac{1}{2}
  \end{align}
  
  Since $(\frac{1}{2},\infty) \cap (-\infty, \frac{2}{5}) = \emptyset$, this means that there is no such $\gamma$ for which the optimal policy is $+2$.
  
  Answer to question 2:
  
  Again the return of $+2$ should be both greater than $+1$ and $+5$:
  
  \begin{align}
  2^{\gamma} > (5^{\gamma})^{\gamma} \cap 2^{\gamma}>1 \\
  \gamma \ln 2 > \gamma^2 \ln 5 \cap \gamma > 0 \\
  \frac{\ln 2}{\ln 5} > \gamma \cap \gamma > 0 \\
  0.43 > \gamma \cap \gamma > 0
  \end{align}
  
  Hence for $0< \gamma < 0.43$, the optimal policy will be to go to $+2$. Is this correct?
  
  **Tags:** reinforcement-learning, value-functions, policies, discount-factor
  > **Comments:**
  > **username_1**: Why are people voting to close this as opinion-based? So, now mathematics is an opinion?! What the...!  
  > **username_2**: @nbro The 3 questions look very similar so I was confused thinking they are duplicates  
  > **username_1**: @ArayKarjauv Ha, ok, don't worry then! ;)  
  > **username_3**: Both look correct  
  
  ---
  
  ## 0 answers
  
  