<p>This question is similar to <a href="https://ai.stackexchange.com/questions/36639/policy-and-value-function-as-a-function-of-discount">this question</a>, however it has a different question.</p>
<p>I'm learning MDP's and I'd like to know if I'm doing these exercises correctly:</p>
<p>Consider the following MDP:</p>
<p><a href="https://i.sstatic.net/R6jDc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/R6jDc.png" alt="enter image description here" /></a></p>
<p>Suppose a behavioural scientist was doing an experiment where they gave rewards to an animal. Suppose <span class="math-container">$X=5$</span> (e.g., 5 food pellets at that location). It turns out that, after repeatedly exploring the grid, the animal seems to prefer going up to the rewards of <span class="math-container">$+2$</span>.</p>
<p><strong>Question 1:</strong> In this setting, prove that no scalar discount <span class="math-container">$\gamma\in[0,1]$</span> exists for which the optimal policy is to go to <span class="math-container">$+2$</span>.</p>
<p><strong>Question 2:</strong> Consider a Monte Carlo return <span class="math-container">$$G_t=R_{t+1}+f(R_{t+1}+f(R_{t+2}+f(...))).$$</span></p>
<p>Standard discounting can be seen as applying a linear transformation <span class="math-container">$f(x) = \gamma x$</span>, by multiplying the remaining return after each step by a factor <span class="math-container">$\gamma$</span>. Consider the following alternative where instead of multiplying with a factor <span class="math-container">$\gamma$</span>, we raise the value to power: <span class="math-container">$f(x)=x^{\gamma}$</span>. Does this mathematical model better explain
the observed behaviour, in the sense that a <span class="math-container">$γ$</span> exists for which the optimal policy
goes to <span class="math-container">$+2$</span>? If so, give such a value for <span class="math-container">$γ$</span>. If not, prove why not.</p>
<p>My answer to question 1: For the optimal policy to go to <span class="math-container">$2$</span>, we need the return for going to <span class="math-container">$+2$</span> to be greater than both the return of going to <span class="math-container">$+1$</span> and <span class="math-container">$+5$</span>, i.e., mathematically</p>
<p><span class="math-container">\begin{align}
2\gamma &gt; \gamma^25 \cap 2\gamma&gt;1 \\
\frac{2}{5} &gt; \gamma \cap \gamma &gt; \frac{1}{2}
\end{align}</span></p>
<p>Since <span class="math-container">$(\frac{1}{2},\infty) \cap (-\infty, \frac{2}{5}) = \emptyset$</span>, this means that there is no such <span class="math-container">$\gamma$</span> for which the optimal policy is <span class="math-container">$+2$</span>.</p>
<p>Answer to question 2:</p>
<p>Again the return of <span class="math-container">$+2$</span> should be both greater than <span class="math-container">$+1$</span> and <span class="math-container">$+5$</span>:</p>
<p><span class="math-container">\begin{align}
2^{\gamma} &gt; (5^{\gamma})^{\gamma} \cap 2^{\gamma}&gt;1 \\
\gamma \ln 2 &gt; \gamma^2 \ln 5 \cap \gamma &gt; 0 \\
\frac{\ln 2}{\ln 5} &gt; \gamma \cap \gamma &gt; 0 \\
0.43 &gt; \gamma \cap \gamma &gt; 0
\end{align}</span></p>
<p>Hence for <span class="math-container">$0&lt; \gamma &lt; 0.43$</span>, the optimal policy will be to go to <span class="math-container">$+2$</span>. Is this correct?</p>

CC BY-SA 4.0
2022-07-07T16:12:54.393Z
Why are people voting to close this as opinion-based? So, now mathematics is an opinion?! What the...!
nbro
CC BY-SA 4.0
2022-07-09T08:47:09.903Z
@nbro The 3 questions look very similar so I was confused thinking they are duplicates
Aray Karjauv
CC BY-SA 4.0
2022-07-09T18:35:15.647Z
@ArayKarjauv Ha, ok, don't worry then! ;)
nbro
CC BY-SA 4.0
2022-08-29T00:21:09.777Z
Both look correct
David
<reinforcement-learning><value-functions><policies><discount-factor>
Policy and Discount Factor
user
# Policy and Discount Factor

(Asked by: username_0 on 2022-07-06)

This question is similar to [this question](https://ai.stackexchange.com/questions/36639/policy-and-value-function-as-a-function-of-discount), however it has a different question.

I'm learning MDP's and I'd like to know if I'm doing these exercises correctly:

Consider the following MDP:

[![enter image description here](https://i.sstatic.net/R6jDc.png)](https://i.sstatic.net/R6jDc.png)

Suppose a behavioural scientist was doing an experiment where they gave rewards to an animal. Suppose $X=5$ (e.g., 5 food pellets at that location). It turns out that, after repeatedly exploring the grid, the animal seems to prefer going up to the rewards of $+2$.

**Question 1:** In this setting, prove that no scalar discount $\gamma\in[0,1]$ exists for which the optimal policy is to go to $+2$.

**Question 2:** Consider a Monte Carlo return $$G\_t=R\_{t+1}+f(R\_{t+1}+f(R\_{t+2}+f(...))).$$

Standard discounting can be seen as applying a linear transformation $f(x) = \gamma x$, by multiplying the remaining return after each step by a factor $\gamma$. Consider the following alternative where instead of multiplying with a factor $\gamma$, we raise the value to power: $f(x)=x^{\gamma}$. Does this mathematical model better explain
the observed behaviour, in the sense that a $γ$ exists for which the optimal policy
goes to $+2$? If so, give such a value for $γ$. If not, prove why not.

My answer to question 1: For the optimal policy to go to $2$, we need the return for going to $+2$ to be greater than both the return of going to $+1$ and $+5$, i.e., mathematically

\begin{align}
2\gamma > \gamma^25 \cap 2\gamma>1 \\
\frac{2}{5} > \gamma \cap \gamma > \frac{1}{2}
\end{align}

Since $(\frac{1}{2},\infty) \cap (-\infty, \frac{2}{5}) = \emptyset$, this means that there is no such $\gamma$ for which the optimal policy is $+2$.

Answer to question 2:

Again the return of $+2$ should be both greater than $+1$ and $+5$:

\begin{align}
2^{\gamma} > (5^{\gamma})^{\gamma} \cap 2^{\gamma}>1 \\
\gamma \ln 2 > \gamma^2 \ln 5 \cap \gamma > 0 \\
\frac{\ln 2}{\ln 5} > \gamma \cap \gamma > 0 \\
0.43 > \gamma \cap \gamma > 0
\end{align}

Hence for $0< \gamma < 0.43$, the optimal policy will be to go to $+2$. Is this correct?

**Tags:** reinforcement-learning, value-functions, policies, discount-factor

> **Comments:**
> **username_1**: Why are people voting to close this as opinion-based? So, now mathematics is an opinion?! What the...!  
> **username_2**: @username_1 The 3 questions look very similar so I was confused thinking they are duplicates  
> **username_1**: @ArayKarjauv Ha, ok, don't worry then! ;)  
> **username_3**: Both look correct  

---

## 0 answers

