<p>(The second equation for the value function of the unemployed should be
<span class="math-container">$$
v(w,U)= \max \{v(w,E); \,u[c,1]+\beta\int v(w', U) dF(w')\}. \quad (*)
$$</span>
)</p>
<blockquote>
<p>...how do you know when your problem solution require more than
one Bellman equation?</p>
</blockquote>
<p>Whenever the state space of the problem contains discrete coordinates,
there would be  &quot;multiple&quot; value functions, indexed by the discrete coordinates.
Here the state space is <span class="math-container">$[0, \infty) \times \{E, U\}$</span>. The second coordinate is discrete. So there are &quot;two&quot; value functions <span class="math-container">$v(w, E)$</span> and <span class="math-container">$v(w, U)$</span>.</p>
<blockquote>
<p>How do you realize that?</p>
</blockquote>
<p>Any variable that determines the optimization problem faced by the agent is part of the state.
In this particular example, an unemployed agent has the option to switch to being employed. An employed agent has no choice but to stay in the job---&quot;...to work forever&quot;.
The choice set, therefore the decision problem, faced by the agent are different depending on his employment status.
This tells you that employment status is part of the state.</p>
<p>(Note that, if the employed worker has the option to leave the job, the decision problem faced by the agent are still different depending on his employment status. In that case you would still have two value functions and they would be &quot;intertwined&quot;, so to speak.)</p>
<blockquote>
<p>Argue...the optimal strategy has the reservation wage property.</p>
</blockquote>
<p>This is more or less immediate from the equation <span class="math-container">$(*)$</span>.
The reservation wage <span class="math-container">$w^*$</span> is given by
<span class="math-container">$$
v(w^*,E) = u[c,1]+\beta\int v(w', U) dF(w').
$$</span>
At <span class="math-container">$w = w^*$</span>, agent is indifferent, <span class="math-container">$v(w^*, U) = v(w^*, E)$</span>.
You would expect that for <span class="math-container">$w &lt; w^*$</span>.
<span class="math-container">$$
v(w,U) = u[c,1]+\beta\int v(w', U) dF(w') &gt; v(w,E).
$$</span></p>
<p>This is a typical option-exercise problem.
The difference <span class="math-container">$v(w, U) - v(w,E)$</span> when <span class="math-container">$w &lt; w^*$</span> is the <em>option value</em>. If the offered wage <span class="math-container">$w$</span> is too low, the agent would rather keep the option and not exercise it.</p>
<p>(This is an American-type option, which can be exercised anytime. Same &quot;exercise threshold&quot; phenomenon occurs in the continuous-time setting, which is sometimes more convenient. There the threshold is given by the <em>smooth-pasting</em> condition.)</p>
<p><strong>General Comments</strong></p>
<p>The general formulation of the Bellman equation is
<span class="math-container">$$
V(s) = \max_{c \in \mathcal{C}(s)} \int_{\mathcal{S}} V(s') dF(s'; s, c),
$$</span>
where <span class="math-container">$\mathcal{C}(s)$</span> is the choice set faced by the agent at state <span class="math-container">$s$</span> and
<span class="math-container">$s' \mapsto dF(s'; s, c)$</span> is the Markov transition kernel if agent makes choice <span class="math-container">$c$</span> at state <span class="math-container">$s$</span>.
(For notational simplicity, assume no period utility/discounting/etc. The discussion would not change.)
Therefore, by definition, any variable that determines the optimization problem faced by the agent is part of the state.</p>

CC BY-SA 4.0
2020-06-18T20:14:39.760Z
"Note that, if the employed worker has the option to leave the job, the decision problem faced by the agent are still different depending on his employment status. In that case you would still have two value functions and they would be "intertwined", so to speak.)" What do you mean by that? I don't know what "intertwined" is.
mmendina
CC BY-SA 4.0
2020-06-18T20:51:46.353Z
Right now the Bellman equation for v(w, E) does not contain v(w, U) on the RHS, because there's no option to switch associated with E. If an E worked has the option to switch to U, you would get for v(w,E) an equation that is similar to $(*)$. So v(w,E) and v(w,U) would both appear on the RHS of each other's equations. Such pair of equations are actually quite common in the *real options* setting.
Michael
CC BY-SA 4.0
2020-06-18T23:26:55.943Z
Thank you I understood that. Going one step further, in that case (Employee can move from E to U) would the BE be equal to  (*) for both "workers" (E and U)?
mmendina
CC BY-SA 4.0
2020-06-18T23:39:12.283Z
No, there would be two equations, just as is currently the case---the discussion regarding what the state space is does not change. The difference would be that v(w,E) would reflect the option value of switching to U, whereas it currently does not (because there is no such option). Whenever you see max {value functions for different states} on the RHS of Bellman eqn---here max { v(w,E) ; continuation value in U which involves v(w,U) }, the eqn reflects the optionality of the problem.
Michael
CC BY-SA 4.0
2020-06-18T23:49:03.057Z
But it wouldn't be the same eq for both states , in that case: something like                
v(w,E)=(w+bv(w,E),v(U,w) ) and v(w,U)=(w+bv(w,E),v(U,w))   ??
mmendina
CC BY-SA 4.0
2020-06-18T23:57:10.130Z
That would depend on the specific model. What is the state space?  Ask yourself the question "does the specification of agent's problem---the choice set, the objective function, the constraint---depend on the variable I am looking at?" Is the choice set, the objective function, the constraint different for employed and unemployed? If answer is yes, then it's part of the *state*. The value function V is a function on the state space, whatever that is. If the state space has a discrete coordinate like E/U, then the Bellman eqn(s) is specified accordingly. For binary state like E/U, "two" eqns.
Michael
CC BY-SA 4.0
2020-06-18T20:04:13.173Z
2020-06-18T21:08:04.070Z
2020-06-18T21:08:04.070Z
Michael
<p>I'm attending to my first dynamic optimization course, and what I don't fully graps yet is that sometimes we have to use more than one bellman equation.</p>
<p>How do you realize that? I mean how do you know when your problem solution require more than one Bellman equation?</p>
<p><em><strong>For example</strong></em> this problem taken from Sargent's Recursive Macroeconomic Theory 2nd edition.</p>
<p>An unemployed worker receives each period a wage offer w drawn from the distribution F(w). The worker has to choose whether to accept the job – and therefore
to work forever – or to search for another offer and collect c in unemployment
compensation. The worker who decides to accept the job must choose the number
of hours to work in each period. The worker chooses a strategy to maximize</p>
<p><span class="math-container">$E\Sigma_{t=0}^{\infty}\beta^{t}u(y_t,l_t)$</span></p>
<p>and <span class="math-container">$y_t=c$</span> if the worker is unemployed, and <span class="math-container">$y_t=w(1-l_t)$</span> if the worker is
employed and works <span class="math-container">$(1-l_t)$</span> with <span class="math-container">$l_t$</span> leisure and  <span class="math-container">$0&lt;l_t&lt;1$</span></p>
<p>Analyze the worker’s problem. Argue that the optimal strategy has the reservation wage property. Show that the number of hours worked is the same in every
period.</p>
<p><em><strong>The solution's manual goes like this for the part of stating the Bellman equations:</strong></em></p>
<p>Let s be the vector of state variables. We choose <span class="math-container">$s=(w,0)$</span> where <span class="math-container">$w$</span> is the wage offer and <span class="math-container">$0=E$</span> if the worker is employed and <span class="math-container">$0=U$</span> if the worker is unemployed. Consider first the situation of an employed worker. Bellman’s equation is:</p>
<p><span class="math-container">$v(w,E)= max \{u[w(1-l),l]+\beta v(w,E)\}$</span></p>
<p>and for unemployed worker:</p>
<p><span class="math-container">$v(w,U)= max \{v(w,E);u[c,1]+\beta\int v(w',E)DF(w')\}$</span></p>
<p>So being more concrete.
Why the solutions requires two bellman equations and how do you realize that when reading the problem?</p>
<p>For example my first guess when trying to solve without looking the solutions I wrote:</p>
<p><span class="math-container">$v(w,E)= max \{u[w(1-l),l];u[c,1]+\beta\int v(w',E)DF(w')\}$</span></p>
<p>Why is this different?</p>
<p>Thanks in advance.</p>

CC BY-SA 4.0
2020-06-16T20:20:02.990Z
"Let s be the state variable" is somewhat misleading. Strictly speaking, the state that enter agent's decision here is a pair (wage offered, employment status). Value functions are, in general, functions of the state. For example, if the state space is a subset of $\mathbb{R}^2$, then the value function is a function, say, V(x,y). Here the second variable is binary (E or U), which gives the appearance of "2 equations".
Michael
CC BY-SA 4.0
2020-06-16T20:22:27.707Z
Thank you @Michael, so actually there isn't two equations? How do you realize that the emplyoment status is a state variable?
mmendina
CC BY-SA 4.0
2020-06-16T20:37:50.510Z
Not disputing that here there are two equations and two functions. In the case that one of the state coordinates is discrete and others non-discrete, you would multiple equations indexed by the discrete variable, so to speak.
Michael
CC BY-SA 4.0
2020-06-16T20:44:15.657Z
Here viewing the solution as two functions, one each for E and U, can be informative. If you look at the second equation v(w, U) = ..., it tells you that the w* where the two functions intersect is where agent is indifferent between E and U. If w < w*, agent chooses U. If w > w*, agent choose E. There also tells you E/U should be part of the state.
Michael
CC BY-SA 4.0
2020-06-16T20:55:55.903Z
"There also tells you E/U should be part of the state" What do you mean by this? Also,  don't fully understand why you need to write the two separate equations. What in the problem text indicates that to you?
mmendina
CC BY-SA 4.0
2020-06-16T21:07:08.683Z
The state (vector, if you'd like) consists of all the variable that determines the scenario faced by the agent (i.e. his optimization problem). Here, given the same w, your optimization problem is different if you were employed than if you were unemployed, therefore your employment status is part of the state.
Michael
CC BY-SA 4.0
2020-06-16T21:12:56.410Z
That clarified a lot, thank you very much.
mmendina
CC BY-SA 4.0
2020-06-16T22:05:06.657Z
An agent in state U has the option to switch to E, whereas the reverse is not true. Therefore the decision problems are different, and E/U is part of the state. The difference between the two functions when w < w* is the *option value*. If the offered wage w is too low, you'd rather keep the option and not exercise it. (Btw, there is a mistake in the second equation v(w,U) = ... on the RHS. The V(w,E) should be v(w,U).)
Michael
<labor-economics><dynamic-programming><bellman-equations><recursive-macroeconomics>
More than one Bellman Equation
mmendina
# More than one Bellman Equation

(Asked by: username_0 on 2020-06-16)

I'm attending to my first dynamic optimization course, and what I don't fully graps yet is that sometimes we have to use more than one bellman equation.

How do you realize that? I mean how do you know when your problem solution require more than one Bellman equation?

***For example*** this problem taken from Sargent's Recursive Macroeconomic Theory 2nd edition.

An unemployed worker receives each period a wage offer w drawn from the distribution F(w). The worker has to choose whether to accept the job – and therefore
to work forever – or to search for another offer and collect c in unemployment
compensation. The worker who decides to accept the job must choose the number
of hours to work in each period. The worker chooses a strategy to maximize

$E\Sigma\_{t=0}^{\infty}\beta^{t}u(y\_t,l\_t)$

and $y\_t=c$ if the worker is unemployed, and $y\_t=w(1-l\_t)$ if the worker is
employed and works $(1-l\_t)$ with $l\_t$ leisure and $0<l\_t<1$

Analyze the worker’s problem. Argue that the optimal strategy has the reservation wage property. Show that the number of hours worked is the same in every
period.

***The solution's manual goes like this for the part of stating the Bellman equations:***

Let s be the vector of state variables. We choose $s=(w,0)$ where $w$ is the wage offer and $0=E$ if the worker is employed and $0=U$ if the worker is unemployed. Consider first the situation of an employed worker. Bellman’s equation is:

$v(w,E)= max \{u[w(1-l),l]+\beta v(w,E)\}$

and for unemployed worker:

$v(w,U)= max \{v(w,E);u[c,1]+\beta\int v(w',E)DF(w')\}$

So being more concrete.
Why the solutions requires two bellman equations and how do you realize that when reading the problem?

For example my first guess when trying to solve without looking the solutions I wrote:

$v(w,E)= max \{u[w(1-l),l];u[c,1]+\beta\int v(w',E)DF(w')\}$

Why is this different?

Thanks in advance.

**Tags:** labor-economics, dynamic-programming, bellman-equations, recursive-macroeconomics

> **Comments:**
> **username_1**: "Let s be the state variable" is somewhat misleading. Strictly speaking, the state that enter agent's decision here is a pair (wage offered, employment status). Value functions are, in general, functions of the state. For example, if the state space is a subset of $\mathbb{R}^2$, then the value function is a function, say, V(x,y). Here the second variable is binary (E or U), which gives the appearance of "2 equations".  
> **username_0**: Thank you @username_1, so actually there isn't two equations? How do you realize that the emplyoment status is a state variable?  
> **username_1**: Not disputing that here there are two equations and two functions. In the case that one of the state coordinates is discrete and others non-discrete, you would multiple equations indexed by the discrete variable, so to speak.  
> **username_1**: Here viewing the solution as two functions, one each for E and U, can be informative. If you look at the second equation v(w, U) = ..., it tells you that the w* where the two functions intersect is where agent is indifferent between E and U. If w < w*, agent chooses U. If w > w*, agent choose E. There also tells you E/U should be part of the state.  
> **username_0**: "There also tells you E/U should be part of the state" What do you mean by this? Also,  don't fully understand why you need to write the two separate equations. What in the problem text indicates that to you?  
> **username_1**: The state (vector, if you'd like) consists of all the variable that determines the scenario faced by the agent (i.e. his optimization problem). Here, given the same w, your optimization problem is different if you were employed than if you were unemployed, therefore your employment status is part of the state.  
> **username_0**: That clarified a lot, thank you very much.  
> **username_1**: An agent in state U has the option to switch to E, whereas the reverse is not true. Therefore the decision problems are different, and E/U is part of the state. The difference between the two functions when w < w* is the *option value*. If the offered wage w is too low, you'd rather keep the option and not exercise it. (Btw, there is a mistake in the second equation v(w,U) = ... on the RHS. The V(w,E) should be v(w,U).)  

---

## 1 answers

### Answer by username_1 on 2020-06-18. Score: 1

(The second equation for the value function of the unemployed should be
$$
v(w,U)= \max \{v(w,E); \,u[c,1]+\beta\int v(w', U) dF(w')\}. \quad (\*)
$$
)

> ...how do you know when your problem solution require more than
> one Bellman equation?

Whenever the state space of the problem contains discrete coordinates,
there would be "multiple" value functions, indexed by the discrete coordinates.
Here the state space is $[0, \infty) \times \{E, U\}$. The second coordinate is discrete. So there are "two" value functions $v(w, E)$ and $v(w, U)$.

> How do you realize that?

Any variable that determines the optimization problem faced by the agent is part of the state.
In this particular example, an unemployed agent has the option to switch to being employed. An employed agent has no choice but to stay in the job---"...to work forever".
The choice set, therefore the decision problem, faced by the agent are different depending on his employment status.
This tells you that employment status is part of the state.

(Note that, if the employed worker has the option to leave the job, the decision problem faced by the agent are still different depending on his employment status. In that case you would still have two value functions and they would be "intertwined", so to speak.)

> Argue...the optimal strategy has the reservation wage property.

This is more or less immediate from the equation $(\*)$.
The reservation wage $w^\*$ is given by
$$
v(w^\*,E) = u[c,1]+\beta\int v(w', U) dF(w').
$$
At $w = w^\*$, agent is indifferent, $v(w^\*, U) = v(w^\*, E)$.
You would expect that for $w < w^\*$.
$$
v(w,U) = u[c,1]+\beta\int v(w', U) dF(w') > v(w,E).
$$

This is a typical option-exercise problem.
The difference $v(w, U) - v(w,E)$ when $w < w^\*$ is the *option value*. If the offered wage $w$ is too low, the agent would rather keep the option and not exercise it.

(This is an American-type option, which can be exercised anytime. Same "exercise threshold" phenomenon occurs in the continuous-time setting, which is sometimes more convenient. There the threshold is given by the *smooth-pasting* condition.)

**General Comments**

The general formulation of the Bellman equation is
$$
V(s) = \max\_{c \in \mathcal{C}(s)} \int\_{\mathcal{S}} V(s') dF(s'; s, c),
$$
where $\mathcal{C}(s)$ is the choice set faced by the agent at state $s$ and
$s' \mapsto dF(s'; s, c)$ is the Markov transition kernel if agent makes choice $c$ at state $s$.
(For notational simplicity, assume no period utility/discounting/etc. The discussion would not change.)
Therefore, by definition, any variable that determines the optimization problem faced by the agent is part of the state.

> **Comments:**
> **username_0**: "Note that, if the employed worker has the option to leave the job, the decision problem faced by the agent are still different depending on his employment status. In that case you would still have two value functions and they would be "intertwined", so to speak.)" What do you mean by that? I don't know what "intertwined" is.  
> **username_1**: Right now the Bellman equation for v(w, E) does not contain v(w, U) on the RHS, because there's no option to switch associated with E. If an E worked has the option to switch to U, you would get for v(w,E) an equation that is similar to $(*)$. So v(w,E) and v(w,U) would both appear on the RHS of each other's equations. Such pair of equations are actually quite common in the *real options* setting.  
> **username_0**: Thank you I understood that. Going one step further, in that case (Employee can move from E to U) would the BE be equal to  (*) for both "workers" (E and U)?  
> **username_1**: No, there would be two equations, just as is currently the case---the discussion regarding what the state space is does not change. The difference would be that v(w,E) would reflect the option value of switching to U, whereas it currently does not (because there is no such option). Whenever you see max {value functions for different states} on the RHS of Bellman eqn---here max { v(w,E) ; continuation value in U which involves v(w,U) }, the eqn reflects the optionality of the problem.  
> **username_0**: But it wouldn't be the same eq for both states , in that case: something like                 v(w,E)=(w+bv(w,E),v(U,w) ) and v(w,U)=(w+bv(w,E),v(U,w))   ??  
> **username_1**: That would depend on the specific model. What is the state space?  Ask yourself the question "does the specification of agent's problem---the choice set, the objective function, the constraint---depend on the variable I am looking at?" Is the choice set, the objective function, the constraint different for employed and unemployed? If answer is yes, then it's part of the *state*. The value function V is a function on the state space, whatever that is. If the state space has a discrete coordinate like E/U, then the Bellman eqn(s) is specified accordingly. For binary state like E/U, "two" eqns.  

---

