ECON 4741H - Quantitative Analysis of the Macroeconomy

Amanda M Michaud

Numerical Methods: Value Function Iteration

We have spent many weeks working with local approximation methods to compute statistics from data generated by real business cycle models. Now we will conclude the course by learning numerical methods suitable for a larger range of problems. In particular, Value Function Iteration will allow us to characterize properties of the policy function in problems for which the policy function does not have a closed-form solution. In order to do this, we will need to apply our knowledge of recursive dynamic programming.

The Value Function

As you recall, the key feature of dynamic programming problems is that we are solving the same type of maximization problem every period, subject to a dierent state variable. In our cake-eating problem, we solve how much cake to eat ( ) in the current period given how much to save for future periods ( ). We dont care why we have cake size , we only care about how much we have today and how much we will save for the future. We also take as given that, when faced with the similar problem tomorrow, we will behave optimally. Therefore, we dont need to solve tomorrows decision problem today. In order to know how much to leave for the future, we only need to know the present discounted value of all cake sizes we could save. This is given by the value function V ().1 V () = max u( ) + V ( )
[0,]

We have discussed how the above problem may be a little tricky in general because V () is an unknown function. In general we must prove this function exists and that the problem is well dened. See previous notes for some sucient conditions. We will now consider the functional equation as dening an operator T in function space: Vn = TVn1 . Explicitly2 : Vn () = max u( ) + Vn1 ( )
[0,]

Just as a function is dened as an operator that maps real numbers into real number, this operator maps functions into functions. We will consider problems that satisfy the following: Constraint set is compact. Current period return function u() is strictly concave, strictly monotone, and continuous.
1

The following are synonymous: Value Function, Bellman Equation, Recursive Dynamic Program, Functional

Equation 2 Dont be confused by the n subscript denoting the iteration number and the time subscript t we used for nite horizon models. We will only consider innite-horizon models when dealing with dynamic programming. Alternatively you could put time in the state space and specify V(T)=0 for your nite horizon

ECON 4741H - Quantitative Analysis of the Macroeconomy

Amanda M Michaud

Discounting: [0, 1) Vn1 is monotone. Contraction Mapping Theorem Under suitable assumptions, the Bellman equation denes an operator Vn = T n V0 that: Will converge to a xed point V = T V Will converge to a the same xed point V given any initial value V0

These assumptions loosely serve as sucient conditions to apply the Contraction Mapping Theorem which states the operator T maps monotone, continuous functions into strictly monotone, strictly concave, continuous functions. This means if we choose some function Vn1 that is monotone and continuous and then apply the operator T we will be left with a strictly monotone, strictly concave, continuous function Vn . Furthermore, the operator denes a contraction towards a unique xed point. This means: d(Vn Vn1 ) 0 as n . In other words, for a large number of iterations we will nd a unique function such that when put into our operator, we get the same function back: V = TV . Starting guess V0 can be any monotone, continuous function and we will converge to the same point. Here is a simple illustration of what is going on. Lets take the functional equation for the cake eating problem with utility u() = ln() and as an input function we will use V0 ( ) = ln( ). Substituting this into the functional equation leaves us with V1 () as the value of a well-dened maximization problem: V1 () = V1 () = max ln( ) + V0 ( ) max ln( ) + ln( ) (1) (2)

[0,]

[0,]

Now we solve the maximization problem to gure out the value of V1 (). It is a concave function, so we know the maximum is where the rst derivative with respect to the choice variable equals zero. Our rst order conditions give a policy function that holds for all : =
+1 .

We can

now substitute the policy function into the functional equation to get the value for all :

V1 () = ln(

) + ln( ) +1 +1

(3) (4)

ECON 4741H - Quantitative Analysis of the Macroeconomy

Amanda M Michaud

Our operator has indeed transformed a monotone, continuous function into a strictly monotone, strictly concave, continuous function. We can continue to repeat this process. Dene
1 A = ln( 1+ ) + ln( 1+ ):

V2 () =

[0,]

max ln( ) + [(1 + )ln( ) + A]

(5) (6)

Which gives new policy: =

+ 2 . 1++ 2

We can see that this is quickly converging to the

analytic solution: = (1 ) , but that it would take innite time to get there exactly.

Value Function Iteration

General Procedure for Value Function Iteration

1. Start with an arbitrary strictly increasing and strictly monotone function V0 2. Compute V1 = T V0 3. Iterate Vn = T Vn1 4. Stop when d(Vn , Vn1 ) < Where d() is some notion of distance between two functions and is a small positive number.

2.1

Numeric Considerations

Matlab is not a symbolic language. It is not going to understand how to dene and store new functions. To see this, consider what it takes to graph any function in Matlab. You must rst dene a vector x of n discrete points and then compute another set of n points dened by yi (xi ) = F (xi ). Asking Matlab to keep track of a new function G(x) = H(F (x)) isnt going to go over so well. Therefore, instead of keeping track of functions, we will keep track of sets of points and the corresponding values of the functions we are interested in at those points. Much like when we plot lines, if we use enough points, we will have a pretty good idea of the properties of the functions we are interested in. 2.1.1 1) Dening Grids

Our value functions are functions of state variables. Therefore, we will need to dene a grid points representing these state variables to be able to characterize the value function. A grid is a set of n points {xi }n [x1 , xn] such that xi < xj i=1 call this collection of points S following considerations: {si }ns . i=1 j > i. For the state variables we will Which points should we select? We must make the

ECON 4741H - Quantitative Analysis of the Macroeconomy

Amanda M Michaud

Which interval? This is rather obvious. We should dene the grid for the state variable(s) over the range of values we are interested in! For Example: Cake Eating Problem: [0, ] where is the largest size of cake we think somebody facing this problem would start out with. Neoclassical Growth Model: [(K), K] such that K ss [(K), K] and the interval captures a reasonable range of business cycle uctuations, say the steady state value for a low and high shock. How Many Points? The accuracy of our characterization of the value function is obviously increasing in the number of points we choose, however you will see our computational time is increasing in the number of points as well. In fact, for problems with multiple state variables, computation time increases exponentially with grid size. Economists, computer scientists, and others in the know refer to this as the Curse of Dimensionality. Choosing the right n seems like a nice optimization problem in itself. Selecting higher n has diminishing returns in accuracy and increasing marginal computational burden. What is the optimal choice? That requires a metric weighting accuracy and patience. You be the judge. At the minimum you should run your program for a given sized grid, then increase the grid size by 5% or so and make sure that your results do not change. Where to put the points? The easiest choice of placement for grid points is to distribute them evenly across the interval you would like to study. The Matlab command linspace does just that: xgrid = linspace(xlow,xhigh,n)

Where xlow and xhigh are the lower and upper bounds on the interval of choice and n is the number of grid points. A better idea is to put more points where the function has more curvature, ie: where the absolute value of the second derivative is very high. This can be accomplished by dening a new grid where points are dened by a concave function over linearly spaced grid points. How do we know where the value function is likely to have curvature if we do not know what the value function is? A good rule of thumb would be to use your objective function to get a sense. We will want to compute a few iterations, graph our function, and then possibly change grid points around. 2.1.2 2) Initial Guess V0

Now that we have a grid for our state variables S {si }ns , we can dene another set of i=1 points V0 {v0i }ns that correspond to the function we have in mind evaluated at each si : i=1 v0i = V0 (si ). How should we choose these points? We know by the contraction mapping theorem that the operator dened by the Bellman equation converges to a unique function regardless of the initial guess. However, we are no longer working with functions, we are working with sets of points that approximate functions. We cannot prove a theorem that says these sets of points 4

ECON 4741H - Quantitative Analysis of the Macroeconomy

Amanda M Michaud

will converge to a unique set of points because of approximation error. Therefore we will want to be careful with our initial guess. Our initial guess should be guided by the problem itself. What do we think are reasonable values? One good start is the present discounted value of the objective function given the current state. For instance, for the cake eating problem we could dene: v0i = si 1 u( 2) 1 /

where u() is a concave utility function and si [0, ] is a point on the state grid. By dividing by 2 I hope to be in a reasonable place on the utility function in terms of curvature, but to also still have enough dierence between the guess for dierent points on the grid. If you nd your initial guess is bad, take a look after a few iterations and update your beliefs on what your guess should look like. 2.1.3 3) Given {v0i }ns solve for the policy i=1

The next step is to compute the updated guess for the value function as dened by the operator. To do this we need to solve for the policy given our current guess {v0i }ns . This reduces to: i=1 gi = argmaxj u(si , sj ) + v0j In other words, we only know values of v0 for points we have chosen, so our policy must yield a state variable for tomorrow that lies on those same grid points.
3

Therefore we have to tell

Matlab: given some current state si nd another point on the state grid sj that gives the highest value of the above problem. The most basic way to do this is the following:
3

We will do better later

ECON 4741H - Quantitative Analysis of the Macroeconomy

Amanda M Michaud

Exploiting Concavity

1. Start with a given point on our state grid si 2. Calculate R(i, 1) = u(si , s1 ) + v01 for the rst possible policy choice sj = s1 3. Calculate R(i, 2) = u(si , s2 ) + v02 for the rst possible policy choice sj = s2 if R(i, 2) > R(i, 1), choosing s = s2 yields higher value than choosing s = s1 , so store gi = 2. 4. Keep going: for each subsequent j, calculate R(i, j) = u(si , sj ) + v0j . if R(i, j) > R(i, gi ), choosing s = sj yields higher value than choosing s = gi , the policy that has yielded the highest value thus far. if R(i, j) < R(i, gi ), choosing s = sj yields lower value than choosing s = gi , so we should stick with our current contender for the max: s = gi . 5. Repeat for all j = 1, 2, ...ns

Now Matlab has a find command that will search over R(i, j) and nd i , j . However, you should note that this requires making ns ns calculations and then searching one-by-one over the grid. For a grid of size 20, that is already 400 calculations and points to sift through. How can we do better? It can be shown that the operator dened by the Bellman equation maps increasing and strictly concave functions into strictly increasing and strictly concave functions. Lets use these two properties.

ECON 4741H - Quantitative Analysis of the Macroeconomy

Amanda M Michaud

Exploiting Concavity

1. Start with a given point on our state grid si 2. Calculate R(i, 1) = u(si , s1 ) + v01 for the rst possible policy choice sj = s1 3. Calculate R(i, 2) = u(si , s2 ) + v02 for the rst possible policy choice sj = s2 if R(i, 2) > R(i, 1), keep going (calc R(i, 3), compare with R(i, 2) and so on. if R(i, n + 1) < R(i, n) STOP!!! Because of concavity, we know the marginal return to increasing the state variable tomorrow will be smaller and smaller and so every point beyond sn will yield lower value.

Exploiting Monotonicity

1. Start with the rst point on our state grid si 2. Do the above (Exploiting Concavity) to nd the policy g1 for this point. Because the value function and objective functions are strictly increasing in the state variable, we know that the policy function should also be strictly increasing. ie: in the cake eating problem, when we have a larger cake, we eat more today and have more left over for tomorrow. Given s2 > s1 start the above procedure (Exploiting Concavity) with g1 as the rst point tested. Continue testing subsequent points until one yields a lower value. Stop here and store g2 . 3. Repeat for s3 by starting with g2 4. Repeat for si , i = 4, 5....n

ECON 4741H - Quantitative Analysis of the Macroeconomy

Amanda M Michaud

2.1.4

4) Calculate updated guess V1

Here we update given our policy and dene a new set of points using our policy rule gi for each i: v1i = u(si , sgi ) + v0gi 2.1.5 5) Check for convergence i = 1, 2, ...ns

Now we see if we have come close enough to a xed point. Matlab loves matrices, so lets dene our notion of distance as the following: err = abs(V1 V2)I Where I is a vector of size ns lled completely with 1s. This gives is matrix notation for the sum of the distance between v0i and v1i for each point on our grid. If err . We can go home. If not, we start the process over with V1 as our new guess for the set of points that characterize the true value function evaluated at the grid points we have chosen. dene V0 = V1 and repeat this process until we get to an iteration where err .

Closing Remarks

That is all I have for you for now. Value function iteration is a powerful tool for problems with no analytic solution. It is widely used and there are many renements on it. In general these operate by reducing computation time or better approximating the actual function. One problem you may have noticed in the above is that our choice of policy was restricted to lie on the state grid. If we have time we will discuss how to do better here.