DYNAMIC DECISION MODELS

ALP E. ATAKAN

1. Course Description 1.1. Overview. The goal of this course is familiarize students with dynamic optimization techniques for both discrete and continuous time stochastic problems. In particular the course will present results in discrete time dynamic programming and continuous time optimal control. 1.2. Texts. I recommend that you purchase Ross (1983) and Stokey, Lucas, and Prescott (1989). 1.3. Grading. The grade will be based on a midterm exam (February 4), a nal exam and homework assignments. 1.4. Oce Hours. Wednesdays 1 - 2:30. 2. Course Outline (1) Introduction and warm-up (a) Finite horizon dynamic programming. Examples: Ross (1983), ch. 1 Reading: Ross (1983), ch. 1 (2) Mathematical preliminaries (a) Continuity, compactness and convexity of correspondences (b) Berges theorem of the maximum under convexity (c) Contraction mapping theorem, Blackwells sucient conditions for a contraction Reading: Stokey, Lucas, and Prescott (1989), ch. 3 (3) Deterministic and countable state dynamic programming with discounting (a) Bellmans functional equation and the principle of optimality (b) Value iteration and policy improvement (c) Linear programming solution (d) Contraction mapping theorem, Blackwells sucient conditions for a contraction Ross (1983), Ch. 2
1

ALP E. ATAKAN

(4)

(5)

(6)

(7)

(8)

(e) Some examples: - The classical one sector growth model - A consumption-savings problem - A seller with unknown demand - Weitzman (1979) or Pandoras problem - Multi-armed bandit problems, Ross (1983), ch.7; or Whittle (1980) Reading: Stokey, Lucas, and Prescott (1989), ch 4 and 5; Ross (1983), ch.7 Other Reading: Bersekas (2005), Volume 2, ch 1 General analysis of discounted dynamic programming problem (a) Existence of stationary optimal policies and the principle of optimality, Measure theoretic issues Shapley (1953), Blackwell (1965), Maitra (1968), Furukawa (1972) (b) Dynamic programming under convexity, Atakan (2003) (c) Envelope Theorems, Dierentiability and Monotonicity, Benveniste and Scheinkman (1979), Santos (1991) and Milgrom and Segal (2002) Minimizing Cost - Negative Dynamic Programming (a) The Bellman equation, value iteration and policy improvement, Strauch (1966), Kreps and Porteus (1977) (b) Examples: - Optimal Stopping - Bayesian Sequential Analysis - Search from an unknown distribution, Rothschild (1974) Reading: Ross (1983), ch.3 Maximizing Reward - Positive Dynamic Programming (a) The Bellman equation, value iteration, policy improvement, linear programming, Kreps and Porteus (1977) (b) Example: - Gambling Reading: Ross (1983), ch.4 Average Reward Criteria (a) Existence of stationary optimal policies (b) Relationship between the limit of discounted optima and the long-run average problem Dutta (1991) Reading: Ross (1983), ch.5 Deterministic Continuous Time Problems, Calculus of Variations and Optimal Control (a) Statement of the Continuous Time Optimization Problem

(b) Calculus of Variation Problems and the Euler Equation (c) Pontryagins Maximum Principle, derivation via dynamic programming and via variational ideas (d) The Maximum Principle in Discrete Time (e) Fixed end points, free end points, transversality conditions Reading: Rochet (2007) (on the class website) Other Reading: Bersekas (2005), Volume 1, ch 3, Kamien and Schwartz (1991) Part I. (9) Stochastic Dierential Equations and Markov Diusion Processes (a) Brownian Motion (b) Stochastic Integration and Itos rule (c) Stochastic Dierential Equations Other Reading: Oksendal (2000), ch 1 - 5, Fleming and Rishel (1975), ch 5 (10) Optimal Control of Markov Diusion Processes (a) Statement of the Problem (b) Hamilton-Jacobi-Bellman Equation (c) Filtering Problems and the Kalman-Bucy Filter (d) Optimal Stopping Problems (e) Stochastic Control with Terminal Conditions Other Reading: Fleming and Rishel (1975), ch 6, Oksendal (2000), ch 6, 10, 11

References
Atakan, A. (2003): Stochastic convexity in dynamic programming, Economic Theory, 22(2). Benveniste, L. M., and J. A. Scheinkman (1979): On the Dierentiability of the Value Function in Dynamic Models of Economics, Econometrica, 47(3). Bersekas, D. (2005): Dynamic Programming and Optimal Control. Athena Scientic. Blackwell, D. (1965): Discounted Dynamic Programming, The Annals of Mathematical Statistics, 36(1), 226 235. Dutta, P. (1991): What Do Discounted Optima Converge to?: A Theory of Discount Rate Asymptotics in Economic Models, Journal of Economic Theory, 55, 64 94. Fleming, W., and R. Rishel (1975): Deterministic and Stochastic Optimal Control. Springer-Verlang. Furukawa, N. (1972): Markovian Decision Processes with Compact Action Spaces, The Annals of Mathematical Statistics, 43(5), 1612 1622. Kamien, M., and N. Schwartz (1991): Dynamic Optimization. Elsevier. Kreps, D. M., and E. L. Porteus (1977): On the Optimality of Structured Policies in Countable Stage Decision Processes. II: Positive and Negative Problems, SIAM Journal on Applied Mathematics, 32(2), 457 466.

ALP E. ATAKAN

Maitra, A. (1968): Discounted Dynamic Programming on Compact Metric Spaces, Sankhya: Series A, 30, 211 216. Milgrom, P., and I. Segal (2002): Envelope theorems for arbitrary choice sets, Econometrica, 70(2). Oksendal, B. (2000): Stochastic Dierential Equations. Springer. Rochet, J.-C. (2007): Dynamic Optimization in Continuous Time. Ross, S. (1983): Introduction to Stochastic Dynamic Programming. Academic Press. Rothschild, M. (1974): Searching for the Lowest Price When the Distribution of Prices Is Unknown, The Journal of Political Economy, 82(4), 689 711. Santos, M. (1991): Smoothness of the Policy Function in Discrete Time Economic Models, Econometrica, 59(5). Shapley, L. (1953): Stochastic Games, Proceedings of the National Academy of Sciences, 39, 1095 1100. Stokey, N., R. Lucas, and E. Prescott (1989): Recursive Methods in Economic Dynamics. Harvard University Press. Strauch, R. (1966): Negative dynamic programming,, The Annals of Mathematical Statistics, 37, 871 890. Weitzman, M. L. (1979): Optimal Search for the Best Alternative, Econometrica, 47(3), 641 654. Whittle, P. (1980): Multi-Armed Bandits and the Gittins Index, Journal of the Royal Statistical Society. Series B (Methodological), 42(2), 143 149. MEDS, Kellogg School of Management, Northwestern University, 2001 Sheridan Road, Evanston, IL 60208 E-mail address : a-atakan@kellogg.northwestern.edu