## Further Topics on Discrete-Time Markov Control Processes

#### Onesimo Hernandez-Lerma and Jean B. Lasserre,  Springer-Verlag, New York, 1999.

ISBN: 0-387-98694-4.

## CHAPTER 7. Ergodicity and Poisson Equation

• 7.1. Introduction
• 7.2. Weighted Norms and Signed Kernels
• A. Weighted norm spaces
• B. Signed kernels
• C. Contraction maps
• 7.3. Recurrence Concepts
• A. Irreducibility and recurrence
• B. Invariant measures
• C. Conditions for irreducibility and recurence
• D. w-Geometric ergodicity
• 7.4. Examples on w-Geometric Ergodicity
• 7.4. Poisson's Equation
• A. The multichain case
• B. The unichain case
• C. Examples

• ## CHAPTER 8. Discounted Dynamic Programming with Weighted Norms

• 8.1. Introduction
• 8.2. The Control Model and Control Policies
• 8.3. The Optimality Equation
• A. Assumptions
• B. The discounted-cost optimality equation
• C. The dynamic programming operator
• D. Proof of Theorem 8.3.6.
• 8.4. Further Analysis of Value Iteration
• A. Asymptotic discount optimality
• B. Estimates of VI convergence
• C. Rolling horizon procedures
• D. Forecast horizon and elimination of non-optimal actions
• 8.5. The Weakly Continuous Case
• 8.6. Examples
• 8.7. Further Remarks

• ## CHAPTER 9. The Expected Total Cost Criterion

• 9.1. Introduction
• 9.2. Preliminaries
• A. Extended real numbers
• B. Integrability
• 9.3. The Expected Total Cost
• 9.4. Occupation Measures
• A. Expected occupation measures
• B. The sufficiency problem
• 9.5. The Optimality Equation
• A. The optimality equation
• B. Optimality criteria
• C. Deterministic stationary policies
• 9.6. The Transient Case
• A. Transient models
• B. Optikmality conditions
• C. Reduction to deterministic policies
• D. The policy iteration algorithm

• ## CHAPTER 10. Undiscounted-Cost Criteria

• 10.1. Introduction
• A. Undiscounted criteria
• B. AC-criteria
• C. Outline of the chapter
• 10.2. Preliminaries
• A. Assumptions
• B. Corollaries
• C. Discussion
• 10.3. From AC-Optimality to Undiscounted Criteria
• A. The AC-optimality inequality
• B. The AC-optimality equation
• C. Uniqueness of the ACOE
• D. Bias-optimal policies
• E. Undiscounted criteria
• 10.4. Proof of Theorem 10.3.1
• A. Preliminary lemmas
• B. Completion of the proof
• 10.5. Proof of Theorem 10.3.6.
• A. Proof of part (a)
• B. Proof of part (b)
• C. Policy iteration
• 10.6. Proof of Theorem 10.3.6.
• 10.7. Proof of Theorem 10.3.10.
• 10.8. Proof of Theorem 10.3.11.
• 10.9. Examples

• ## CHAPTER 11. Sample-Path Average Cost

• 11.1. Introduction
• A. Definitions
• B. Outline of the Chapter
• 11.2. Preliminaries
• A. Positive Harris Recurrence
• B. Limiting average variance
• 11.3. The w-Geometrically Ergodic Case
• A. Optimality in PI_DS
• B. Optimality in PI
• C. Variance Minimization
• D. Proof of Theorem 11.3.5
• E. Proof of Theorem 11.3.8
• 11.4. Strictly Unbounded Costs
• 11.5. Examples

• ## CHAPTER 12. The Linear Programming Approach

• 12.1. Introduction
• A. Outline of the chapter
• 12.2. Preliminaries
• A. Dual pairs of vector spaces
• B. Infinite linear programming,br>
• C. Approximations of linear programs
• D. Tightness and invariant measures
• 12.3. Linear Programs for the AC Problem
• A. The linear programs
• B. Solvability of (P)
• C. Absence of duality gap
• D. The Farkas alternative
• 12.4. Approximating sequences and strong duality
• A. Minimizing sequences for (P)
• B. Maximizing sequences for (P*)
• 12.5. Finite LP Approximations
• A. Aggregation
• B. Aggregation-relaxation
• C. Aggregation-relaxation-inner approximations
• 12.6. Proofs of Theorems 12.5.3., 12.5.5., 12.5.7