index.qmd

---
listing:
  id: summary
  template: ejs/links.ejs
  contents: summary.yml

aliases:
  - notes/index.html
---

# About the course {.unnumbered}

These notes started as the lecture notes for ECSE 506 (Stochastic Control and Decision Theory) that I teach in the Winter term of every even year. These notes are not meant to be exhaustive; rather my focus is to convey the key ideas in their simplest form. For a more exhaustive treatment of the subject, please refer to the reference books mentioned below.

::: {#summary .column-screen-inset-right style="max-width: 850px;"}
:::

If you find any typos/mistakes in the notes, please let me know. [Pull requests are welcome](https://github.com/adityam/stochastic-control/tree/quarto).


## Reference books

-   Kumar and Varaiya, [*Stochastic Systems: Estimation,
    Identification, and Adaptive
    Control,*](http://bookstore.siam.org/cl75/) Prentice Hall, 1986.
    Reprinted by SIAM 2015
    <br />
    A gentle introduction which emphaisizes the key conceptual ideas.

-   Bertsekas, [*Dynamic programming and optimal
    control*,](http://www.athenasc.com/dpbook.html) vol 1 and
    2, Athena Publications, 2005.
    <br />
    Perhaps the most comprehensive book of different topics in
    dynamic programming.

-   Puterman, [*Markov decision processes: discrete time dynamic
    programming*](http://onlinelibrary.wiley.com/book/10.1002/9780470316887), Wiley 1994.
    <br />
    Excellent source algorithms for perfectly observed systems, in
    particular, infinite horizon dynamic programs.

-   Ross, [*Introduction to Stochastic Dynamic
Programming,*](https://www.elsevier.com/books/introduction-to-stochastic-dynamic-programming/ross/978-0-12-598420-1) Academic
    Press, 1983.
    <br />
    Excellent introduction to dynamic programming, from the
    point-of-view of applied mathematics.

-   Dernardo, *Dynamic Programming: Models and Applications,*
    Prentice Hall, 1982.
    <br />
    Excellent introduction to dynamic programming, from the
    point-of-view of operations research.

-   Powell, [*Approximate Dynamic Programming*](http://adp.princeton.edu),
    John Wiley and Sons, 2011.
    <br />
    Comprehensive overview of approximate dynamic programming

-   Krishnamurty, [*Partially Observable Markov Decision
    Processes*](https://www.cambridge.org/core/books/partially-observed-markov-decision-processes),
    Cambridge University Press, 2016.
    <br />
    Comprehensive overview of POMDPs

-   Sargent and Stachurski, [*Dynamic Programming*](https://dp.quantecon.org/), 2023.
    <br />
    Nice summary of DP ideas applied to economic models. Good mix of theory and numerical examples. 

-   Kochenderfer, Wheeler, and Wray, [*Algorithms for decision making*](https://algorithmsbook.com/),
    MIT Press, 2022. 
    <br />
    Broad introduction to decision making under uncertainty. Lots of nice examples.


## How to cite these notes

To cite these lecture notes, please use:

```
@misc{506notes,
  author        = {Aditya Mahajan},
  title         = {Lecture notes on Stochastic Control and Decision Theory},
  year          = {2024},
  howpublished  = "\url{https://adityam.github.io/stochastic-control/}",
}
```