Variational Inference and Generative Model Reading/Resource List

Papers

Listed in (somewhat) chronological order, with some subdivisions by topic:

Reviews and Tutorials

• MI Jordan, Z. Ghahramani, T.S. Jaakkola, and L.K. Saul. An Introduction to Variational Methods for Graphical Models. Machine Learning. 1999;37:183-233
• C. Fox & SJ Roberts. A tutorial on variational Bayesian inference. Artificial intelligence review. 2012. 38;2:85-95
• D. Blei, A. Kucukelbir, and J. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859-877, 2017.
• C. Zhang, et al. Advances in Variational Inference

VI Papers Uncategorized (Yet)

• M. Hoffman, D. Blei, J. Paisley, and C. Wang. Stochastic variational inference. Journal of Machine Learning Research, 14:1303-1347, 2013.
• Kingma, D. P., & Welling, M. (2014). Auto-Encoding Variational Bayes. In International Conference on Learning Representations.
• Ranganath, R., Gerrish, S., & Blei, D. M. (2014). Black Box Variational Inference. In Artificial Intelligence and Statistics.
• Rezende, D. J., & Mohamed, S. (2015). Variational Inference with Normalizing Flows. In International Conference on Machine Learning.
• Salimans, T., Kingma, D. P., & Welling, M. (2015). Markov Chain Monte Carlo and Variational Inference: Bridging the Gap. In International Conference on Machine Learning.
• S. Mandt, M. Hoffman, and D. Blei. A variational analysis of stochastic gradient algorithms. International Conference on Machine Learning, 2016.

Structured Variational Inference

• Ranganath, R., Tran, D., & Blei, D. M. (2016). Hierarchical Variational Models. In International Conference on Machine Learning.

Non-KL Variational Inference

• R Ranganath, J Altosaar, D Tran, DM. Blei. Operator variational inference. NIPS 2016
• Q. Liu and D. Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. NIPS 2016

Implicit Models

• Li, Y., Swersky, K., & Zemel, R. (2015). Generative Moment Matching Networks. In International Conference on Machine Learning.
• Mohamed, S., & Lakshminarayanan, B. (2016). Learning in Implicit Generative Models. ArXiv.org.
• Tran, D., Ranganath, R., & Blei, D. M. (2017). Deep and Hierarchical Implicit Models. ArXiv.org.
• F. Husczar. Variational inference using implicit distributions. arXiv 2017

Important Classic Papers

• H. Robbins & S. Monro. A stochastic approximation method. The annals of mathematical statistics. 1951; 22(3):400-407
• S. Kullback & RA Leibler. On information and sufficiency. The annals of mathematical statistics. 1951 22(1):79-86
• R. J. Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine Learning, vol. 8, no. 23, 1992

Lecture Videos

• D. Blei. Variational Inference: Foundations and Innovations
• D. Blei. R. Ranganath, and S. Mohamed. Variational Inference: Foundations and Modern Methods (NIPS 2016)
• R. Ranganath. Bayesian Deep Learning and Black Box Variational Inference (2017)

Courses

• S. Mohamed. Variational Inference for Machine Learning (Tutorial)
• S. Mohamed & D. Rezende. Tutorial on Deep Generative Models
• Rajesh Ranganath's course on Deep Generative Models
• David Duvenaud's course on Learning Discrete Latent Structure
• David Duvenaud's course on Probabilistic Learning and Reasoning
• David Duvenaud's course on Differentiable Inference and Generative Models