Paper List: Methods and Metrics for Predictive Uncertainty Quantification and Probabilistic Forecasting
This is a partial list (and a work in progress), which covers only a small subset of the many papers in this area.
-
Probabilistic forecasts, calibration and sharpness
-
Strictly Proper Scoring Rules, Prediction, and Estimation
-
Using Bayesian Model Averaging to Calibrate Forecast Ensembles
-
Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation
- By Tilmann Gneiting, Adrian E. Raftery, Anton H. Westveld III, and Tom Goldman
- 2005
- [PDF]
-
Making and Evaluating Point Forecasts
-
Probabilistic Forecasting
- By Tilmann Gneiting and Matthias Katzfuss
- 2014
- [Link]
-
Accurate Uncertainties for Deep Learning Using Calibrated Regression
- By Volodymyr Kuleshov, Nathan Fenner, Stefano Ermon
- [ArXiv]
-
On Calibration of Modern Neural Networks
- By Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger
- [ArXiv]
-
Verified Uncertainty Calibration
- By Ananya Kumar, Percy Liang, Tengyu Ma
- [ArXiv]
-
Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
- By Youngseog Chung, Willie Neiswanger, Ian Char, Jeff Schneider
- [ArXiv]
-
Individual Calibration with Randomized Forecasting
- By Shengjia Zhao, Tengyu Ma, Stefano Ermon
- [ArXiv]
-
Single-Model Uncertainties for Deep Learning
- By Natasa Tagasovska, David Lopez-Paz
- [ArXiv]
-
High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach
- By Tim Pearce, Mohamed Zaki, Alexandra Brintrup and Andy Neely
- [ArXiv]
-
Methods for Comparing Uncertainty Quantifications for Material Property Predictions
- By Kevin Tran, Willie Neiswanger, Junwoong Yoon, Qingyang Zhang, Eric Xing, Zachary W Ulissi
- Empirical comparison of UQ methods for regression
- [ArXiv],
-
Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- By Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek
- Empirical comparison of UQ methods under dataset shift, for classification
- [ArXiv]
-
How Good is the Bayes Posterior in Deep Neural Networks Really?
- Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Świątkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, Sebastian Nowozin
- Empirical study on performance of posterior of Bayesian neural networks
- [ArXiv]
-
Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
- By Eyke Hüllermeier and Willem Waegeman
- Overview of concepts and methods in UQ
- [ArXiv]
-
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
- By Balaji Lakshminarayanan, Alexander Pritzel, Charles Blundell
- [ArXiv]
-
Uncertainty in Neural Networks: Bayesian Ensembling
- By Tim Pearce, Mohamed Zaki, Alexandra Brintrup, Nicolas Anastassacos, Andy Neely
- [ArXiv]
-
Ensemble methods in Machine Learning
- By Thomas G. Dietterich
- [PDF]
-
Representation of Uncertainty in Deep Neural Networks through Sampling
- By Patrick McClure, Nikolaus Kriegeskorte
- [PDF]
-
Ensemble Sampling
- By Xiuyuan Lu, Benjamin Van Roy
- [ArXiv]
-
Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles
- By Siddhartha Jain, Ge Liu, Jonas Mueller, David Gifford
- [ArXiv]
-
Hands-on Bayesian Neural Networks - a Tutorial for Deep Learning Users
- By Laurent Valentin Jospin, Wray Buntine, Farid Boussaid, Hamid Laga, Mohammed Bennamoun
- Comprehensive tutorial on various methods for BNN
- [ArXiv]
-
Bayesian Learning for Neural Networks
- By Radford M. Neal
- Seminal work on Markov Chain Monte Carlo (MCMC) based learning for neural networks
- [PDF]
-
Stochastic Gradient Hamiltonian Monte Carlo
- By Tianqi Chen, Emily B. Fox, Carlos Guestrin
- MCMC based
- [ArXiv]
-
Bayesian Learning via Stochastic Gradient Langevin Dynamics
- By Max Welling, Yee Whye Teh
- MCMC based
- [PDF]
-
Weight Uncertainty in Neural Networks
- By Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, Daan Wierstra
- Variational Inference (VI) based
- [ArXiv]
-
Deterministic Variational Inference for Robust Bayesian Neural Networks
- By Anqi Wu, Sebastian Nowozin, Edward Meeds, Richard E. Turner, José Miguel Hernández-Lobato, Alexander L. Gaunt
- VI based
- [ArXiv]
-
Noisy Natural Gradient as Variational Inference
- Guodong Zhang, Shengyang Sun, David Duvenaud, Roger Grosse
- VI based
- [ArXiv]
-
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
- Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, Akash Srivastava
- VI based
- [ArXiv]
-
Noise Contrastive Priors for Functional Uncertainty
- By Danijar Hafner, Dustin Tran, Timothy Lillicrap, Alex Irpan, James Davidson
- [ArXiv]
-
Bayesian Layers: A Module for Neural Network Uncertainty
- By Dustin Tran, Michael W. Dusenberry, Mark van der Wilk, Danijar Hafner
- [ArXiv]
-
A Simple Baseline for Bayesian Uncertainty in Deep Learning
- By Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, Andrew Gordon Wilson
- [ArXiv]
-
Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- By Yarin Gal, Zoubin Ghahramani
- [ArXiv]
-
Deep Kernel Learning
- By Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, Eric P. Xing
- [ArXiv]
-
Neural Processes
- By Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S.M. Ali Eslami, Yee Whye Teh
- [ArXiv]
-
On the Connection between Neural Processes and Gaussian Processes with Deep Kernels
- By Tim GJ Rudner, Vincent Fortuin, Yee Whye Teh, Yarin Gal
- [PDF]
-
Predictive Uncertainty Estimation via Prior Networks
- By Andrey Malinin, Mark Gales
- [ArXiv]
-
Learning Confidence for Out-of-Distribution Detection in Neural Networks
- By Terrance DeVries, Graham W. Taylor
- [ArXiv]
-
Learning for Single-Shot Confidence Calibration in Deep Neural Networks through Stochastic Inferences
- By Seonguk Seo, Paul Hongsuck Seo, Bohyung Han
- [ArXiv]
-
Detecting Adversarial Examples and Other Misclassifications in Neural Networks by Introspection
- By Jonathan Aigrain, Marcin Detyniecki
- [ArXiv]
-
Towards Better Confidence Estimation for Neural Models
- By Vishal Thanvantri Vasudevan, Abhinav Sethy, Alireza Roshan Ghias
- [PDF]
-
Density estimation in representation space to predict model uncertainty
- By Tiago Ramalho, Miguel Miranda
- [ArXiv]
-
Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
- By Kimin Lee, Honglak Lee, Kibok Lee, Jinwoo Shin
- [ArXiv]
-
Uncertainty Estimation Using a Single Deep Deterministic Neural Network
- By Joost van Amersfoort, Lewis Smith, Yee Whye Teh, Yarin Gal
- [ArXiv]
-
Deep Bayesian Active Learning with Image Data
- By Yarin Gal, Riashat Islam, Zoubin Ghahramani
- [ArXiv]
-
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
- By Alex Kendall, Yarin Gal
- [ArXiv]
-
Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
- Kurtland Chua, Roberto Calandra, Rowan McAllister, Sergey Levine
- [ArXiv]
-
MOPO: Model-based Offline Policy Optimization
- By Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, Tengyu Ma
- [ArXiv]
-
When to Trust Your Model: Model-Based Policy Optimization
- By Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine
- [ArXiv]
-
Calibrated Model-Based Deep Reinforcement Learning
- By Ali Malik, Volodymyr Kuleshov, Jiaming Song, Danny Nemer, Harlan Seymour, Stefano Ermon
- [ArXiv]
- Incorporating Uncertainty into Deep Learning for Spoken Language Assessment
- By Andrey Malinin, Anton Ragni, Kate M. Knill, Mark J. F. Gales
- [PDF]