bayesian-deep-learning-notes

One-phrase-summary for Bayesian deep learning papers. We here organize these papers in the following categories. But some of them might have overlap.

(1). Uncertainty in deep learning

Model uncertainty in deep learning via Bayesian modelling by variatial inference etc.

• [1705]. Concrete Dropout - [arxiv] [Note]
• [1703]. Dropout Inference in Bayesian Neural Networks with Alpha-divergences - [arxiv] [Note]
• [1703]. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? - [arxiv] [Note]
• [2016]. Uncertainty in Deep Learning - [PDF] [Blog] [Note]
• [1505]. Weight Uncertainty in Neural Networks - [arxiv] [Note]
• [2015]. On Modern Deep Learning and Variational Inference - [NIPS] [Note]
• [1995]. Bayesian learning for neural networks

(2). Probabilistic deep models

Use probabilistic model to imitate deep neural networks.

• [1711]. Deep Gaussian Mixture Models - [arxiv]
• [1411]. Deep Exponential Families - [arxiv] [Note]

(3). Probabilistic neural networks

Use probabilistic methods to do the inference in neural networks.

• [1611]. Natural-Parameter Networks: A Class of Probabilistic Neural Networks - [arxiv] [Note]

(4). Approximate inference

Approximate inference or variational inference mostly is the building block for Bayesian deep learning. Variational inference: the main idea behind variational inference is to pick a family of distributions over the latent variables with its own parameters which is called variational parameters.

(4.1) General

• [1712]. Vprop: Variational Inference using RMSprop - [arxiv] [Note]
• [1709]. Perturbative Black Box Variational Inference - [arxiv] [Note]
• [1703]. Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models - [arxiv] [Note]
• [1611]. Variational Inference via χ-Upper Bound Minimization - [arxiv]
• [1601]. Variational Inference: A Review for Statisticians - [arxiv]
• [1401]. Black Box Variational Inference - [arxiv] [Note]
• [2014]. Smoothed Gradients for Stochastic Variational Inference - [NIPS] [Note]
• [1206]. Stochastic Variational Inference - [arxiv] [Note]
• [2011]. Practical Variational Inference for Neural Networks - [NIPS]
• [1999]. An Introduction to Variational Methods for Graphical Models - [PDF]

(4.2) Reparametrization trick in variational inference

• [1506]. Variational Dropout and the Local Reparameterization Trick - [arxiv]
• [1401]. Stochastic Backpropagation and Approximate Inference in Deep Generative Models - [arxiv]
• [1312]. Auto-Encoding Variational Bayes - [arxiv] [Note]

(4.3) Others

• [NA]. A roadmap to research on EP
• [1608]. Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm - [arxiv] [Note]

(5) Continuous relaxation

Use continuous distribution to approximate discrete random variables, e.g. concrete distribution is a continuous distribution used to approximate discrete random variables.

• [1611]. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables - [arxiv]
• [1611]. Categorical Reparameterization with Gumbel-Softmax - [arxiv]

(6) Bayesian neural network pruning

Sparse prior can be used to induce sparse weight or neuron in neural networks thus favor smaller network structure for mobile devices etc.

• [1711]. Interpreting Convolutional Neural Networks Through Compression - [arXiv] [Note]
• [1705]. Structural compression of convolutional neural networks based on greedy filter pruning - [arXiv] [Note]
• [1705]. Structured Bayesian Pruning via Log-Normal Multiplicative Noise - [arxiv]
• [1705]. Bayesian Compression for Deep Learning - [arxiv] [Note]
• [1701]. Variational Dropout Sparsifies Deep Neural Networks - [arxiv]

Contribution

Any contribution is welcome. But notice that we need 'one phrase summary' to give an overview guidance to the readers RATHER THAN a list of papers. And please add yourself into the contributor list!

Contributors

• Jun Lu