book/generative/gmm/gmm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gaussian Mixture Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "We will refer to Gaussian mixture model as GMM in this section.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "We will refer to Gaussian distribution as simply Gaussian in this section\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What are GMMs?\n",
    "\n",
    "A GMM uses many Gaussians to approximate the probability of events. With the model, you can guess at what place the new event will happen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When to use GMMs?\n",
    "\n",
    "GMMs are good for creating input. Because it models a distribution by combining several Gaussians, it is capable of generating sample data from that distribution.\n",
    "A GMM is also super useful in that it uses minimal parameters, one average vector $ \\mu $, and one covariance matrix $ \\sigma $ for each Gaussian. And that's it! Usually, it's used when we want a fast way to have a relatively good model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why GMMs use Gaussians?\n",
    "\n",
    "We all know gaussians right? The one that looks a bit funny, like a pile of slime. It's also called normal distribution, because of how common it is in modelling the real world. That's the reason for many cases using a few Gaussians in a GMM will produce sufficely good results.\n",
    "Gaussian also has a special property, that it can approximate any distribution given enough Gaussian distributions, which is also the reason it's used in GMM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How does GMM look?\n",
    "\n",
    "We said that GMM is basically a mix of multiple Gaussian distributions. So how does this new distribution look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gaussian(x, mean, stddev):\n",
    "    exponent = - (((x - mean) / stddev) ** 2) / 2\n",
    "    numerator = np.exp(exponent)\n",
    "    denominator = (stddev * np.sqrt(2 * np.pi))\n",
    "    return numerator / denominator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose number of gaussians\n",
    "num_gaussians = 6\n",
    "\n",
    "means = np.random.randn(num_gaussians)\n",
    "# stddev should be larger than 0\n",
    "stddevs = abs(np.random.randn(num_gaussians))\n",
    "\n",
    "# weights should sum to 1\n",
    "weights = abs(np.random.randn(num_gaussians))\n",
    "weights /= weights.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = (np.arange(-200, 201) / 20).tolist()\n",
    "\n",
    "for n in range(num_gaussians):\n",
    "    y = [gaussian(x[i], means[n], stddevs[n]) * weights[n] for i in range(len(x))]\n",
    "    plt.plot(x, y)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sum of Gaussians is the look of the new distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = [0] * len(x)\n",
    "for n in range(num_gaussians):\n",
    "    for i in range(len(x)):\n",
    "        y[i] += gaussian(x[i], means[n], stddevs[n]) * weights[n]\n",
    "\n",
    "plt.plot(x, y)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}