Session 3 - Features_III.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](http://datascience.uci.edu/wp-content/uploads/sites/2/2014/09/data_science_logo_with_image1.png 'UCI_data_science')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Predictive Modeling with Python \n",
    "## Session #3: Feature Engineering III\n",
    "Author: [Eric Nalisnick](http://www.ics.uci.edu/~enalisni/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Schedule for Today\n",
    "\n",
    "|Start Time | Session |\n",
    "|-----------|---------|\n",
    "|8:30am     | Check In|\n",
    "|9:00am     | Feature Engineering I |\n",
    "|10:30am    | Break|\n",
    "|10:45am    | Feature Engineering II |\n",
    "|12:30pm    | Lunch |\n",
    "|1:00pm    | **Feature Engineering III** |\n",
    "|2:30pm    | Break |\n",
    "|2:45pm    | Ensembling |\n",
    "|5:00pm    | End |\n",
    "\n",
    "### Goals of this Lesson\n",
    "- Random Projections for Dimensionality Reduction\n",
    "\n",
    "### References \n",
    "- [Random Projections in Dimensionality Reduction](http://www.ime.unicamp.br/~wanderson/Artigos/randon_projection_kdd.pdf)\n",
    "- [*Dropout: A Simple Way to Prevent NNs from Overfitting](http://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf)\n",
    "- [SciKit-Learn's documentation on dimensionality reduction](http://scikit-learn.org/stable/modules/decomposition.html#decompositions)\n",
    "\n",
    "## 0.  Preliminaries\n",
    "First we need to import Numpy, Pandas, MatPlotLib..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import time\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again we need functions for shuffling the data and calculating classification errrors.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "### function for shuffling the data and labels\n",
    "def shuffle_in_unison(features, labels):\n",
    "    rng_state = np.random.get_state()\n",
    "    np.random.shuffle(features)\n",
    "    np.random.set_state(rng_state)\n",
    "    np.random.shuffle(labels)\n",
    "    \n",
    "### calculate classification errors\n",
    "# return a percentage: (number misclassified)/(total number of datapoints)\n",
    "def calc_classification_error(predictions, class_labels):\n",
    "    n = predictions.size\n",
    "    num_of_errors = 0.\n",
    "    for idx in xrange(n):\n",
    "        if (predictions[idx] >= 0.5 and class_labels[idx]==0) or (predictions[idx] < 0.5 and class_labels[idx]==1):\n",
    "            num_of_errors += 1\n",
    "    return num_of_errors/n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.1  Load the dataset of handwritten digits\n",
    "We are going to use the [MNIST dataset](https://en.wikipedia.org/wiki/MNIST_database) throughout this session.  Let's load the data...    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# load the 70,000 x 784 matrix\n",
    "from sklearn.datasets import fetch_mldata\n",
    "mnist = fetch_mldata('MNIST original')\n",
    "idxs_to_keep = []\n",
    "for idx in xrange(mnist.data.shape[0]): \n",
    "    if mnist.target[idx] == 0 or mnist.target[idx] == 1: idxs_to_keep.append(idx)\n",
    "mnist_x, mnist_y = (mnist.data[idxs_to_keep,:]/255., mnist.target[idxs_to_keep])\n",
    "shuffle_in_unison(mnist_x, mnist_y)\n",
    "print \"Dataset size: %d x %d\"%(mnist_x.shape)\n",
    "\n",
    "# make a train / test split\n",
    "x_train, x_test = (mnist_x[:10000,:], mnist_x[10000:,:])\n",
    "y_train, y_test = (mnist_y[:10000], mnist_y[10000:])\n",
    "\n",
    "# subplot containing first image\n",
    "ax1 = plt.subplot(1,2,1)\n",
    "digit = mnist_x[1,:]\n",
    "ax1.imshow(np.reshape(digit, (28, 28)), cmap='Greys_r')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 1 Random Projections\n",
    "\n",
    "We saw in the previous session that simply adding noise to the input of an Autoencoder improves it's performance.  Let's see how far we can stretch this idea.  Can we simply multiply our data by a *random matrix* and reduce its dimensionality while still preserving its structure?  Yes!  The answer is provided in a famous result called the **Johnson-Lindenstrauss Lemma**, for $\\epsilon < 1$:\n",
    "$$ (1- \\epsilon) || \\mathbf{x}_{i} - \\mathbf{x}_{j} ||^{2} \\le || \\mathbf{x}_{i}\\mathbf{W} - \\mathbf{x}_{j}\\mathbf{W}  ||^{2}  \\le (1 + \\epsilon) || \\mathbf{x}_{i} - \\mathbf{x}_{j} ||^{2} \\text{ where } \\mathbf{W} \\text{ is a random matrix. }$$  In fact Scikit-Learn has a built in function that can tell you what $\\epsilon$ should be for a given dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.random_projection import johnson_lindenstrauss_min_dim\n",
    "\n",
    "johnson_lindenstrauss_min_dim(n_samples=x_train.shape[0], eps=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "This is a nice function if we truly care about theoretical guarantees and about preserving distances, but in practice we can just see what works empirically.  Let's next generate a random matrix..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# set the random number generator for reproducability\n",
    "np.random.seed(49)\n",
    "\n",
    "# define the dimensionality of the hidden rep.\n",
    "n_components = 200\n",
    "\n",
    "# Randomly initialize the Weight matrix\n",
    "W = np.random.normal(size=(x_train.shape[1], n_components), scale=1./x_train.shape[1])\n",
    "\n",
    "train_red = np.dot(x_train, W)\n",
    "test_red = np.dot(x_test, W)\n",
    "print \"Dataset is now of size: %d x %d\"%(train_red.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Let's run a kNN classifier on the projections..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "knn = KNeighborsClassifier(n_neighbors=3)\n",
    "knn.fit(x_train, y_train) \n",
    "preds = knn.predict(x_test)\n",
    "knn_error_orig = calc_classification_error(preds, y_test) * 100\n",
    "\n",
    "lr = LogisticRegression()\n",
    "lr.fit(x_train, y_train) \n",
    "preds = lr.predict(x_test)\n",
    "lr_error_orig = calc_classification_error(preds, y_test) * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier(n_neighbors=3)\n",
    "knn.fit(train_red, y_train) \n",
    "preds = knn.predict(test_red)\n",
    "knn_error_red = calc_classification_error(preds, y_test) * 100\n",
    "\n",
    "lr = LogisticRegression()\n",
    "lr.fit(train_red, y_train) \n",
    "preds = lr.predict(test_red)\n",
    "lr_error_red = calc_classification_error(preds, y_test) * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plt.bar([0,1,2,3], [knn_error_orig, lr_error_orig, knn_error_red, lr_error_red], color=['r','r','b','b'], align='center')\n",
    "plt.xticks([0,1,2,3], ['kNN - OS', 'Log. Reg - OS', 'kNN - RP', 'Log. Reg. - RP'])\n",
    "plt.ylim([0,5.])\n",
    "plt.xlabel(\"Classifers and Features\")\n",
    "plt.ylabel(\"Classification Error\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## <span style=\"color:red\">STUDENT ACTIVITY (until end of session)</span> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\">Subtask 1: Train an Autoencoder and PCA; Compare kNN Classifer on Compressed Representation</span> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### TO DO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:red\">Subtask 2: For each model, plot classification error (y-axis) vs dimensionality of compression (x-axis).</span> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### TO DO\n",
    "\n",
    "### Shoud see graph trend downward, with classification error decreasing as dimensionality increases."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}