Meetup9-DeepLearningTensorFlowinAzure-Part-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<h1 align=\"center\">Deep learning with TensorFlow in Azure PART 2</h1>\n",
    "<h1 align=\"center\">Convolutional Networks</h1>\n",
    "<h1 align=\"center\">Meetup DFW Data & AI - Microsoft</h1>\n",
    "## Setting Up Environment "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 1) Deploy Linux Data Science VM in Azure - Ubuntu version with GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In order to complete this notebook, you must deploy a Linux DSVM in your azure subscription. Click [HERE](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/microsoft-ads.linux-data-science-vm-ubuntu), then click on GET IT NOW.\n",
    "\n",
    "**In Basics blade:**<br>\n",
    "**Name:** meetupdsvmgpu <br>\n",
    "**VM Disk type:** HDD<br>\n",
    "**Username:** sshuser<br>\n",
    "**Password:** Passw0rd.1!!<br>\n",
    "**Resource Group:** meetupdsvmgpu_rg <br>\n",
    "**Location:** Pick among East US, North Central US, South Central US or West US 2<br>\n",
    "\n",
    "**In Size blade:\n",
    "Size:** NC6\n",
    "\n",
    "**In Settings blade:**\n",
    "Leave as is\n",
    "\n",
    "\n",
    "[The data science VM](https://docs.microsoft.com/en-us/azure/machine-learning/machine-learning-data-science-virtual-machine-overview) can be used for training model using deep learning algorithms on GPU (Graphics processing units) based hardware. Utilizing VM scaling capabilites of Azure cloud, DSVM helps you use GPU based hardware on the cloud as per need. One can switch to a GPU based VM when training large models or need high speed computations while keeping the same OS disk. The Windows Server 2016 edition of DSVM comes pre-installed with GPU drivers, frameworks and GPU version of the deep learning algorithms. On the Linux, deep learning on GPU is enabled only on the Data Science Virtual Machine for Linux (Ubuntu) edition. You can deploy the Ubuntu/Windows-2016 edition of Data Science VM to non GPU based Azure virtual machine in which case all the deep learning frameworks will fallback to the CPU mode. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 2) SSH into the VM and git clone the meetup repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "```\n",
    "> ssh sshuser@YOUR.VM.IP.ADDRESS\n",
    "\n",
    "> cd notebooks\n",
    "\n",
    "> git clone https://github.com/pablomarin/Meetups-Data-AI-DFW.git\n",
    "\n",
    "> sudo ln -s /anaconda/envs/py35/bin/pip /usr/bin/pip3\n",
    "\n",
    "> sudo pip3 install tqdm\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3) Open the Jupyter notebook from your VM on your local browser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "> https://YOUR.VM.IP.ADDRESS:8000 <br>\n",
    "> Login with your VM username and password<br>\n",
    "> Go to the ***Meetups-Data-AI-DFW folder***<br>\n",
    "> Open the Notebook:***Meetup9-DeepLearningTensorFlowinAzure-Part-2.ipynb***<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## PART 2 - CONVOLUTIONAL NEURAL NETWORKS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 1) What are Convolutional Networks?\n",
    "In machine learning, a convolutional neural network (CNN, or ConvNet) is a class of deep, feed-forward artificial neural networks that has successfully been applied to analyzing visual imagery.<br>\n",
    "Convolutional networks were inspired by biological processes in which the connectivity pattern between neurons is inspired by the organization of the animal visual cortex. Individual cortical neurons respond to stimuli only in a restricted region of the visual field known as the receptive field. The receptive fields of different neurons partially overlap such that they cover the entire visual field.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 3 Concepts to understand ConvNets:\n",
    "\n",
    "1. **Filtering / Kernel Convolution**:<br>\n",
    "Image filtering is useful for many applications, including smoothing, sharpening, removing noise, and edge detection. A filter is defined by a kernel, which is a small array applied to each pixel and its neighbors within an image. In most applications, the center of the kernel is aligned with the current pixel, and is a square with an odd number (3, 5, 7, etc.) of elements in each dimension. The process used to apply filters to an image is known as convolution.<br>\n",
    "<img src=\"https://i.stack.imgur.com/9Iu89.gif\" alt=\"IMAGE\" /><br>\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/6/6a/Convolution_of_box_signal_with_itself2.gif\" alt=\"IMAGE\" /><br> \n",
    "According to <a href=\"https://en.wikipedia.org/wiki/Kernel_(image_processing)\" target=\"_blank\"> Wikipedia </a>: In image processing, a kernel, convolution matrix, or mask is a small matrix. It is used for blurring, sharpening, embossing, edge detection, and more. This is accomplished by doing a convolution between a kernel and an image.<br>\n",
    "This [link](http://setosa.io/ev/image-kernels/) shows a great visual explanation of what Image Kernels are.<br>\n",
    "This [link](http://www.aishack.in/tutorials/image-convolution-examples/) shows varios examples of Convolutions.<br><br>\n",
    "in Convolutional Neural Networks (CNNs) the network learns the filters that in traditional algorithms were hand-engineered. This independence from prior knowledge and human effort in feature design is a major advantage.<br>\n",
    "What's important here is that we are grouping together adjacent pixels and treating them as a collective.<br>\n",
    "In a normal, non-convolutional neural network, we would have ignored this adjacency. In a normal network, we would have connected every pixel in the input image to a neuron in the next layer. In doing so, we would not have taken advantage of the fact that pixels in an image are close together for a reason and have special meaning.<br>\n",
    "By taking advantage of this local structure, our CNN learns to classify local patterns, like shapes and objects, in an image.<br><br>\n",
    "\n",
    "2. **Pooling**: <br>\n",
    "Pooling is to progressively reduce the spatial size of the representation to reduce the amount of parameters and computation in the network, and hence to also control overfitting. The Pooling Layer operates independently on every depth slice of the input and resizes it spatially, using the MAX operation. The most common form is a pooling layer with filters of size 2x2 applied with a stride of 2 downsamples every depth slice in the input by 2 along both width and height, discarding 75% of the activations. Every MAX operation would in this case be taking a max over 4 numbers (little 2x2 region in some depth slice). The depth dimension remains unchanged. More generally, the pooling layer:\n",
    "<img width=\"36%\" src=\"http://cs231n.github.io/assets/cnn/pool.jpeg\">\n",
    "<img width=\"59%\" style=\"border-left: 1px solid black;\" src=\"http://cs231n.github.io/assets/cnn/maxpool.jpeg\">\n",
    "\n",
    "3. **Normalization / ReLUs**: <br>\n",
    "The reason we may want to have normalization layers in our CNN is that we want to have some kind of inhibition scheme.\n",
    "In neurobiology, there is a concept called “lateral inhibition”. Now what does that mean? This refers to the capacity of an excited neuron to subdue its neighbors. We basically want a significant peak so that we have a form of local maxima. This tends to create a contrast in that area, hence increasing the sensory perception. Increasing the sensory perception is a good thing! We want to have the same thing in our CNNs.<br>\n",
    "Normalization allows to diminish responses that are uniformly large for the neighborhood and make large activation more pronounced within a neighborhood i.e. create higher contrast in activation map. It is particulary useful with unbounded activation functions as RELU.<br><br>\n",
    "Lately though, It seems that these kinds of layers have a minimal impact and are not used any more. Basically, their role have been outplayed by other regularization techniques (such as dropout and batch normalization), better initializations and training methods. [This](http://cs231n.github.io/convolutional-networks/#norm) is what is written in the lecture notes for the Stanford Course CS321n on ConvNets. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 2) How ConvNets work?:\n",
    "1. Introduction:\n",
    "<a href=\"http://www.youtube.com/watch?feature=player_embedded&v=ISHGyvsT0QY\" target=\"_blank\"><img src=\"http://img.youtube.com/vi/ISHGyvsT0QY/1.jpg\" alt=\"IMAGE\" width=\"240\" height=\"180\" border=\"10\" /></a>\n",
    "2. The simplest ConvNet:\n",
    "<a href=\"http://www.youtube.com/watch?feature=player_embedded&v=utOv-BKI_vo\" target=\"_blank\"><img src=\"http://img.youtube.com/vi/utOv-BKI_vo/0.jpg\" alt=\"IMAGE\" width=\"240\" height=\"180\" border=\"10\" /></a>\n",
    "3. Full Architecture:\n",
    "<a href=\"http://www.youtube.com/watch?feature=player_embedded&v=FmpDIaiMIeA&start=525&end=1092\" target=\"_blank\"><img src=\"http://img.youtube.com/vi/FmpDIaiMIeA/0.jpg\" alt=\"IMAGE\" width=\"240\" height=\"180\" border=\"10\" /></a>\n",
    "<img src=\"http://cs231n.github.io/assets/cnn/convnet.jpeg\" alt=\"IMAGE\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This concludes our high-level discussion of Convolutional Neural Networks. \n",
    "Next you'll practice actually building these networks in TensorFlow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 3) ConvNets in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Adding Two more API calls (from PART 1 notebook) for using ConvNets in TensorFlow:\n",
    "\n",
    "11. **Convolutional Layer**:<br>\n",
    "TensorFlow provides the tf.nn.conv2d() and tf.nn.bias_add() functions to create your own convolutional layers.\n",
    "```\n",
    "# Output depth\n",
    "k_output = 64\n",
    "# Image Properties\n",
    "image_width = 10\n",
    "image_height = 10\n",
    "color_channels = 3\n",
    "# Convolution filter\n",
    "filter_size_width = 5\n",
    "filter_size_height = 5\n",
    "# Input/Image\n",
    "input = tf.placeholder(\n",
    "    tf.float32,\n",
    "    shape=[None, image_height, image_width, color_channels])\n",
    "# Weight and bias\n",
    "weight = tf.Variable(tf.truncated_normal(\n",
    "    [filter_size_height, filter_size_width, color_channels, k_output]))\n",
    "bias = tf.Variable(tf.zeros(k_output))\n",
    "# Apply Convolution\n",
    "conv_layer = tf.nn.conv2d(input, weight, strides=[1, 2, 2, 1], padding='SAME')\n",
    "# Add bias\n",
    "conv_layer = tf.nn.bias_add(conv_layer, bias)\n",
    "# Apply activation function\n",
    "conv_layer = tf.nn.relu(conv_layer)\n",
    "```\n",
    "The code above uses the tf.nn.conv2d() function to compute the convolution with weight as the filter and [1, 2, 2, 1] for the strides. TensorFlow uses a stride for each input dimension, [batch, input_height, input_width, input_channels]. We are generally always going to set the stride for batch and input_channels (i.e. the first and fourth element in the strides array) to be 1.<br><br>\n",
    "\n",
    "12. **Max Pooling Layer**:<br>\n",
    "TensorFlow provides the tf.nn.max_pool() function to apply max pooling to your convolutional layers.\n",
    "```\n",
    "conv_layer = tf.nn.conv2d(input, weight, strides=[1, 2, 2, 1], padding='SAME')\n",
    "conv_layer = tf.nn.bias_add(conv_layer, bias)\n",
    "conv_layer = tf.nn.relu(conv_layer)\n",
    "#Apply Max Pooling\n",
    "conv_layer = tf.nn.max_pool(conv_layer, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')\n",
    "```\n",
    "The tf.nn.max_pool() function performs max pooling with the ksize parameter as the size of the filter and the strides parameter as the length of the stride. 2x2 filters with a stride of 2x2 are common in practice.<br>\n",
    "The ksize and strides parameters are structured as 4-element lists, with each element corresponding to a dimension of the input tensor ([batch, height, width, channels]). For both ksize and strides, the batch and channel dimensions are typically set to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 4) LAB - Classify images from Cifar-10 dataset "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now that we understand what ConvNets are, how they work, and what functions in TensorFlow to use, let's put all the above concepts to use and build a CNN to predict/classify images from a famous dataset [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html)\n",
    "\n",
    "Click [here](Meetup9-Lab-Cifar-10-Tensorflow.ipynb ) to open the notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}