6. Video Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Video Analysis\n",
    "___\n",
    "\n",
    "Here we would be dealing with analysis of videos and it's various aspects like object tracking, optical flow and background reduction. OpenCV can be effectively used for analysis of real-time as well as non real-time (I don't know if it's a word, but I think you get the idea..) videos. Hence we would see different functions and algorithms defined in OpenCV related to Video Processing in the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Object Tracking in videos\n",
    "To track objects and their movements in video feeds, we would be using and comparing two different techniques defined in OpenCV.\n",
    "1. Mean Shift Tracking\n",
    "2. Camshift Tracking\n",
    "\n",
    "\n",
    "### Mean Shift Tracking\n",
    "Suppose you have a set of points (it can be a pixel distribution like histogram backprojection) and you are given a small window (like a rectangle). Now you want to move that window to the area of maximum pixel density (or maximum number of points). This is where Meanshift Tracking would be used.\n",
    "\n",
    "In Meanshift Tracking we normally pass the histogram backprojected image and initial target location. When the object moves, obviously the movement is reflected in histogram backprojected image. As a result, meanshift algorithm moves our window to\n",
    "the new location with maximum density.\n",
    "\n",
    "To use meanshift in OpenCV, first we need to setup the target, find its histogram so that we can backproject the target\n",
    "on each frame for calculation of meanshift. We also need to provide initial location of window. For histogram, only\n",
    "Hue is considered here. Also, to avoid false values due to low light, low light values are discarded using _cv2.inRange()_\n",
    "function.\n",
    "\n",
    "The following code shows an example where Meanshift Tracking is used to track cars in a video of expressway traffic. The video to be used as the input is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video controls>\n",
       "  <source src=\"videos/slow_traffic.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "HTML(\"\"\"\n",
    "<video controls>\n",
    "  <source src=\"videos/slow_traffic.mp4\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# import numpy as np\n",
    "import cv2\n",
    "cap = cv2.VideoCapture('slow_traffic.mp4')\n",
    "\n",
    "# take first frame of the video\n",
    "ret,frame = cap.read()\n",
    "\n",
    "# setup initial location of window\n",
    "r,h,c,w = 200,20,300,20  # simply hardcoded the values\n",
    "track_window = (c,r,w,h)\n",
    "\n",
    "# set up the ROI for tracking\n",
    "roi = frame[r:r+h, c:c+w]\n",
    "hsv_roi = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)\n",
    "mask = cv2.inRange(hsv_roi, np.array((0., 60.,32.)), np.array((180.,255.,255.)))\n",
    "roi_hist = cv2.calcHist([hsv_roi],[0],mask,[180],[0,180])\n",
    "cv2.normalize(roi_hist,roi_hist,0,255,cv2.NORM_MINMAX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The function _calcHist()_ used here calculates the histogram of one or more arrays. The elements of a tuple used to increment a histogram bin are taken from the corresponding input arrays at the same location. It's parameters are:\n",
    "\n",
    "* images - Source arrays. They all should have the same depth, CV_8U or CV_32F, and the same size. Each of them can have an arbitrary number of channels.\n",
    "* nimages - Number of source images.\n",
    "* channels - List of the dims channels used to compute the histogram. The first array channels are numerated from 0 to images[0].channels()-1 , the second array channels are counted from images[0].channels() to images[0].channels() + images[1].channels()-1, and so on.\n",
    "* mask - Optional mask. If the matrix is not empty, it must be an 8-bit array of the same size as images[i] . The non-zero mask elements mark the array elements counted in the histogram.\n",
    "* hist - Output histogram, which is a dense or sparse dims -dimensional array.\n",
    "* dims - Histogram dimensionality that must be positive and not greater than CV_MAX_DIMS (equal to 32 in the current OpenCV version).\n",
    "* histSize - Array of histogram sizes in each dimension.\n",
    "* ranges - Array of the dims arrays of the histogram bin boundaries in each dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Setup the termination criteria, either 10 iteration or move by atleast 1 pt\n",
    "term_crit = ( cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 1 )\n",
    "while(1):\n",
    "    ret ,frame = cap.read()\n",
    "    \n",
    "    if ret == True:\n",
    "        hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)\n",
    "        dst = cv2.calcBackProject([hsv],[0],roi_hist,[0,180],1)\n",
    "        \n",
    "        # apply meanshift to get the new location\n",
    "        ret, track_window = cv2.meanShift(dst, track_window, term_crit)\n",
    "        \n",
    "        # Draw it on image\n",
    "        x,y,w,h = track_window\n",
    "        img2 = cv2.rectangle(frame, (x,y), (x+w,y+h), 255,2)\n",
    "        cv2.imshow('img2',img2)\n",
    "        \n",
    "        k = cv2.waitKey(60) & 0xff\n",
    "        if k == 27:\n",
    "            break\n",
    "        else:\n",
    "            cv2.imwrite(chr(k)+\".jpg\",img2)\n",
    "    \n",
    "    else:\n",
    "        break\n",
    "        \n",
    "cv2.destroyAllWindows()\n",
    "cap.release()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The functions _calcBackProject()_ calculates the back project of the histogram. Similar to _calcHist()_, at each location (x, y), the function collects the values from the selected channels in the input images and finds the corresponding histogram bin. However instead of incrementing it, the function reads the bin value, scales it by scale, and stores in backProject(x,y).\n",
    "\n",
    "Then we use the _meanshift()_ function. It's parameters are:\n",
    "* probImage - Back projection of the object histogram.\n",
    "* window - Initial search window.\n",
    "* criteria - Stop criteria for the iterative search algorithm.\n",
    "\n",
    "The resulting output of the code is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video controls>\n",
       "  <source src=\"videos/meanshiftoutput.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(\"\"\"\n",
    "<video controls>\n",
    "  <source src=\"videos/meanshiftoutput.mp4\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "There is one problem with Mean Shift.  Our window always has the same size when car is farther away and it is very close to camera. That is not good. We need to adapt the window size with size and rotation of the target. To overcome this problem OpenCV has another technique called CAMshift (Continuous Adaptive Mean Shift).\n",
    "\n",
    "### CamShift Tracking\n",
    "The CamShift algorithm is iterative, meaning that it seeks to optimize the tracking criterion. In this case, we’ll set the termination criterion to perform two checks. \n",
    "* The first check is the epsilon associated with the centroids of our selected ROI and the tracked ROI according to the CamShift algorithm. If the tracked centroid has not changed by at least one pixel, then terminate the CamShift algorithm.\n",
    "* The second check controls the number of iterations of CamShift. Using more iterations will allow CamShift to (ideally) find a closer centroid match between the selected ROI and the tracked ROI; however, this comes at the cost of runtime. If the iterations are set too high, then we will drop below real-time performance, which is substantially less than ideal in most situations. Let’s go ahead and use a maximum of 10 iterations so we don’t fall into this scenario.\n",
    "\n",
    "CAMShift expects three arguments:\n",
    "1. backProj: Which is the output of the histogram back projection.\n",
    "2. roiBox: The estimated bounding box containing the object that we want to track.\n",
    "3. termination: Our termination criterion which we defined\n",
    "\n",
    "The steps followed in applying CAMShift are as follows:\n",
    "1. Capture the frames form the camera. Get the initial ROI of the region to be tracked. For each frame -\n",
    "2. Get the HSV histogram for the ROI.\n",
    "3. Normalize the histogram.\n",
    "4. Convert the frame into the HSV color space.\n",
    "5. Calculate the backprojection of the histogram of the ROI to be tracked on the HSV frame.\n",
    "6. Apply the CAMshift on the resulting backprojected image with the initial ROI location to get a new location to use in the feedback loop.\n",
    "\n",
    "Following code shows the implementation of this algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import cv2\n",
    "cap = cv2.VideoCapture('videos/slow_traffic.mp4')\n",
    "\n",
    "# take first frame of the video\n",
    "ret,frame = cap.read()\n",
    "\n",
    "# setup initial location of window\n",
    "r,h,c,w = 250,40,400,80 # simply hardcoded the values\n",
    "track_window = (c,r,w,h)\n",
    "\n",
    "# set up the ROI for tracking\n",
    "roi = frame[r:r+h, c:c+w]\n",
    "hsv_roi = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)\n",
    "mask = cv2.inRange(hsv_roi, np.array((0., 60.,32.)), np.array((180.,255.,255.)))\n",
    "roi_hist = cv2.calcHist([hsv_roi],[0],mask,[180],[0,180])\n",
    "cv2.normalize(roi_hist,roi_hist,0,255,cv2.NORM_MINMAX)\n",
    "\n",
    "# Setup the termination criteria, either 10 iteration or move by atleast 1 pt\n",
    "term_crit = ( cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 1 )\n",
    "\n",
    "while(1):\n",
    "    ret ,frame = cap.read()\n",
    "    \n",
    "    if ret == True:\n",
    "        hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)\n",
    "        dst = cv2.calcBackProject([hsv],[0],roi_hist,[0,180],1)\n",
    "        \n",
    "        # apply meanshift to get the new location\n",
    "        ret, track_window = cv2.CamShift(dst, track_window, term_crit)\n",
    "        \n",
    "        # Draw it on image\n",
    "        pts = cv2.boxPoints(ret)\n",
    "        pts = np.int0(pts)\n",
    "        img2 = cv2.polylines(frame,[pts],True, 255,2)\n",
    "        cv2.imshow('img2',img2)\n",
    "        k = cv2.waitKey(60) & 0xff\n",
    "        if k == 27:\n",
    "            break\n",
    "        else:\n",
    "            cv2.imwrite(chr(k)+\".jpg\",img2)\n",
    "            \n",
    "    else:\n",
    "        break\n",
    "cv2.destroyAllWindows()\n",
    "cap.release()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The resulting output video would have the following frames:\n",
    "\n",
    "<img src=\"images/camshift_result.jpg\"> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "This method involves hardcoding our rectangle which decides the object to be tracked. Sometimes in different conditions, this may become a tedious job, which you might have encountered if you are using your own video as input. A lot of trial and error values have to be used. So instead of that, the following code might prove useful. \n",
    "\n",
    "It is already documented and self explanatory- pretty much similiar to what we did above. The basic difference is: instead of drawing our own rectangle, we have used a method to stop the video whenever 'i' key is pressed. After that you just have to click at 4 different points on the video to mark your region of interest; this rectangle would then guide the tracking. After drawing those 4 points just click anywhere other than 'i' and the video would continue and the object would be tracked using camshift using the above explained steps. Feel free to use your own videos to test. Also make sure to make the video to run slowly, because this method isn't best suited to track fast moving objects. You can slow down your video using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "cv2.imshow(\"frame\", frame)\n",
    "cv2.waitKey(25)\n",
    "# here 25 is the time in milliseconds for which the particular frame would be displayed.\n",
    "# You can slow down the video by increasing this number accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Run the following code by specifying the path to the video you want to analyse in _camera = cv2.VideoCapture('your/video.mp4')_. Once your video is loaded, press the 'i' key and select four points surround the object that you want to track. After your four points are selected, press any key to exit input mode. Now that our script knows what to track and the reference histogram is computed, you should see the object being tracked across the screen in subsequent frames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import cv2\n",
    "\n",
    "# initialize the current frame of the video, along with the list of\n",
    "# ROI points along with whether or not this is input mode\n",
    "\n",
    "frame = None  # current frame of the video that we are processing.\n",
    "roiPts = []  # points corresponding to the Region of Interest (ROI) in our video.\n",
    "inputMode = False  # indicates whether or not we are currently selecting the object we want to track in the video.\n",
    "\n",
    "def selectROI(event, x, y, flags, param):\n",
    "    # grab the reference to the current frame, list of ROI points and whether or not it is ROI selection mode\n",
    "    global frame, roiPts, inputMode\n",
    "\n",
    "    # if we are in ROI selection mode, the mouse was clicked,\n",
    "    # and we do not already have four points, then update the list of ROI points with the (x, y) location of the click\n",
    "    # and draw the circle\n",
    "    if inputMode and event == cv2.EVENT_LBUTTONDOWN and len(roiPts) < 4:\n",
    "        roiPts.append((x, y))\n",
    "        cv2.circle(frame, (x, y), 4, (0, 255, 0), 2)\n",
    "        cv2.imshow(\"frame\", frame)\n",
    "\n",
    "def main():\n",
    "    # grab the reference to the current frame, list of ROI points and whether or not it is ROI selection mode\n",
    "    global frame, roiPts, inputMode\n",
    "\n",
    "    # Supplying our input video\n",
    "    # If you want you can use VideoCapture(0) to use the webcam\n",
    "    camera = cv2.VideoCapture('videos/slow_traffic.mp4')\n",
    "\n",
    "    # setup the mouse callback\n",
    "    cv2.namedWindow(\"frame\")\n",
    "    cv2.setMouseCallback(\"frame\", selectROI)\n",
    "\n",
    "    # initialize the termination criteria for cam shift, \n",
    "    # indicating a maximum of ten iterations or movement by a least one pixel along with the bounding box of the ROI\n",
    "    termination = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 1)\n",
    "    roiBox = None\n",
    "\n",
    "    # keep looping over the frames\n",
    "    while True:\n",
    "        # grab the current frame\n",
    "        (grabbed, frame) = camera.read()\n",
    "\n",
    "        # check to see if we have reached the end of the\n",
    "        # video\n",
    "        if not grabbed:\n",
    "            break\n",
    "\n",
    "        # if the see if the ROI has been computed\n",
    "        if roiBox is not None:\n",
    "            # convert the current frame to the HSV color space and perform mean shift\n",
    "            hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)\n",
    "            backProj = cv2.calcBackProject([hsv], [0], roiHist, [0, 180], 1)\n",
    "\n",
    "            # apply cam shift to the back projection, convert the points to a bounding box, and then draw them\n",
    "            (r, roiBox) = cv2.CamShift(backProj, roiBox, termination)\n",
    "            pts = np.int0(cv2.boxPoints(r))\n",
    "            cv2.polylines(frame, [pts], True, (0, 255, 0), 2)\n",
    "\n",
    "        # show the frame and record if the user presses a key\n",
    "        cv2.imshow(\"frame\", frame)\n",
    "        key = cv2.waitKey(80) & 0xFF\n",
    "\n",
    "        # handle if the 'i' key is pressed, then go into ROI selection mode\n",
    "        if key == ord(\"i\") and len(roiPts) < 4:\n",
    "                        # indicate that we are in input mode and clone the frame\n",
    "            inputMode = True\n",
    "            orig = frame.copy()\n",
    "\n",
    "            # keep looping until 4 reference ROI points have been selected; \n",
    "            # press any key to exit ROI selction mode once 4 points have been selected\n",
    "            while len(roiPts) < 4:\n",
    "                cv2.imshow(\"frame\", frame)\n",
    "                cv2.waitKey(60)\n",
    "\n",
    "            # determine the top-left and bottom-right points\n",
    "            roiPts = np.array(roiPts)\n",
    "            s = roiPts.sum(axis = 1)\n",
    "            tl = roiPts[np.argmin(s)]\n",
    "            br = roiPts[np.argmax(s)]\n",
    "\n",
    "            # grab the ROI for the bounding box and convert it to the HSV color space\n",
    "            roi = orig[tl[1]:br[1], tl[0]:br[0]]\n",
    "            roi = cv2.cvtColor(roi, cv2.COLOR_BGR2HSV)\n",
    "            #roi = cv2.cvtColor(roi, cv2.COLOR_BGR2LAB)\n",
    "\n",
    "            # compute a HSV histogram for the ROI and store the bounding box\n",
    "            roiHist = cv2.calcHist([roi], [0], None, [16], [0, 180])\n",
    "            roiHist = cv2.normalize(roiHist, roiHist, 0, 255, cv2.NORM_MINMAX)\n",
    "            roiBox = (tl[0], tl[1], br[0], br[1])\n",
    "\n",
    "        # if the 'q' key is pressed, stop the loop\n",
    "        elif key == ord(\"q\"):\n",
    "            break\n",
    "\n",
    "    camera.release()\n",
    "    cv2.destroyAllWindows()\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Optical Flow\n",
    "<a href=\"http://en.wikipedia.org/wiki/Optical_flow\">Optical flow</a> is the pattern of apparent motion of image objects between two consecutive frames caused by the movemement of object or camera. It is 2D vector field where each vector is a displacement vector showing the movement of points from first frame to second. \n",
    "\n",
    "Optical flow has many applications in areas like :\n",
    "* Structure from Motion\n",
    "* Video Compression\n",
    "* Video Stabilization ...\n",
    "\n",
    "The Lucas-Kanade method is one of the methods used to solve the problem of finding optical flow in video frames. The gist of the method is that we give some points to track, we receive the optical flow vectors of those points. For large motion videos we have to use pyramids. As we go up in the pyramid, small motions are removed and large motions becomes small motions. So applying Lucas-Kanade there, we get optical flow along with the scale.\n",
    "\n",
    "In OpenCV <a href= \"http://docs.opencv.org/3.0.0/dc/d6b/group__video__track.html#ga473e4b886d0bcc6b65831eb88ed93323\">cv2.calcOpticalFlowPyrLK()</a> is used for this purpose. The following code is used to tracks some points in a video. To decide the points, we use <a href=\"http://docs.opencv.org/3.0.0/dd/d1a/group__imgproc__feature.html#ga1d6bb77486c8f92d79c8793ad995d541\">cv2.goodFeaturesToTrack()</a>. We take the first frame, detect some Shi-Tomasi corner points in it, then we iteratively track those points using Lucas-Kanade optical flow. For the function _cv2.calcOpticalFlowPyrLK()_ we pass the previous frame, previous points and next frame. It returns next points along with some status numbers which has a value of 1 if next point is found, else zero. We iteratively pass these next points as previous points in next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import cv2\n",
    "cap = cv2.VideoCapture('videos/slow_traffic.mp4')\n",
    "\n",
    "# params for ShiTomasi corner detection\n",
    "feature_params = dict( maxCorners = 100,\n",
    "                        qualityLevel = 0.3,\n",
    "                        minDistance = 7,\n",
    "                        blockSize = 7 )\n",
    "\n",
    "# Parameters for lucas kanade optical flow\n",
    "lk_params = dict( winSize = (15,15),\n",
    "                    maxLevel = 2,\n",
    "                    criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))\n",
    "\n",
    "# Create some random colors\n",
    "color = np.random.randint(0,255,(100,3))\n",
    "\n",
    "# Take first frame and find corners in it\n",
    "ret, old_frame = cap.read()\n",
    "old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)\n",
    "p0 = cv2.goodFeaturesToTrack(old_gray, mask = None, **feature_params)\n",
    "\n",
    "# Create a mask image for drawing purposes\n",
    "mask = np.zeros_like(old_frame)\n",
    "\n",
    "while(1):\n",
    "    ret,frame = cap.read()\n",
    "    frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)\n",
    "\n",
    "    # calculate optical flow\n",
    "    p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)\n",
    "\n",
    "    # Select good points\n",
    "    good_new = p1[st==1]\n",
    "    good_old = p0[st==1]\n",
    "\n",
    "    # draw the tracks\n",
    "    for i,(new,old) in enumerate(zip(good_new,good_old)):\n",
    "        a,b = new.ravel()\n",
    "        c,d = old.ravel()\n",
    "        mask = cv2.line(mask, (a,b),(c,d), color[i].tolist(), 2)\n",
    "        frame = cv2.circle(frame,(a,b),5,color[i].tolist(),-1)\n",
    "    img = cv2.add(frame,mask)\n",
    "    \n",
    "    cv2.imshow('frame',img)\n",
    "    k = cv2.waitKey(30) & 0xff\n",
    "    if k == 27:\n",
    "        break\n",
    "\n",
    "    # Now update the previous frame and previous points\n",
    "    old_gray = frame_gray.copy()\n",
    "    p0 = good_new.reshape(-1,1,2)\n",
    "    \n",
    "cv2.destroyAllWindows()\n",
    "cap.release()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A frame from the resulting output video would show the optical flow as follows: \n",
    "\n",
    "<img src=\"captures/opticalflow.png\">\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Dense Optical Flow in OpenCV\n",
    "\n",
    "Lucas-Kanade method computes optical flow for a sparse feature set (in our example, corners detected using Shi-Tomasi algorithm). OpenCV provides another algorithm to find the dense optical flow. It computes the optical flow for all the points in the frame.\n",
    "\n",
    "The following code shows how to find dense optical flow. We get a 2-channel array with optical flow vectors, (u,v). We find their magnitude and direction. We color code the result for better visualization. Direction corresponds to Hue value of the image. Magnitude corresponds to Value plane.\n",
    "\n",
    "Here I used the slow-motion video of landing of a sparrow from youtube.com/user/ultraslo. I say it everytime, feel free to use your own videos. This time too, I won't be any different, except this time, make sure you download the following video and have a look at the amazing result yourself! :P \n",
    "This is the video I downloaded and used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video controls>\n",
       "  <source src=\"videos/sparrow.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(\"\"\"\n",
    "<video controls>\n",
    "  <source src=\"videos/sparrow.mp4\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "cap = cv2.VideoCapture(\"sparrow.mp4\")\n",
    "\n",
    "ret, frame1 = cap.read()\n",
    "prvs = cv2.cvtColor(frame1,cv2.COLOR_BGR2GRAY)\n",
    "hsv = np.zeros_like(frame1)\n",
    "hsv[...,1] = 255\n",
    "\n",
    "while(1):\n",
    "    ret, frame2 = cap.read()\n",
    "    next = cv2.cvtColor(frame2,cv2.COLOR_BGR2GRAY)\n",
    "    \n",
    "    flow = cv2.calcOpticalFlowFarneback(prvs,next, None, 0.5, 3, 15, 3, 5, 1.2, 0)\n",
    "    \n",
    "    mag, ang = cv2.cartToPolar(flow[...,0], flow[...,1])\n",
    "    hsv[...,0] = ang*180/np.pi/2\n",
    "    hsv[...,2] = cv2.normalize(mag,None,0,255,cv2.NORM_MINMAX)\n",
    "    rgb = cv2.cvtColor(hsv,cv2.COLOR_HSV2BGR)\n",
    "    \n",
    "    cv2.imshow('frame2',rgb)\n",
    "    k = cv2.waitKey(30) & 0xff\n",
    "    if k == 27:\n",
    "        break\n",
    "        \n",
    "    elif k == ord('s'):\n",
    "        cv2.imwrite('opticalfb.png',frame2)\n",
    "        cv2.imwrite('opticalhsv.png',rgb)\n",
    "    prvs = next\n",
    "\n",
    "cap.release()\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A frame of the resulting output looked like this:\n",
    "\n",
    "<img src=\"captures/sparrow.png\"> \n",
    "\n",
    "**The dense optical flow of the landing sparrow looks totally amazing! :D**\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Background Subtraction\n",
    "\n",
    "Background subtraction is a major preprocessing steps in many vision based applications. For example, consider the cases like analyzing human activities from a live feed or that of a visitor counter where a static camera takes the number of visitors entering or leaving the room, or a traffic camera extracting information about the vehicles etc. In all these cases, basically you need to extract the moving foreground from static background.\n",
    "\n",
    "Cases where we have the image of background alone, like image of the room without visitors, image of the road without vehicles etc, we simply have to subtract the new image from the background. You get the foreground objects alone. However in most of the cases, you may not have such an image, so we need to extract the background from whatever images we have. \n",
    "\n",
    "### BackgroundSubtractorMOG\n",
    "\n",
    "It is a Gaussian Mixture-based Background/Foreground Segmentation Algorithm. While using this method, we need to create a background object using the function, <a href=\"http://docs.opencv.org/3.0.0/d6/d17/group__cudabgsegm.html#ga6742648a654585013544eaa300f77b57\">cv2.createBackgroundSubtractorMOG()</a>. It has optional parameters like length of history, number of gaussian mixtures, threshold etc. which are all set to their default values. Then inside the video loop, use backgroundsubtractor.apply() method to get the foreground mask.\n",
    "\n",
    "The following example shows the implementation of this algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import cv2\n",
    "\n",
    "#cap = cv2.VideoCapture('slow_traffic.mp4')\n",
    "cap = cv2.VideoCapture('sparrow.mp4')\n",
    "fgbg = cv2.createBackgroundSubtractorMOG2()\n",
    "\n",
    "while(1):\n",
    "    ret, frame = cap.read()\n",
    "\n",
    "    fgmask = fgbg.apply(frame)\n",
    " \n",
    "    cv2.imshow('fgmask',frame)\n",
    "    cv2.imshow('frame',fgmask)\n",
    "\n",
    "    \n",
    "    k = cv2.waitKey(30) & 0xff\n",
    "    if k == 27:\n",
    "        break\n",
    "\n",
    "cap.release()\n",
    "cv2.destroyAllWindows()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "I tested this code on our initial slow_traffic video and the sparrow one.\n",
    "The result was background reduction of each frame in the videos and the comparison between two individual frames looked like the following:\n",
    "\n",
    "<img src=\"captures/br.png\">\n",
    "\n",
    "<img src=\"captures/sparrows.png\">\n",
    "\n",
    "___"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}