FTE_Week5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 5 Munging Continued - Apply, Groupby Operations\n",
    "\n",
    "Week 5 reading: **Pandas for Everyone** chapters 9, 10 (pages 171 - 211)\n",
    "\n",
    "Outline:\n",
    "\n",
    "* Chapter 9 - Apply\n",
    "    1. Terminology\n",
    "    2. Functions vs. methods\n",
    "    3. Classes and objects\n",
    "    4. How `apply()` really works \n",
    "    5. **Very** brief tutorial on  Python *Decorators*\n",
    "    6. Lambda functions\n",
    "* Chapter 10 - Groupby Operations: Split - Apply - Combine\n",
    "    1. Mapping transformations to data\n",
    "\n",
    "    \n",
    "## Overview\n",
    "\n",
    "This week, **Pandas for Everyone** introduces the `apply()` method as a way for user defined functions (UDFs) to operate on rows, columns and dataframes in chapter 9 and grouping operations in chapter 10. From the Experts will look at some theory behind all the magic to prepare students for writing their own functions and classes (chapter 9) and demonstrate how functional programming concepts like `map()` and `reduce()` work (chapter 10)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter 9 - Apply\n",
    "\n",
    "### 1. Terminology\n",
    "\n",
    "First, some terms and definitions to insure we are all talking about the same thing. **Note:** These definitions come from the author's experience rather than any particular textbook or dictionary. Your mileage may vary :).\n",
    "\n",
    "* **Function** - A reusable chunk of code that takes input and returns the output. The `add()` operation is a good example. `add()` is the function name for the + operation. + takes two numbers as input and gives one number as output. \n",
    "* **Method** - A method is a function that is part of a class and specifically designed to interact with the class's data in some way.\n",
    "* **Procedure** - The word 'procedure' in this context describes a function that does **not** return a value. We won't really have any use for procedures but some authors seem to use the words 'function' and 'procedure' interchangeably.\n",
    "* **Class** - A class is a template for creating objects. Consider a **class** to be a blueprint for creating objects of the same type. Generally, we create classes when we want to create a new variable *type* (conceptually).\n",
    "* **Object** - Objects are like the houses built from blueprints. Their structure is identical, although attributes like 'front door type' or 'main paint color' could vary.\n",
    "* **Decorator** - Decorators \"wrap\" functions and modify their behavior.\n",
    "* **Lambda function** - Lambda functions are 'anonymous functions,' that is functions without a name. They consist of only 1 line and are usually used in functions that take *other* functions as parameters, like map, reduce, filter, etc.\n",
    "\n",
    "Don't get too hung up on the terms. We will illustrate them in more detail as we go. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Functions vs. methods\n",
    "\n",
    "As stated earlier, **functions** are a stand-alone package of code that takes input and returns output. The most obvious reason to package code as a function is that we want to reuse the same set of steps later. A less obvious reason is, if the code is collected in one place, it is far easier to debug and modify than if we have it scattered throughout the program and have to rely on our memory to find all the places. \n",
    "\n",
    "**Methods** are just special-purpose functions that belong to classes. We have already used tons of methods. Some examples:\n",
    "* `read_csv()` from DataFrame\n",
    "* `upper()`, 'lower()`, `split()` from string\n",
    "\n",
    "To create a stand-alone function, we use the `def` keyword with the name we want to give it and a list of input variables:\n",
    "\n",
    "`def my_function(input_variable):`\n",
    "\n",
    "Functions will generally have a `return` statement at the bottom that sends data back to the place it was called. \n",
    "\n",
    "Here is a function to square a number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def square_num(num):\n",
    "    return num ** 2\n",
    "\n",
    "square_num(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few finer points to note:\n",
    "* The name \"square_num\" is completely arbitrary, but should be chosen to be descriptive and not conflict with any other functions or types.\n",
    "    * I once had a student name a function \"print\" in his program. This conflicted with Python's `print()` so that every time he wanted to put something to the screen, it called his function and would crash.\n",
    "* The input variable, num, is also named arbitrarily but descriptively. This \"version\" of num only exists while the function is running, so it is OK to have other \"num\" variables in your code, although that might be confusing. For the nerds, this is known as **variable scope**.\n",
    "* The actual math operation is on the same line as the return. This is not necessary, but is done quite often for tiny functions that only have a step or two. It could also have been written as:\n",
    "\n",
    "```\n",
    "\n",
    "def square_num(num):\n",
    "    squared = num ** 2\n",
    "    return squared\n",
    "```\n",
    "\n",
    "Now, let's look at a method belonging to the string class, `upper()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'str'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'THE QUICK BROWN FOX JUMPS OVER THE LAZY DOG.'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_string = 'The quick brown fox jumps over the lazy dog.'\n",
    "print(type(my_string))\n",
    "my_string.upper()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, my_str is an object of class str. In other words, `str` is the *blueprint* and my_str is the *house* built from the blueprint. \n",
    "\n",
    "Since my_string is an object of type str, it has all the methods of any other str, like `upper()`. Also notice that (in this case) `upper()` did not have an input variable. It just *knew* that it should use the data contained in the variable. That is very common with class methods. Input variables are generally used to modify the data in the object. \n",
    "\n",
    "An example of this would be the `replace()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The quick brown fox jumps over the indolent dog.'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_string.replace('lazy','indolent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A final word, here, on return values. In the case of `square_num()`,  `upper()`, and `replace()` above, the result was printed below the cell. This happened because the functions had `return` statements. Usually, you capture that return in another variable using `=`, but in this case it was returned to the notebook itself and printed under the cell. \n",
    "\n",
    "Here is what happens when there is no `return`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cube_num(num):\n",
    "    cube = num ** 3\n",
    "    \n",
    "cube_num(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nothing comes back and the result of our calculation is lost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Classes and objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But what, really, is a class? \n",
    "\n",
    "* __Class:__ A collection of related data and the functions that operate on that data.\n",
    "\n",
    "Let's build a simple **Employee** class. Employee objects will have a first name, last name, department code, and salary for data (attributes), and for methods will have an initializer to take data passed in and a tax calculator.\n",
    "\n",
    "*Don't get too caught up in the minutiae. We are just trying to point out some of the gross mechanics that will be relevant when discussing `apply()` later.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Employee:\n",
    "    def __init__(self, fname='', lname='', dept='', salary=0):\n",
    "        self.fname = fname\n",
    "        self.lname = lname\n",
    "        self.dept = dept\n",
    "        self.salary = salary\n",
    "        \n",
    "    def calc_tax(self, tax_rate):\n",
    "        return self.salary * tax_rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, now we have a blueprint for making employees but no actual employees.\n",
    "\n",
    "To make an employee object, we use the class name:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "emp1 = Employee()\n",
    "print(emp1.fname)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've seen this before, constructing DataFrames, etc. In this case we didn't put anything in the parentheses so we got an empty employee (an employer's nightmare!).\n",
    "\n",
    "We can give our employee object some data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bob Dylan Communication 125000.0\n"
     ]
    }
   ],
   "source": [
    "emp1.fname = 'Bob'\n",
    "emp1.lname = 'Dylan'\n",
    "emp1.dept = 'Communication'\n",
    "emp1.salary = 125000.00\n",
    "\n",
    "print(emp1.fname, emp1.lname, emp1.dept, emp1.salary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Employee()` class has one method associated with it, `calc_tax()`, that **looks** like it takes 2 input parameters -- *self* and `tax_rate`. I emphasize **looks** because *self* is a **hidden parameter**. It is necessary for the way classes work, but when calling the `calc_tax()` method on an employee object, you **do not** have to include it. As a matter of fact, you will get an error if you try.\n",
    "\n",
    "Let's say we want to calculate 10% tax on Bob's salary. We would do it by passing the decimal value for the tax rate into the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12500.0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emp1.calc_tax(.10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But how does the `calc_tax()` method know what data to use? Well, it sounds like I'm being a smart aleck when I say this, but basically it is **magic**. Let me explain:\n",
    "\n",
    "Since `calc_tax()` is a method belonging to the `Employee()` class, we know it **must**, by definition, use data from the specific employee object it belongs to. Well, OK,  *technically* it doesn't have to, but if it doesn't use data from its object, it **really should not be part of the class**. \n",
    "\n",
    "I hear you asking, \"But how does it know to use the employee's salary?\" **That** is the magic. As the author of the `Employee()` class, I **tell you** in the class documentation that the method uses `salary`. You, as the user of the `Employee()` class, must **take it on faith** that I did it correctly.\n",
    "\n",
    "This is how `DataFrame()` and `Series()` objects know what data to operate on. Keep this in mind in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. How `apply()` really works\n",
    "\n",
    "To show how `apply()` works for DataFrames and Series, we will redefine the `Employee()` class to have an `apply()` function instead of the tax calculation, and then we will define a couple of standalone functions to be applied to the salary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Employee_2:\n",
    "    def __init__(self, fname='', lname='', dept='', salary=0):\n",
    "        self.fname = fname\n",
    "        self.lname = lname\n",
    "        self.dept = dept\n",
    "        self.salary = salary\n",
    "        \n",
    "    def apply(self, *args):\n",
    "        func = args[0]\n",
    "        attr = getattr(self, args[1])\n",
    "        if len(args) == 2:\n",
    "            return func(attr)\n",
    "        else:\n",
    "            data = args[2]\n",
    "            return func(attr, data)\n",
    "    \n",
    "def calc_tax(salary, rate):\n",
    "    return salary * rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of our `Employee_2` class, we must tell the `apply()` function what class attribute we are operating on and use a couple of Python tricks to make it work. \n",
    "\n",
    "* `*args` takes all parameters passed in and puts them in a list. NOTE: the * is the important part. The word **args** is used by convention but could be named something else.\n",
    "* `getattr` gets the value from an object's attribute. In this case, the object is itself (you could use `getattr()` with a different object), and we want to get something like *fname*, *lname*, *salary*, etc.\n",
    "* Since our calc_tax function is no longer part of the `Employee_2` object, we must pass in the salary we want to use as a calculation. At first this may seem like a negative, but realize that now the `calc_tax()` function will work for **any** tax calculation, not just the object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "emp2 = Employee_2('Jim', 'Morrison', 'Marketing', 200000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20000.0"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emp2.apply(calc_tax, 'salary', 0.10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Notice we did NOT use parentheses when we passed `calc_tax` to `apply()`. This is because '()' means 'evaluate the function and return the result.' But in this case, we are handing the function off to someone else to run.* \n",
    "\n",
    "Now, we even have the ability to affect other attributes, too!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'JIM'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remember, str is the string class and upper is the uppercase function\n",
    "emp2.apply(str.upper, 'fname') # No '()' on str.upper here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. **Very** brief tutorial on Python *Decorators*\n",
    "\n",
    "Page 185 of **Pandas for Everyone** mentions a 'vectorize decorator.' It should be noted that **decorators** are a general concept. This particular 'vectorize' decorator comes from Anaconda Python's *Numba* library.\n",
    "\n",
    "But what is a decorator? \n",
    "\n",
    "Decorators \"wrap\" functions and modify their behavior.\n",
    "\n",
    "The `apply()` function we looked at as part of `Employee_2` is the first step to \"wrapping\" a function. Let's look at a complete example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decorate(func):\n",
    "    def wrapper():\n",
    "        print('Before function is called')\n",
    "        func()\n",
    "        print('After function is called')\n",
    "    return wrapper\n",
    "\n",
    "def print_slogan():\n",
    "    print('I love programming in Python!')\n",
    "    \n",
    "print_slogan = decorate(print_slogan)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, what is going on there? Let's add some comments.\n",
    "\n",
    "```\n",
    "def decorate(func):    <========== Some function gets passed in here as the variable 'func'\n",
    "###################\n",
    "\n",
    "    def wrapper():    <========== \"Inner\" function. This gets defined everytime decorate runs\n",
    "        print('Before function is called')\n",
    "        \n",
    "        func()    \n",
    "        \n",
    "        print('After function is called')\n",
    "        \n",
    "###################\n",
    "    return wrapper  <========== That \"inner function\" is passed back out, with code before and after func.\n",
    "\n",
    "def print_slogan():\n",
    "    print('I love programming in Python!')\n",
    "    \n",
    "print_slogan = decorate(print_slogan) <========== the print_slogan function is passed in, the new function comes out\n",
    "```\n",
    "\n",
    "Let's see what happens when we run print_slogan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before function is called\n",
      "I love programming in Python!\n",
      "After function is called\n"
     ]
    }
   ],
   "source": [
    "print_slogan()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, functionality was added to `print_slogan()`. The actual **decoration** happened when we called the `decorate()` function.\n",
    "\n",
    "Let's do something a little more exciting. We will change the message based on current time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-12-10 14:45:17.105447\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "print(str(datetime.now()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, it is about 2:45 pm at the writing of this guide. Let's first show `print_slogan()` printing during business hours:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I love programming in Python!\n"
     ]
    }
   ],
   "source": [
    "def decorate(func):\n",
    "    def wrapper():\n",
    "        if 9 <= datetime.now().hour < 17:\n",
    "            func()\n",
    "        else:\n",
    "            print('Too late to shout.')\n",
    "    return wrapper\n",
    "\n",
    "def print_slogan():\n",
    "    print('I love programming in Python!')\n",
    "    \n",
    "print_slogan = decorate(print_slogan)\n",
    "\n",
    "print_slogan()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we had run at night, we would have gotten the \"Too late to shout.\" message.\n",
    "\n",
    "Python has a shortcut for that `print_slogan = decorate(print_slogan)` line. Programmers refer to shortcuts like this as \"syntactic sugar.\" This is where that `@` comes in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I love programming in Python!\n"
     ]
    }
   ],
   "source": [
    "def decorate(func):\n",
    "    def wrapper():\n",
    "        if 9 <= datetime.now().hour < 17:\n",
    "            func()\n",
    "        else:\n",
    "            print('Too late to shout.')\n",
    "    return wrapper\n",
    "\n",
    "@decorate\n",
    "def print_slogan():\n",
    "    print('I love programming in Python!')\n",
    "\n",
    "print_slogan()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decorators are a fascinating and useful tool... well, they are if you are a Computer Scientist, anyway!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Lambda functions\n",
    "\n",
    "Finally, the book introduces lambda functions at the end of Chapter 9. Lambda functions may seem like black magic but they are really very easy once you get the hang of them.\n",
    "\n",
    "Lambda functions are one-liners that don't have names and are usually used in map, reduce, filter, apply, etc. Let's look at an example with `filter()`.\n",
    "\n",
    "`filter()` takes a function and an iterable (you remember what those are, right?) as parameters, then applies the function to every element in the iterable, giving back a True or False. \n",
    "\n",
    "We will define a range of 100 integers in a list then use `filter()` to give us the evens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100]\n"
     ]
    }
   ],
   "source": [
    "numbers = list(range(1,101))\n",
    "print(numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, 70, 72, 74, 76, 78, 80, 82, 84, 86, 88, 90, 92, 94, 96, 98, 100]\n"
     ]
    }
   ],
   "source": [
    "print(list(filter(lambda x: x%2==0, numbers)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`lambda x: x%2==0` is our lambda function. Remember the % operator means **'modulo'** and returns the remainder of a division. In this case we are dividing our numbers by 2 and seeing if that remainder equals 0. Like this: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "4%2==0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And it does that **for each** number in the `numbers` list.\n",
    "\n",
    "The trick to lambda functions is keeping track of what is in your container (*numbers in this case*), and remembering that the function is applied to **every** element in that container.\n",
    "\n",
    "That's all there is to it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter 10 -- Groupby Operations: Split - Apply - Combine\n",
    "\n",
    "Section 10.3 of **Pandas for Everyone** introduces the idea of *transform* functions, via the DataFrame's `transform()` function. We will end this week's **Frome the Experts** by demonstrating another important transformational function -- `map()`.\n",
    "\n",
    "### 1. Mapping transformations to data\n",
    "\n",
    "Much of what we call *Data Science* today was kicked off by a paper from Google researchers Jeff Dean and Sanjay Ghemawat called \"MapReduce: Simplified Data Processing on Large Clusters\" published in 2004 (Ref: https://ai.google/research/pubs/pub62)\n",
    "\n",
    "MapReduce itself is the combination of two **functional programming** transformations: `map()` and `reduce()`. But wait -- what is this **functional programming** that we are calling attention to?\n",
    "\n",
    "Without going into a [Boring Amount of Detail](https://ai.google/research/pubs/pub62), **functional programming** is a way of looking at the data such that each 'function' represents a transformation applied to every element of the data. From this point of view, `map()` literally maps a transformation onto a set of data. \n",
    "\n",
    "In Pandas, the Series object has a `map()` function, and remember that when you isolate a DataFrame column, you get a Series. Here is a familiar example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>emp_id</th>\n",
       "      <th>emp_name</th>\n",
       "      <th>dept_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Tom</td>\n",
       "      <td>HR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Mary</td>\n",
       "      <td>Development</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>John</td>\n",
       "      <td>Marketing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Tim</td>\n",
       "      <td>HR</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  emp_id emp_name      dept_id\n",
       "0      1      Tom           HR\n",
       "1      2     Mary  Development\n",
       "2      3     John    Marketing\n",
       "3      4      Tim           HR"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "emp_df = pd.DataFrame({'emp_id':['1','2','3','4'], 'emp_name':['Tom', 'Mary','John', 'Tim'], 'dept_id':['HR','Development', 'Marketing', 'HR'] })\n",
    "emp_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e_name = emp_df.emp_name\n",
    "type(e_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One *extremely* useful application of `map()` is to change category name strings (like we have above) into integers for use in a machine learning model. Let's try that with our dept_id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "emp_df['dept_id'] = emp_df['dept_id'].map({'HR':1, 'Development':2, 'Marketing':3})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>emp_id</th>\n",
       "      <th>emp_name</th>\n",
       "      <th>dept_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Tom</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Mary</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>John</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Tim</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  emp_id emp_name  dept_id\n",
       "0      1      Tom        1\n",
       "1      2     Mary        2\n",
       "2      3     John        3\n",
       "3      4      Tim        1"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emp_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also notice that by virtue of our dictionary's values being integers, our dept_id column is now int64."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4 entries, 0 to 3\n",
      "Data columns (total 3 columns):\n",
      "emp_id      4 non-null object\n",
      "emp_name    4 non-null object\n",
      "dept_id     4 non-null int64\n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 176.0+ bytes\n"
     ]
    }
   ],
   "source": [
    "emp_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also pass a function in to `map()` -- that **is** the purpose of `map()`, after all. Let's use a **lambda** function to make employee names lower case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>emp_id</th>\n",
       "      <th>emp_name</th>\n",
       "      <th>dept_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>tom</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>mary</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>john</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>tim</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  emp_id emp_name  dept_id\n",
       "0      1      tom        1\n",
       "1      2     mary        2\n",
       "2      3     john        3\n",
       "3      4      tim        1"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emp_df['emp_name'] = emp_df['emp_name'].map(lambda x: x.lower())\n",
    "emp_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember the trick of lambda functions is to keep in mind what each element of data set is as it is going into the x variable. In this case, each element was a string so we could call the `lower()` function from the string class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}