Introduction to Machine Learning - 4.2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the very basic level, Pandas objects can be thought of as enhanced versions of\n",
    "NumPy structured arrays in which the rows and columns are identified with labels\n",
    "rather than simple integer indices. Let’s introduce the three\n",
    "fundamental Pandas data structures: the Series, DataFrame, and Index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Pandas Series Object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Series object is 1D array, similar to a column in a spreadsheet (with a column name and row labels)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2\n",
       "1   -1\n",
       "2    3\n",
       "3    5\n",
       "dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = pd.Series([2, -1, 3, 5])\n",
    "s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access both, values and the sequence of indices in the Series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2, -1,  3,  5], dtype=int64)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=4, step=1)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data can be accessed by the asocciated index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1   -1\n",
       "2    3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s[1:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Series objects behave much like one-dimensional NumPy ndarrays, and you can often pass them as parameters to NumPy functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      7.389056\n",
       "1      0.367879\n",
       "2     20.085537\n",
       "3    148.413159\n",
       "dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    12\n",
       "1     9\n",
       "2    13\n",
       "3    15\n",
       "dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s + 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    12\n",
       "1     1\n",
       "2     6\n",
       "3     6\n",
       "dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s + [10, 2, 3, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     True\n",
       "1     True\n",
       "2     True\n",
       "3    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s < 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Index Labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The essential difference with a NumPy array is the presence\n",
    "of the index: while the NumPy array has an implicitly defined integer index used\n",
    "to access the values, the Pandas Series has an explicitly defined index associated with\n",
    "the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    0.25\n",
       "b    0.50\n",
       "c    0.75\n",
       "d    1.00\n",
       "dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.Series([0.25, 0.5, 0.75, 1.0], \n",
    "                 index=['a', 'b', 'c', 'd'])\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['b']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can still accessing the index by integer location:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.75"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even use noncontiguous or nonsequential indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    0.25\n",
       "5    0.50\n",
       "3    0.75\n",
       "7    1.00\n",
       "dtype: float64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.Series([0.25, 0.5, 0.75, 1.0], \n",
    "                 index=[2, 5, 3, 7])\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - iloc: integer location\n",
    " - loc: label location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Key Error:  0\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    data.loc[0]\n",
    "except KeyError as e:\n",
    "    print('Key Error: ', e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series as specialized dictionary\n",
    "In this way, you can think of a Pandas Series a bit like a specialization of a Python\n",
    "dictionary. A dictionary is a structure that maps arbitrary keys to a set of arbitrary\n",
    "values, and a Series is a structure that maps typed keys to a set of typed values. This\n",
    "typing is important: just as the type-specific compiled code behind a NumPy array\n",
    "makes it more efficient than a Python list for certain operations, the type information\n",
    "of a Pandas Series makes it much more efficient than Python dictionaries for certain\n",
    "operations.\n",
    "\n",
    "We can make the Series-as-dictionary analogy even more clear by constructing a\n",
    "Series object directly from a Python dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "California    38332521\n",
       "Texas         26448193\n",
       "New York      19651127\n",
       "Florida       19552860\n",
       "Illinois      12882135\n",
       "Name: population, dtype: int64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "population_dict = {'California': 38332521,\n",
    "                    'Texas': 26448193,\n",
    "                    'New York': 19651127,\n",
    "                    'Florida': 19552860,\n",
    "                    'Illinois': 12882135}\n",
    "population = pd.Series(population_dict, name='population')\n",
    "population"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Texas       26448193\n",
       "New York    19651127\n",
       "Florida     19552860\n",
       "Name: population, dtype: int64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "population['Texas': 'Florida']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that in this case, the Series is populated only with the explicitly identified\n",
    "keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    c\n",
       "2    a\n",
       "dtype: object"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series({2:'a', 1:'b', 3:'c'}, index=[3, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Automatic Alignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "alice       68\n",
       "bob         83\n",
       "charles    112\n",
       "darwin      68\n",
       "dtype: int64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s1 = pd.Series([68, 83, 112, 68], index=[\"alice\", \"bob\", \"charles\", \"darwin\"])\n",
    "s1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "alice     68\n",
       "bob       83\n",
       "colin     86\n",
       "darwin    68\n",
       "dtype: int64"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights = {\"alice\": 68, \"bob\": 83, \"colin\": 86, \"darwin\": 68}\n",
    "s2 = pd.Series(weights)\n",
    "s2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When an operation involves multiple Series objects, pandas automatically aligns items by matching index labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "alice      136.0\n",
       "bob        166.0\n",
       "charles      NaN\n",
       "colin        NaN\n",
       "darwin     136.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s1 + s2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "The resulting Series contains the union of index labels from s2 and s3. Since \"colin\" is missing from s2 and \"charles\" is missing from s3, these items have a NaN result value. (ie. Not-a-Number means missing)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plotting a Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VfX9x/HX594sEggQCGEECHvPhACuKm5FsA5wayui1m1tq12OtlardbS2VXAUxAGuOuoedbISQGSDJIEwQiCMQMj+/v7I1R+lKDeQ5Nzxfj4eeeSOk3Pe9wG8OTnne77HnHOIiEhk8XkdQEREGp7KXUQkAqncRUQikMpdRCQCqdxFRCKQyl1EJAKp3EVEIpDKXUQkAqncRUQiUIxXG27btq3LyMjwavMiImEpNzd3q3Mu9WDLeVbuGRkZ5OTkeLV5EZGwZGYFwSynwzIiIhFI5S4iEoFU7iIiEUjlLiISgVTuIiIRSOUuIhKBVO4iIhHIs3Hu0vh2lVeRm7+dlUWl9ElrQWZGa5ITYr2OJSJNQOUeQbburmB+Xglz80qYn1/C8k27qN3nFrlm0K99MtndUsjulsKIjBRSW8R7F1hEGo3KPYwVbi9jXl5J3Vd+CWuL9wCQEOtjeJfWXH98L7IzUujbIZkVm3cxP2878/K3MXP+ev75RT4A3dsmfVv02d1SSG/dDDPz8FOJSEMw59zBl2oEWVlZTtMPBM85x9fFu+v2ygOFvnFnOQDJCTGMyEhhRGCPfGDHlsTFfPfplKqaWpZs2Mm8wB7+vLwSdpVXA9ChZcK3e/bZGSn0bNdcZS8SQsws1zmXddDlVO6hqbqmluWbSpmXX8K8vG3Mz99OyZ5KAFJbxH9bvtndUuiT1gKf79ALuLbWsWpLKfO+OaSTV8KW0goAUpLiyOramuxuKYzs1oZ+HVoQ49d5eBGvqNzDTHlVDYsLdzI/v65gFxRsZ3dF3d50l5TE/yrzrm0SG3Vv2jlHwbaywH8sdXv3BdvKAEiK85OZkUJ2Rmuyu7VhcHpLEmL9jZZFRP6byj3E7a6oZkHB9m+Ply9av4PK6loA+qS1YES3uvLMzkihfcsEj9NC0a7y/z++n1fCyqJSAOJifAxNb1V33L5bCpldW9M8XqdyRBqLyj3ElFfV8PGq4rrj5fklLN24i5pah99nDOzU8ts94ayurWmdFOd13IPaUVbJ/Pzt3/6msWTDTmpqHT6DAR1bfnuStnNKM4ymP2bfKjGWjq2aNfl2RRqbyj3EXDE9h/eWFREf42No51aMDOzpDu/SmqQI2NPdU1HNwnU7mJe3jXn5JSxct4OKwG8iXvn9mQO5aFRXTzOINLRgyz38WyUM5G/dw3vLirji6G7ccnIf4mMi7xh1UnwMR/Vqy1G92gJQUV3Dkg07KS6t9CTPzPnr+PW/llBRXcvlR3XzJIOIl1TuTeCZuQXE+Iwrju4ekcV+IPExfjK7pni2/TF923HD8wv53RvLKK+q4ZrjenqWRcQLGtPWyPZW1jArp5CTB7anXbL3J0ajRVyMj7+eP4wzh3bkvndWcv87K/HqEKSIF4IqdzNrZWYvmtkKM1tuZqP3e9/M7C9mtsbMFpvZ8MaJG35eX7yRnXuruFjHfptcjN/HnycM5bwRnXnkozX8/t/LVfASNYI9LPMw8LZz7hwziwMS93v/VKBX4Gsk8I/A96g3Y04BvdOaM7Kbd4coopnfZ9z9w0EkxPp54rM8yqtq+N34gYd10ZdIODhouZtZMnAMcBmAc64S2P8s2XhguqvbLZoT2NPv4Jzb1MB5w8qi9TtYXLiT340foEv4PeTzGbef0Z/4WB+PfbyW8qpa/nTOYPwqeIlgwey5dweKgafMbAiQC9zgnNuzzzKdgPX7PC8MvBbV5f707AKS4vycOayT11Ginplx6yl9aRbr56H3V1NRXcODE4cSq6kUJEIF8zc7BhgO/MM5NwzYA9y63zIH2gX6n4ObZjbZzHLMLKe4uLjeYcNJyZ5KXl+8kbOGp9NCc6iHBDPjxhN6c+upfXlj8SZ+8swCKqprvI4l0iiCKfdCoNA5Nzfw/EXqyn7/ZTrv8zwd2Lj/ipxzU5xzWc65rNTU1EPJGzZeyFlPZXUtF4/WidRQc9UPenDnuAF11x5Mz2VvpQpeIs9By905txlYb2Z9Ai8dDyzbb7HXgEsCo2ZGATuj+Xh7Ta1jxtwCRnZLoXdaC6/jyAFcekQG9549iE9XF/Ojf85jT2CSNpFIEewBx+uAZ8xsMTAUuNvMrjKzqwLvvwmsBdYAU4GfNHjSMPLJqmLWl+zVXnuImziiCw9NHMr8/O1c/MRcdpVXeR1JpMEENRTSObcI2H8ug0f3ed8B1zRgrrD29JwCUlvEc1L/9l5HkYMYP7QTcX4f1z+/kAunzmX6j7PDYuI2kYPRUIEGtr6kjI9WbuH87C7fezckCR2nDurAYxdnsrKolPOnzqE4cKMSkXCm9mlgM+YW4DPjguwuXkeRehjTN42nLhtBwbYyJk6ZzebALQxFwpXKvQGVV9Uwa/56TuqfFhI32JD6ObJnW6b9OJstuyqY8Nhs1peUeR1J5JCp3BvQvxdvYnuZ5pEJZ9ndUpgxaSQ7yiqZ+Nhs8rbuOfgPiYQglXsDmj6ngB6pSYzu0cbrKHIYhnZuxXOTR1FeXcuEx2azKnBLQZFwonJvIIsLd/Dl+h1cPKqr5pGJAAM6tmTm5FEYcN6UOSzZsNPrSCL1onJvIDPmFJAY5+eszHSvo0gD6ZXWgllXjqZZrJ8Lps5h4brtXkcSCZrKvQHsKKvk1UUbOXNYJ5I1j0xEyWibxMwrR9EqMY6LHp/L3LXbvI4kEhSVewN4MbeQiupanUiNUOmtE5l15Wjat0zg0qfm8dnqrV5HEjkolfthqq11PD2ngBEZrenXIdnrONJI2rdMYOaVo8lok8SPp83ng+VFXkcS+V4q98P06ZqtFGwr4yLttUe8ts3jeX7yKPq2b8GVT+fy5ldROzeehAGV+2F6enYBbZvHcerADl5HkSbQKjGOGZNGMqRzK659dgGvLCz0OpLIAancD0Ph9jI+XFHEeSM0j0w0SU6IZfqPsxnZrQ03z/qS5+et8zqSyP9QIx2GZ+fW/aM+f6TmkYk2SfExPPWjERzTK5VbX/6Kf36e53Ukkf+icj9EFdU1zJy/nhP6pdGpVTOv44gHEmL9TLkkk5MHpHHH68t49OOvvY4k8i2V+yF666vNbNtTqRtyRLn4GD+PXDCccUM6cs9bK3jwvVXU3d5AxFtB3axD/tf02fl0b5vEkT3aeh1FPBbr9/HgxKHEx/h4+IPV+My44YReXseSKKdyPwRLNuxkwbod/GZsf3w+zSMj4PcZ9549mFoHD76/ih7tkhg7uKPXsSSK6bDMIXhmbgEJsT7OGa55ZOT/+XzG3WcNZERGa34660sWF+7wOpJEMZV7Pe3cW8W/Fm7kzKGdaJmoeWTkv8XH+Hn0okxSW8RzxfQc3dFJPKNyr6eXcgvZW1WjK1LlO7VpHs/jl2axu7yayU/nsLeyxutIEoVU7vVQW+uYMaeA4V1aMbBTS6/jSAjr2z6Zh88bxlcbdvKzF7/UCBppcir3evji622s3bpHwx8lKCf0T+MXp/TljcWb+MsHa7yOI1FGo2Xq4ek5+aQkxXHaIM0jI8G58pjurCoq5cH3V9Errbn+7kiT0Z57kDbu2Mt7y4qYOKIz8TF+r+NImDAz/njWIDK7tubmWYt0uz5pMir3ID03bx0OuCBb88hI/cTH+Hns4kzaJMUzaVoOW3ZpBI00PpV7ECqra3lu3nqO79uOzimJXseRMNQ2MIJmV3kVVzydS3mVRtBI41K5B+HtpZvZurtCwx/lsPTrkMxDE4eyuHAHP39xsUbQSKMKqtzNLN/MvjKzRWaWc4D3jzWznYH3F5nZbxs+qneenp1P1zaJHNMr1esoEuZOGtCeW07qw2tfbuRvH2kEjTSe+oyWOc459313Bv7UOTf2cAOFmuWbdjE/fzu/Oq2f5pGRBvGTY3uwZstu7n93FT3bNecU3cVLGoEOyxzEjDkFxMf4ODdL88hIw/hmBM2wLq24aeaXLN2oETTS8IItdwe8a2a5Zjb5O5YZbWZfmtlbZjbgQAuY2WQzyzGznOLi4kMK3JR2lVfxysINjBvSkVaJcV7HkQiSEFs3gqZ1YixXTMthS6lG0EjDCrbcj3TODQdOBa4xs2P2e38B0NU5NwT4K/CvA63EOTfFOZflnMtKTQ3949evLNhAWWWNrkiVRtGuRQJTL81ie1kVV2oEjTSwoMrdObcx8H0L8AqQvd/7u5xzuwOP3wRizSys72LhnOPpOQUM6dyKwemtvI4jEWpAx5Y8OHEIC9ft4LaXv9IIGmkwBy13M0sysxbfPAZOApbst0x7M7PA4+zAerc1fNymM3vtNtZs2c3FGv4ojeyUgR245aTevLJwA//QfVilgQQzWiYNeCXQ3THAs865t83sKgDn3KPAOcDVZlYN7AXOc2G+CzJjTgGtEmMZO1gjGaTxXXNcT1YV7ea+d1bSM7U5Jw1o73UkCXMHLXfn3FpgyAFef3Sfx48AjzRsNO9s3lnOO0uLmHRUNxJiNY+MND4z40/nDKZg2x5unLmIF686gv4dk72OJWFMQyEP4Ll566h1jgtH6pCMNJ2EWD9TL8kiOSGWK6bnUFxa4XUkCWMq9/1U1dTy3Lx1HNs7lS5tNI+MNK12yQk8fmkW2/ZUcNWMXCqqNYJGDo3KfT/vLi1iS2mFhj+KZwZ2askDE4aSW7CdX768RCNo5JCo3PczfXY+nVOa8YPe7byOIlHstEEduOmE3ry0oJApn6z1Oo6EIZX7PlYVlTI3r4QLR3bFr3lkxGPXH9+TsYM7cM/bK3h/WZHXcSTMqNz3MWNOAXExPiZkdfY6ighmxn3nDGFQp5bc8PxCVmze5XUkCSMq94DdFdW8vGADYwd3ICVJ88hIaGgW52fKxVkkxccwaVoO23ZrBI0ER+Ue8MrCDeyuqNYVqRJy2rdMYOolWRSXagSNBE/lTmAemdn5DOrUkqGdNY+MhJ4hnVtx/7lDmJ+/nV+/ohE0cnAqd2BeXgmriurmkQlMsyAScs4Y0pHrj+/FC7mFPPFZntdxJMTV505MEWv6nAJaNovljCEdvY4i8r1uPL4Xa7aUcveby+mR2pzj+mrIrhxY1O+5b9lVzjtLNnNuZjrN4jSPjIQ2n8/487lD6d8xmeueW8iqolKvI0mIivpyf37+eqprHRfqRKqEiWZxdXPQNIvzc/m0+ZTsqfQ6koSgqC736ppanp27jmN6p9KtbZLXcUSC1qFlM6ZcnEnRrroRNJXVtV5HkhAT1eX+/vIiNu8q1/BHCUvDurTmvnMGMy+vhN++qhE08t+i+oTq9NkFdGrVjDE6KSVhavzQTqwu2s0jH62hV1oLLj+qm9eRJERE7Z77mi2lfPH1Ni4Y2UXzyEhYu/nE3pwyoD1/+Pcy/rNyi9dxJEREbbnPmLOOOL+PiSM0j4yEN5/PeGDiEPq0T+a6ZxeyZotG0EiUlvueimpeyi3ktEHtads83us4IoctMS6Gxy/NIj7Wz+XTctiuETRRLyrL/dVFGymtqNYNOSSidGrVjMcuzmTTjnKufiaXqhqNoIlmUVfuzjmmz86nf4dkhndp7XUckQaV2bU1954ziDlrS7j9taUaQRPFoq7ccwu2s2JzKReP1jwyEpl+OCydq4/twbNz1zHti3yv44hHoq7cp88uoEVCDOOHah4ZiVw/O6kPJ/ZP4643lvHJqmKv44gHoqrcN+zYy1tLNnFOZjqJcVE9xF8inM9nPDRxKL3TWnDNswv4uni315GkiUVVud/97+X4fcako7t7HUWk0SXF142gifP7mDQthx1lGkETTaKm3D9bvZV/f7WJa47tSadWzbyOI9Ik0lsn8tjFmWzYvpdrnl2gETRRJCrKvbK6lttfW0KXlESuOEZ77RJdsjJSuPusQXy+Zht3vb7M6zjSRII68Gxm+UApUANUO+ey9nvfgIeB04Ay4DLn3IKGjXropn2Rz9fFe3ji0iwSYjVnu0SfczLTWV1UymOfrKV3WnMuHp3hdSRpZPU5q3icc27rd7x3KtAr8DUS+Efgu+e27CrnofdXMaZvO47vl+Z1HBHP/PyUvqzZsps7Xl9G99TmHNmzrdeRpBE11GGZ8cB0V2cO0MrMOjTQug/LH99aQVWN47dj+3sdRcRTfp/x8PnD6JnanJ88s4C8rXu8jiSNKNhyd8C7ZpZrZpMP8H4nYP0+zwsDr3lqXl4JryzcwORjupOhm3GI0DwwgsbvMy6fNp+de6u8jiSNJNhyP9I5N5y6wy/XmNkx+71/oEs9/+e6ZzObbGY5ZpZTXNy4F1ZU19Ty21eX0LFlAj85rkejbksknHROSeTRizJZX1LGtc8uoFojaCJSUOXunNsY+L4FeAXI3m+RQmDfuXPTgY0HWM8U51yWcy4rNTX10BIH6dl561ixuZRfj+2vC5ZE9pPdLYU/nDmIT1dv5ff/Xu51HGkEBy13M0sysxbfPAZOApbst9hrwCVWZxSw0zm3qcHTBmnb7gruf2clR/Vsy6kD23sVQySkTRjRmUlHdeOfX+TzzNwCr+NIAwtmlzYNeCUwyVYM8Kxz7m0zuwrAOfco8CZ1wyDXUDcU8keNEzc4972zkrLKGu4Y11+Tg4l8j9tO68fXxbu5/dWldGubxBE9NIImUphXU4JmZWW5nJycBl/vovU7+OHfP+eKo7vzy9P6Nfj6RSJNaXkVZ/39C4p3V/CvnxypwQchzsxy97/W6EAi6grV2lrH7a8uIbV5PNeN6el1HJGw0CIhlicuHYEBk6bnsKtcI2giQUSV+wu56/mycCe/PK0fLRJivY4jEja6tEnk7xdmkr91D9c9u1AjaCJAxJT7zrIq7n17JSMyWmuudpFDMLpHG3535kA+XlXM3W+u8DqOHKaIGSP4wHsr2VFWyZ3jRuokqsghOj+7C6uKSnny8zx6pzXnvOwuXkeSQxQRe+7LNu7i6TkFXDyqK/07JnsdRySs/eq0fhzTO5Vf/2sJc9Zu8zqOHKKwL3fnHLe/toRWiXHcfGIfr+OIhL0Yv49HLhhG1zaJXD0jl3XbyryOJIcg7Mv91UUbmZ+/nV+c0oeWiTqJKtIQkgMjaGodXD5tPqUaQRN2wrrcS8ur+MObyxmS3pJzMzsf/AdEJGgZbZP4x4XDydu6h+ufW0hNrTfXxMihCety/+uHa9i6u4K7xg/E59NJVJGGdkTPttwxbgAfrSzmnrc0B004CdvRMmu2lPLkZ3lMzOrMkM6tvI4jErEuGtWV1UWlTP00j15pLZiQpd+Sw0FY7rk757jjtWUkxvn52ck6iSrS2H4ztj9H92rLr175inl5JV7HkSCEZbm/vWQzn63Zyi0n96FN83iv44hEvBi/j0fOH07n1olcNSOX9SUaQRPqwq7c91bW8Ls3ltGvQzIX6AILkSbTMjGWxy/NorqmlknTcthdUe11JPkeYVfuf/toDRt3lnPX+AHE+MMuvkhY657anL9fmMma4t3c+LxG0ISysGrH/K17mPLJWn44rBMjMlK8jiMSlY7q1Zbbz+jP+8u38Kd3NAdNqAqr0TJ3vbGMWL9x26l9vY4iEtUuGZ3BqqJSHvt4Lb3ateCczHSvI8l+wmbP/YPlRXy4Ygs3ntCbdskJXscRiXq3nzGAI3u24Zcvf8WCddu9jiP7CYtyL6+q4c7Xl9GzXXMuOzLD6zgiAsT6ffztguG0S47nxucXsUcnWENKWJT71E/Wsq6kjDvOGECsTqKKhIxWiXE8MGEo67eX8ft/6wrWUBLyTVm4vYy//WcNpw1qz1G9dPNekVCT3S2Fycd057l56/hgeZHXcSQg5Mv9D4G9gV+d3t/jJCLyXW4+sTd927fgFy99xbbdFV7HEUK83D9dXcxbSzZz7XE96dSqmddxROQ7xMf4eei8oezaW8UvX/kK5zT+3WshW+6V1bXc8dpSurZJZNLR3b2OIyIH0bd9Mj89qTfvLC3ipQUbvI4T9UK23P/5RR5fF+/hjjMGkBDr9zqOiARh0tHdye6Wwh2vLdX8Mx4LyXIv2lXOw++v5oR+7Tiubzuv44hIkPw+48/nDgHglhe+pFbTE3gmJMv9j28up6rW8ZuxOokqEm46pyTy2zP6MzevhCc+y/M6TtQKuXKfu3Yb/1q0kauO6U7XNklexxGRQ3BuZjon9U/jvndWsnJzqddxolJIlXt1TS23v7aUTq2acfWxPb2OIyKHyMz441mDSG4Ww40zF1FRXeN1pKgTdLmbmd/MFprZGwd47zIzKzazRYGvSYcS5pm561ixuZTfjO1HszidRBUJZ22ax3PPWYNZvmkXD72/2us4Uac+e+43AN93ffFM59zQwNfj9Q2ydXcFf353JUf3asvJA9rX98dFJASd0D+N80Z05tGPv2Z+vm7P15SCKnczSwdOB+pd2sG67+2VlFXWcPsZAzCzxtqMiDSxX4/tT3rrZtw8a5Hu3tSEgt1zfwj4OVD7PcucbWaLzexFM6vX7dEXrd/BzJz1XH5UN3q2a16fHxWRENc8PoYHJwxlw/a9/P6NZV7HiRoHLXczGwtscc7lfs9irwMZzrnBwPvAtO9Y12QzyzGznOLiYgBqax2/fXUJ7VrEc93xver/CUQk5GVlpHDlD3rw/Pz1vLdMk4s1hWD23I8ExplZPvA8MMbMZuy7gHNum3Pum9mCpgKZB1qRc26Kcy7LOZeVmpoKwKyc9Swu3MmvTu9H8/iwujGUiNTDTSf0pl+HZG57eTFbNblYoztouTvnbnPOpTvnMoDzgA+dcxftu4yZddjn6Ti+/8Trt3aUVXLv2yvIzkhh3JCO9YgtIuEmLsbHQxOHsmtvNbe9rMnFGtshj3M3s7vMbFzg6fVmttTMvgSuBy4LZh0PvLeKnXuruHO8TqKKRIM+7Vvws5P78N6yIl7ILfQ6TkQzr/73HDBkmNt72u+5ZHQGd4wb4EkGEWl6tbWO86fOYenGXbx1w9F0Tkn0OlJYMbNc51zWwZbz7ArVjTv20joxjptO7O1VBBHxgM9n/HlC3eRiP531JTWaXKxReFbuZZU1/OKUvrRsFutVBBHxSHrrRO4YN4B5+SU8/ular+NEJM/KPSkuhnMy073avIh47OzhnThlQHv+/O4qlm/a5XWciONZuXdPTcLn00lUkWhlZtx91iCSm8VykyYXa3AhNSukiESXlKQ4/nTOIFZsLuWB91Z5HSeiqNxFxFNj+qZxfnYXpnyylrlrt3kdJ2Ko3EXEc78+vR9dUhL56QtfUlpe5XWciKByFxHPJcXH8MCEIWzcsZffaXKxBqFyF5GQkNk1hauP7cGsnELeXbrZ6zhhT+UuIiHjhuN7M6BjMre9/BXFpZpc7HCo3EUkZMTF+Hhw4lBKK6q57eXFmlzsMKjcRSSk9E5rwc9P7sP7y7cwK2e913HClspdRELOj4/sxujubbjr9WWs21bmdZywpHIXkZDj8xn3TxiCz4ybZy3S5GKHQOUuIiGpU6tm3Dl+ADkF25nyiSYXqy+Vu4iErB8O68SpA9vzwHsrWbpxp9dxworKXURClpnxhx8OolViHDfP/JLyKk0uFiyVu4iEtLrJxQazskiTi9WHyl1EQt5xfdpx4cguTP10LXM0uVhQVO4iEhZ+dXo/uqYk8tNZX7JLk4sdlMpdRMJCYlwMD0wcyqade7nzNU0udjAqdxEJG8O7tOaa43ry0oJC3l6iycW+j8pdRMLK9cf3YmCnZH75yldsKS33Ok7IUrmLSFiJ9ft4cMJQdldUc/WMBSzZoPHvB6JyF5Gw0yutBfeePYhVm0sZ+9fP+NFT88gtKPE6Vkgxr6bUzMrKcjk5OZ5sW0Qiw869VUz/Ip8nP89je1kVo7u34doxPTmiRxvMzOt4jcLMcp1zWQddTuUuIuFuT0U1z81bx5RP1rKltIKhnVtx7XE9Ob5fu4greZW7iESd8qoaXswt5NGPv6Zw+176tm/BNcf15LRBHfD7IqPkgy33oI+5m5nfzBaa2RsHeC/ezGaa2Rozm2tmGfWLKyJy+BJi/Vw0qisf3XIsfz53CFU1tVz33EJOfOBjXshZT1VNrdcRm0x9TqjeACz/jvcuB7Y753oCDwL3Hm4wEZFDFev3cXZmOu/e9AP+dsFw4mP9/OzFxRx73394enZ+VExAFlS5m1k6cDrw+HcsMh6YFnj8InC8RdqBLhEJO36fcfrgDrx5/VE8eVkWacnx/ObVpRz9p4+Y+sla9lRUex2x0QS75/4Q8HPgu36n6QSsB3DOVQM7gTb7L2Rmk80sx8xyiouLDyGuiEj9mRlj+qbx0tVH8OwVI+nVrjl/eHM5R977IX/5YDU790beXDUHLXczGwtscc7lft9iB3jtf87UOuemOOeynHNZqamp9YgpInL4zIwjerTl2StG8fJPjiCzS2seeG8VR97zIfe+vYKtuyu8jthggtlzPxIYZ2b5wPPAGDObsd8yhUBnADOLAVoCuqJARELW8C6teeKyEbx5/dH8oE8qj378NUfd+yF3vr6UTTv3eh3vsNVrKKSZHQvc4pwbu9/r1wCDnHNXmdl5wFnOuQnfty4NhRSRUPJ18W7+8Z+v+dfCDZjBOZnpXPWDHnRtk+R1tP/S4EMhD7CBu8xsXODpE0AbM1sD3AzceqjrFRHxQo/U5tx/7hA+uuVYJo7ozEsLNnDc/f/hxucXsrqo1Ot49aaLmEREDmDLrnKmfrqWZ+auo6yyhlMGtOfaMT0Z2Kmlp7l0haqISAPYvqeSpz7P46kv8iktr+YHvVO5dkxPRmSkeJKn0Q/LiIhEg9ZJcdx8Uh8+v3UMPzu5D0s27OTcR2fzwLsr8WrnOBgqdxGRICQnxHLNcT357BdjmJCVzl8+XMPdby4P2YKP8TqAiEg4aRbn556zBtMs1s/UT/Mor6rlznED8IXYxGQqdxGRevL5jDvGDSAh1s9jn6ylvKqGe84eHFIzT6rcRUQOgZlx66l9SYj18/AHqymvruUNh+wiAAAHL0lEQVSBCUOI9YfG0W6Vu4jIITIzbjqxNwmxfu59ewWV1TX85fxhxMf4vY6mE6oiIofr6mN7cMcZ/XlnaRFXPp0bElMKq9xFRBrAZUd2449nDeLjVcX86Kn5nk8nrHIXEWkg52d34YEJQ5ibt41Ln5zHrnLvphJWuYuINKAfDkvnkQuGs2j9Di56fC47yio9yaFyFxFpYKcN6sCUSzJZsbmU86bM8WSeeJW7iEgjGNM3jScvHUH+tj1MfGw2RbvKm3T7KncRkUZyVK+2TP/xSDbvLGfCY7Mp3F7WZNtWuYuINKLsbinMmDSS7XsqmfDobPK37mmS7arcRUQa2bAurXlu8ijKq2uZ8NjsJrn5h8pdRKQJDOjYkucnj8IBE6fMYenGnY26PZW7iEgT6Z3WgllXjiYhxsf5U+awaP2ORtuWyl1EpAl1a5vEzCtH0zIxlosen8v8/JJG2Y7KXUSkiXVOSeSFK4+gXXI8lzwxj8/XbG3wbajcRUQ80L5lAjMnj6Zrm0R+9M/5fLRiS4OuX+UuIuKR1BbxPHfFKHqnNWfy0zm8vWRTg61b5S4i4qHWSXE8M2kUgzq15JpnF/Lqog0Nsl6Vu4iIx1o2i+Xpy0cyIqM1N85cxKz56w97nSp3EZEQkBQfw1OXZXN0r1R+/tJips/OP6z1qdxFREJEszg/Uy/J5MT+afz21aVM+eTrQ16Xyl1EJITEx/j5+4XDGTu4A3e/uYKH31+Nc67e6zlouZtZgpnNM7MvzWypmd15gGUuM7NiM1sU+JpU7yQiIgJArN/Hw+cN45zMdB58fxX3vr2y3gUfE8QyFcAY59xuM4sFPjOzt5xzc/ZbbqZz7tp6bV1ERA7I7zP+dPZgEmJ9PPrx15RX1fDbsf2D/vmDlrur++9id+BpbOCr/r8jiIhIvfh8xu/GDyQ+xs8Tn+VRXlUT9M8Gs+eOmfmBXKAn8Dfn3NwDLHa2mR0DrAJucs4d/lgeEZEoZ2b8+vR+NIv188hHa4L+uaBOqDrnapxzQ4F0INvMBu63yOtAhnNuMPA+MO07Qk42sxwzyykuLg46pIhINDMzbjm5D7ec1Dv4n6nvQXozux3Y45y7/zve9wMlzrmW37eerKwsl5OTU69ti4hEOzPLdc5lHWy5YEbLpJpZq8DjZsAJwIr9lumwz9NxwPL6xRURkYYUzDH3DsC0wB65D5jlnHvDzO4CcpxzrwHXm9k4oBooAS5rrMAiInJw9T4s01B0WEZEpP4a7LCMiIiEH5W7iEgEUrmLiEQglbuISARSuYuIRCDPRsuYWSmw0pONe6ct0PC3OQ9t+szRQZ+56XR1zqUebKGg5pZpJCuDGc4TScwsR5858ukzR4dQ/8w6LCMiEoFU7iIiEcjLcp/i4ba9os8cHfSZo0NIf2bPTqiKiEjj0WEZEZEI5Em5m9kpZrbSzNaY2a1eZGhKZtbZzD4ys+WBm4zf4HWmpmBmfjNbaGZveJ2lKZhZKzN70cxWBP6sR3udqbGZ2U2Bv9NLzOw5M0vwOlNDM7MnzWyLmS3Z57UUM3vPzFYHvrf2MuOBNHm5B6YO/htwKtAfON/Mgr/ra3iqBn7qnOsHjAKuiYLPDHAD0TW3/8PA2865vsAQIvyzm1kn4Hogyzk3EPAD53mbqlH8Ezhlv9duBT5wzvUCPgg8Dyle7LlnA2ucc2udc5XA88B4D3I0GefcJufcgsDjUur+0XfyNlXjMrN04HTgca+zNAUzSwaOAZ4AcM5VOud2eJuqScQAzcwsBkgENnqcp8E55z6h7j4V+xrP/99OdBpwZpOGCoIX5d4J2Pfm2YVEeNHty8wygGHAgW4yHkkeAn4O1HodpIl0B4qBpwKHoh43sySvQzUm59wG4H5gHbAJ2Omce9fbVE0mzTm3Cep23oB2Huf5H16Uux3gtagYsmNmzYGXgBudc7u8ztNYzGwssMU5l+t1liYUAwwH/uGcGwbsIQR/VW9IgePM44FuQEcgycwu8jaVfMOLci8EOu/zPJ0I/FVuf2YWS12xP+Oce9nrPI3sSGCcmeVTd9htjJnN8DZSoysECp1z3/xG9iJ1ZR/JTgDynHPFzrkq4GXgCI8zNZWib+4dHfi+xeM8/8OLcp8P9DKzbmYWR90JmNc8yNFkzMyoOxa73Dn3gNd5Gptz7jbnXLpzLoO6P98PnXMRvUfnnNsMrDezPoGXjgeWeRipKawDRplZYuDv+PFE+EnkfbwGXBp4fCnwqodZDqjJJw5zzlWb2bXAO9SdXX/SObe0qXM0sSOBi4GvzGxR4LVfOufe9DCTNLzrgGcCOy1rgR95nKdROefmmtmLwALqRoQtJMSv2jwUZvYccCzQ1swKgduBe4BZZnY5df/JnetdwgPTFaoiIhFIV6iKiEQglbuISARSuYuIRCCVu4hIBFK5i4hEIJW7iEgEUrmLiEQglbuISAT6P9f91CqkKz0vAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "temperatures = [4.4,5.1,6.1,6.2,6.1,6.1,5.7,5.2,4.7,4.1,3.9,3.5]\n",
    "temp_s = pd.Series(temperatures, name=\"Temperature\")\n",
    "temp_s.plot();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Pandas DataFrame Object\n",
    "\n",
    "A DataFrame object represents a spreadsheet, with cell values, column names and row index labels. You can define expressions to compute columns based on other columns, create pivot-tables, group rows, draw graphs, etc. You can see DataFrames as dictionaries of Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "area_dict = {'California': 423967, 'Texas': 695662, 'New York': 141297,\n",
    "            'Florida': 170312, 'Illinois': 149995}\n",
    "area = pd.Series(area_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>California</th>\n",
       "      <td>38332521</td>\n",
       "      <td>423967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>26448193</td>\n",
       "      <td>695662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>New York</th>\n",
       "      <td>19651127</td>\n",
       "      <td>141297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Florida</th>\n",
       "      <td>19552860</td>\n",
       "      <td>170312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Illinois</th>\n",
       "      <td>12882135</td>\n",
       "      <td>149995</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            population    area\n",
       "California    38332521  423967\n",
       "Texas         26448193  695662\n",
       "New York      19651127  141297\n",
       "Florida       19552860  170312\n",
       "Illinois      12882135  149995"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states = pd.DataFrame({'population': population,\n",
    "                        'area': area})\n",
    "states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['California', 'Texas', 'New York', 'Florida', 'Illinois'], dtype='object')"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['population', 'area'], dtype='object')"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[38332521,   423967],\n",
       "       [26448193,   695662],\n",
       "       [19651127,   141297],\n",
       "       [19552860,   170312],\n",
       "       [12882135,   149995]], dtype=int64)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Another convenient way to create a DataFrame is to pass all the values to the constructor as an ndarray, or a list of lists, and specify the column names and row index labels separately:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birthyear</th>\n",
       "      <th>children</th>\n",
       "      <th>hobby</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>alice</th>\n",
       "      <td>1985</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Biking</td>\n",
       "      <td>68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bob</th>\n",
       "      <td>1984</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Dancing</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>charles</th>\n",
       "      <td>1992</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>112</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         birthyear  children    hobby  weight\n",
       "alice         1985       NaN   Biking      68\n",
       "bob           1984       3.0  Dancing      83\n",
       "charles       1992       0.0      NaN     112"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "values = [\n",
    "            [1985, np.nan, \"Biking\",   68],\n",
    "            [1984, 3,      \"Dancing\",  83],\n",
    "            [1992, 0,      np.nan,    112]\n",
    "         ]\n",
    "d3 = pd.DataFrame(\n",
    "        values,\n",
    "        columns=[\"birthyear\", \"children\", \"hobby\", \"weight\"],\n",
    "        index=[\"alice\", \"bob\", \"charles\"]\n",
    "     )\n",
    "d3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "It is also possible to create a DataFrame with a dictionary (or list) of dictionaries (or list):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birthyear</th>\n",
       "      <th>hobby</th>\n",
       "      <th>weight</th>\n",
       "      <th>children</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>alice</th>\n",
       "      <td>1985</td>\n",
       "      <td>Biking</td>\n",
       "      <td>68</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bob</th>\n",
       "      <td>1984</td>\n",
       "      <td>Dancing</td>\n",
       "      <td>83</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>charles</th>\n",
       "      <td>1992</td>\n",
       "      <td>NaN</td>\n",
       "      <td>112</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         birthyear    hobby  weight  children\n",
       "alice         1985   Biking      68       NaN\n",
       "bob           1984  Dancing      83       3.0\n",
       "charles       1992      NaN     112       0.0"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "people = pd.DataFrame({\n",
    "    \"birthyear\": {\"alice\":1985, \"bob\": 1984, \"charles\": 1992},\n",
    "    \"hobby\": {\"alice\":\"Biking\", \"bob\": \"Dancing\"},\n",
    "    \"weight\": {\"alice\":68, \"bob\": 83, \"charles\": 112},\n",
    "    \"children\": {\"bob\": 3, \"charles\": 0}\n",
    "})\n",
    "people"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Selection in DataFrames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>California</th>\n",
       "      <td>38332521</td>\n",
       "      <td>423967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>26448193</td>\n",
       "      <td>695662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>New York</th>\n",
       "      <td>19651127</td>\n",
       "      <td>141297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Florida</th>\n",
       "      <td>19552860</td>\n",
       "      <td>170312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Illinois</th>\n",
       "      <td>12882135</td>\n",
       "      <td>149995</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            population    area\n",
       "California    38332521  423967\n",
       "Texas         26448193  695662\n",
       "New York      19651127  141297\n",
       "Florida       19552860  170312\n",
       "Illinois      12882135  149995"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "California    38332521\n",
       "Texas         26448193\n",
       "New York      19651127\n",
       "Florida       19552860\n",
       "Illinois      12882135\n",
       "Name: population, dtype: int64"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states['population']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "California    38332521\n",
       "Texas         26448193\n",
       "New York      19651127\n",
       "Florida       19552860\n",
       "Illinois      12882135\n",
       "Name: population, dtype: int64"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states.population"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dictionary-style syntax can also be\n",
    "used to modify the object, in this case to add a new column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>area</th>\n",
       "      <th>density</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>California</th>\n",
       "      <td>38332521</td>\n",
       "      <td>423967</td>\n",
       "      <td>90.413926</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>26448193</td>\n",
       "      <td>695662</td>\n",
       "      <td>38.018740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>New York</th>\n",
       "      <td>19651127</td>\n",
       "      <td>141297</td>\n",
       "      <td>139.076746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Florida</th>\n",
       "      <td>19552860</td>\n",
       "      <td>170312</td>\n",
       "      <td>114.806121</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Illinois</th>\n",
       "      <td>12882135</td>\n",
       "      <td>149995</td>\n",
       "      <td>85.883763</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            population    area     density\n",
       "California    38332521  423967   90.413926\n",
       "Texas         26448193  695662   38.018740\n",
       "New York      19651127  141297  139.076746\n",
       "Florida       19552860  170312  114.806121\n",
       "Illinois      12882135  149995   85.883763"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states['density'] = states['population'] / states['area']\n",
    "states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>California</th>\n",
       "      <th>Texas</th>\n",
       "      <th>New York</th>\n",
       "      <th>Florida</th>\n",
       "      <th>Illinois</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>population</th>\n",
       "      <td>3.833252e+07</td>\n",
       "      <td>2.644819e+07</td>\n",
       "      <td>1.965113e+07</td>\n",
       "      <td>1.955286e+07</td>\n",
       "      <td>1.288214e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>area</th>\n",
       "      <td>4.239670e+05</td>\n",
       "      <td>6.956620e+05</td>\n",
       "      <td>1.412970e+05</td>\n",
       "      <td>1.703120e+05</td>\n",
       "      <td>1.499950e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>density</th>\n",
       "      <td>9.041393e+01</td>\n",
       "      <td>3.801874e+01</td>\n",
       "      <td>1.390767e+02</td>\n",
       "      <td>1.148061e+02</td>\n",
       "      <td>8.588376e+01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              California         Texas      New York       Florida  \\\n",
       "population  3.833252e+07  2.644819e+07  1.965113e+07  1.955286e+07   \n",
       "area        4.239670e+05  6.956620e+05  1.412970e+05  1.703120e+05   \n",
       "density     9.041393e+01  3.801874e+01  1.390767e+02  1.148061e+02   \n",
       "\n",
       "                Illinois  \n",
       "population  1.288214e+07  \n",
       "area        1.499950e+05  \n",
       "density     8.588376e+01  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38332521"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states.loc['California', 'population']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38332521"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states.iloc[0, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use masking (conditional selection):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>area</th>\n",
       "      <th>density</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>New York</th>\n",
       "      <td>19651127</td>\n",
       "      <td>141297</td>\n",
       "      <td>139.076746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Florida</th>\n",
       "      <td>19552860</td>\n",
       "      <td>170312</td>\n",
       "      <td>114.806121</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          population    area     density\n",
       "New York    19651127  141297  139.076746\n",
       "Florida     19552860  170312  114.806121"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states[states.density > 100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>density</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>New York</th>\n",
       "      <td>19651127</td>\n",
       "      <td>139.076746</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Florida</th>\n",
       "      <td>19552860</td>\n",
       "      <td>114.806121</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          population     density\n",
       "New York    19651127  139.076746\n",
       "Florida     19552860  114.806121"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "states.loc[states.density > 100, ['population', 'density']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handling Missing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Detecting Null Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        1\n",
       "1      NaN\n",
       "2    hello\n",
       "3     None\n",
       "dtype: object"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.Series([1, np.nan, 'hello', None])\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2    False\n",
       "3     True\n",
       "dtype: bool"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        1\n",
       "2    hello\n",
       "dtype: object"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[data.notnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dropping Null Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        1\n",
       "2    hello\n",
       "dtype: object"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1  2\n",
       "0  1.0  NaN  2\n",
       "1  2.0  3.0  5\n",
       "2  NaN  4.0  6"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame([[1, np.nan, 2],\n",
    "                    [2, 3, 5],\n",
    "                    [np.nan, 4, 6]])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We cannot drop single values from a DataFrame; we can only drop full rows or full\n",
    "columns. Depending on the application, you might want one or the other, so\n",
    "dropna() gives a number of options for a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1  2\n",
       "1  2.0  3.0  5"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   2\n",
       "0  2\n",
       "1  5\n",
       "2  6"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna(axis='columns')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1  2   3\n",
       "0  1.0  NaN  2 NaN\n",
       "1  2.0  3.0  5 NaN\n",
       "2  NaN  4.0  6 NaN"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[3] = np.nan\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1  2\n",
       "0  1.0  NaN  2\n",
       "1  2.0  3.0  5\n",
       "2  NaN  4.0  6"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna(axis='columns', how='all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1  2   3\n",
       "1  2.0  3.0  5 NaN"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dropna(axis='rows', thresh=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filling null values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    1.0\n",
       "b    NaN\n",
       "c    2.0\n",
       "d    NaN\n",
       "e    3.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.Series([1, np.nan, 2, None, 3], index=list('abcde'))\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    1.0\n",
       "b    0.0\n",
       "c    2.0\n",
       "d    0.0\n",
       "e    3.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    1.0\n",
       "b    1.0\n",
       "c    2.0\n",
       "d    2.0\n",
       "e    3.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.fillna(method='ffill')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a    1.0\n",
       "b    2.0\n",
       "c    2.0\n",
       "d    3.0\n",
       "e    3.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.fillna(method='bfill')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1  2   3\n",
       "0  1.0  NaN  2 NaN\n",
       "1  2.0  3.0  5 NaN\n",
       "2  NaN  4.0  6 NaN"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0    1    2    3\n",
       "0  1.0  1.0  2.0  2.0\n",
       "1  2.0  3.0  5.0  5.0\n",
       "2  NaN  4.0  6.0  6.0"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.fillna(method='ffill', axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aggregation and Grouping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1035, 6)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets = sns.load_dataset('planets')\n",
    "planets.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>method</th>\n",
       "      <th>number</th>\n",
       "      <th>orbital_period</th>\n",
       "      <th>mass</th>\n",
       "      <th>distance</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Radial Velocity</td>\n",
       "      <td>1</td>\n",
       "      <td>269.300</td>\n",
       "      <td>7.10</td>\n",
       "      <td>77.40</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Radial Velocity</td>\n",
       "      <td>1</td>\n",
       "      <td>874.774</td>\n",
       "      <td>2.21</td>\n",
       "      <td>56.95</td>\n",
       "      <td>2008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Radial Velocity</td>\n",
       "      <td>1</td>\n",
       "      <td>763.000</td>\n",
       "      <td>2.60</td>\n",
       "      <td>19.84</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Radial Velocity</td>\n",
       "      <td>1</td>\n",
       "      <td>326.030</td>\n",
       "      <td>19.40</td>\n",
       "      <td>110.62</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Radial Velocity</td>\n",
       "      <td>1</td>\n",
       "      <td>516.220</td>\n",
       "      <td>10.50</td>\n",
       "      <td>119.47</td>\n",
       "      <td>2009</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            method  number  orbital_period   mass  distance  year\n",
       "0  Radial Velocity       1         269.300   7.10     77.40  2006\n",
       "1  Radial Velocity       1         874.774   2.21     56.95  2008\n",
       "2  Radial Velocity       1         763.000   2.60     19.84  2011\n",
       "3  Radial Velocity       1         326.030  19.40    110.62  2007\n",
       "4  Radial Velocity       1         516.220  10.50    119.47  2009"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Aggregation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Planets Mass:\n",
      "    sum: 1353.37638\n",
      "    mean: 2.6381605847953233\n",
      "    std: 3.8186166509616046\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\"\"Planets Mass:\n",
    "    sum: {}\n",
    "    mean: {}\n",
    "    std: {}\n",
    "\"\"\".format(\n",
    "    planets['mass'].sum(),\n",
    "    planets['mass'].mean(),\n",
    "    planets['mass'].std()\n",
    "))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "method            Radial VelocityRadial VelocityRadial VelocityR...\n",
       "number                                                         1848\n",
       "orbital_period                                          1.98689e+06\n",
       "mass                                                        1353.38\n",
       "distance                                                     213368\n",
       "year                                                        2079388\n",
       "dtype: object"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets[['mass', 'distance']].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "number               1.785507\n",
       "orbital_period    2002.917596\n",
       "mass                 2.638161\n",
       "distance           264.069282\n",
       "year              2009.070531\n",
       "dtype: float64"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "number                1.240976\n",
       "orbital_period    26014.728304\n",
       "mass                  3.818617\n",
       "distance            733.116493\n",
       "year                  3.972567\n",
       "dtype: float64"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>number</th>\n",
       "      <th>orbital_period</th>\n",
       "      <th>mass</th>\n",
       "      <th>distance</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1035.000000</td>\n",
       "      <td>992.000000</td>\n",
       "      <td>513.000000</td>\n",
       "      <td>808.000000</td>\n",
       "      <td>1035.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.785507</td>\n",
       "      <td>2002.917596</td>\n",
       "      <td>2.638161</td>\n",
       "      <td>264.069282</td>\n",
       "      <td>2009.070531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.240976</td>\n",
       "      <td>26014.728304</td>\n",
       "      <td>3.818617</td>\n",
       "      <td>733.116493</td>\n",
       "      <td>3.972567</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.090706</td>\n",
       "      <td>0.003600</td>\n",
       "      <td>1.350000</td>\n",
       "      <td>1989.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>5.442540</td>\n",
       "      <td>0.229000</td>\n",
       "      <td>32.560000</td>\n",
       "      <td>2007.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>39.979500</td>\n",
       "      <td>1.260000</td>\n",
       "      <td>55.250000</td>\n",
       "      <td>2010.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>526.005000</td>\n",
       "      <td>3.040000</td>\n",
       "      <td>178.500000</td>\n",
       "      <td>2012.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>7.000000</td>\n",
       "      <td>730000.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>8500.000000</td>\n",
       "      <td>2014.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            number  orbital_period        mass     distance         year\n",
       "count  1035.000000      992.000000  513.000000   808.000000  1035.000000\n",
       "mean      1.785507     2002.917596    2.638161   264.069282  2009.070531\n",
       "std       1.240976    26014.728304    3.818617   733.116493     3.972567\n",
       "min       1.000000        0.090706    0.003600     1.350000  1989.000000\n",
       "25%       1.000000        5.442540    0.229000    32.560000  2007.000000\n",
       "50%       1.000000       39.979500    1.260000    55.250000  2010.000000\n",
       "75%       2.000000      526.005000    3.040000   178.500000  2012.000000\n",
       "max       7.000000   730000.000000   25.000000  8500.000000  2014.000000"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>number</th>\n",
       "      <th>orbital_period</th>\n",
       "      <th>mass</th>\n",
       "      <th>distance</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>498.00000</td>\n",
       "      <td>498.000000</td>\n",
       "      <td>498.000000</td>\n",
       "      <td>498.000000</td>\n",
       "      <td>498.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.73494</td>\n",
       "      <td>835.778671</td>\n",
       "      <td>2.509320</td>\n",
       "      <td>52.068213</td>\n",
       "      <td>2007.377510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.17572</td>\n",
       "      <td>1469.128259</td>\n",
       "      <td>3.636274</td>\n",
       "      <td>46.596041</td>\n",
       "      <td>4.167284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.328300</td>\n",
       "      <td>0.003600</td>\n",
       "      <td>1.350000</td>\n",
       "      <td>1989.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.00000</td>\n",
       "      <td>38.272250</td>\n",
       "      <td>0.212500</td>\n",
       "      <td>24.497500</td>\n",
       "      <td>2005.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.00000</td>\n",
       "      <td>357.000000</td>\n",
       "      <td>1.245000</td>\n",
       "      <td>39.940000</td>\n",
       "      <td>2009.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2.00000</td>\n",
       "      <td>999.600000</td>\n",
       "      <td>2.867500</td>\n",
       "      <td>59.332500</td>\n",
       "      <td>2011.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>6.00000</td>\n",
       "      <td>17337.500000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>354.000000</td>\n",
       "      <td>2014.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          number  orbital_period        mass    distance         year\n",
       "count  498.00000      498.000000  498.000000  498.000000   498.000000\n",
       "mean     1.73494      835.778671    2.509320   52.068213  2007.377510\n",
       "std      1.17572     1469.128259    3.636274   46.596041     4.167284\n",
       "min      1.00000        1.328300    0.003600    1.350000  1989.000000\n",
       "25%      1.00000       38.272250    0.212500   24.497500  2005.000000\n",
       "50%      1.00000      357.000000    1.245000   39.940000  2009.000000\n",
       "75%      2.00000      999.600000    2.867500   59.332500  2011.000000\n",
       "max      6.00000    17337.500000   25.000000  354.000000  2014.000000"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets.dropna().describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Groupby: Split, Apply, Combine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='images/split-apply-combine.png'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pandas.core.groupby.generic.DataFrameGroupBy object at 0x000001FCF1E0C940>"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets.groupby('method')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>number</th>\n",
       "      <th>orbital_period</th>\n",
       "      <th>mass</th>\n",
       "      <th>distance</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>method</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Astrometry</th>\n",
       "      <td>2</td>\n",
       "      <td>1.262360e+03</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>35.75</td>\n",
       "      <td>4023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eclipse Timing Variations</th>\n",
       "      <td>15</td>\n",
       "      <td>4.276480e+04</td>\n",
       "      <td>10.25000</td>\n",
       "      <td>1261.44</td>\n",
       "      <td>18090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Imaging</th>\n",
       "      <td>50</td>\n",
       "      <td>1.418973e+06</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>2166.91</td>\n",
       "      <td>76347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Microlensing</th>\n",
       "      <td>27</td>\n",
       "      <td>2.207500e+04</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>41440.00</td>\n",
       "      <td>46225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Orbital Brightness Modulation</th>\n",
       "      <td>5</td>\n",
       "      <td>2.127920e+00</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>2360.00</td>\n",
       "      <td>6035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pulsar Timing</th>\n",
       "      <td>11</td>\n",
       "      <td>3.671511e+04</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>1200.00</td>\n",
       "      <td>9992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pulsation Timing Variations</th>\n",
       "      <td>1</td>\n",
       "      <td>1.170000e+03</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Radial Velocity</th>\n",
       "      <td>952</td>\n",
       "      <td>4.553151e+05</td>\n",
       "      <td>1341.65638</td>\n",
       "      <td>27348.11</td>\n",
       "      <td>1110158</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Transit</th>\n",
       "      <td>776</td>\n",
       "      <td>8.377523e+03</td>\n",
       "      <td>1.47000</td>\n",
       "      <td>134242.77</td>\n",
       "      <td>798461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Transit Timing Variations</th>\n",
       "      <td>9</td>\n",
       "      <td>2.393505e+02</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>3313.00</td>\n",
       "      <td>8050</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               number  orbital_period        mass   distance  \\\n",
       "method                                                                         \n",
       "Astrometry                          2    1.262360e+03     0.00000      35.75   \n",
       "Eclipse Timing Variations          15    4.276480e+04    10.25000    1261.44   \n",
       "Imaging                            50    1.418973e+06     0.00000    2166.91   \n",
       "Microlensing                       27    2.207500e+04     0.00000   41440.00   \n",
       "Orbital Brightness Modulation       5    2.127920e+00     0.00000    2360.00   \n",
       "Pulsar Timing                      11    3.671511e+04     0.00000    1200.00   \n",
       "Pulsation Timing Variations         1    1.170000e+03     0.00000       0.00   \n",
       "Radial Velocity                   952    4.553151e+05  1341.65638   27348.11   \n",
       "Transit                           776    8.377523e+03     1.47000  134242.77   \n",
       "Transit Timing Variations           9    2.393505e+02     0.00000    3313.00   \n",
       "\n",
       "                                  year  \n",
       "method                                  \n",
       "Astrometry                        4023  \n",
       "Eclipse Timing Variations        18090  \n",
       "Imaging                          76347  \n",
       "Microlensing                     46225  \n",
       "Orbital Brightness Modulation     6035  \n",
       "Pulsar Timing                     9992  \n",
       "Pulsation Timing Variations       2007  \n",
       "Radial Velocity                1110158  \n",
       "Transit                         798461  \n",
       "Transit Timing Variations         8050  "
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets.groupby('method').sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>number</th>\n",
       "      <th>orbital_period</th>\n",
       "      <th>mass</th>\n",
       "      <th>distance</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>method</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Astrometry</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>631.180000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17.875000</td>\n",
       "      <td>2011.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eclipse Timing Variations</th>\n",
       "      <td>1.666667</td>\n",
       "      <td>4751.644444</td>\n",
       "      <td>5.125000</td>\n",
       "      <td>315.360000</td>\n",
       "      <td>2010.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Imaging</th>\n",
       "      <td>1.315789</td>\n",
       "      <td>118247.737500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>67.715937</td>\n",
       "      <td>2009.131579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Microlensing</th>\n",
       "      <td>1.173913</td>\n",
       "      <td>3153.571429</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4144.000000</td>\n",
       "      <td>2009.782609</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Orbital Brightness Modulation</th>\n",
       "      <td>1.666667</td>\n",
       "      <td>0.709307</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1180.000000</td>\n",
       "      <td>2011.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pulsar Timing</th>\n",
       "      <td>2.200000</td>\n",
       "      <td>7343.021201</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1200.000000</td>\n",
       "      <td>1998.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pulsation Timing Variations</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1170.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2007.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Radial Velocity</th>\n",
       "      <td>1.721519</td>\n",
       "      <td>823.354680</td>\n",
       "      <td>2.630699</td>\n",
       "      <td>51.600208</td>\n",
       "      <td>2007.518987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Transit</th>\n",
       "      <td>1.954660</td>\n",
       "      <td>21.102073</td>\n",
       "      <td>1.470000</td>\n",
       "      <td>599.298080</td>\n",
       "      <td>2011.236776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Transit Timing Variations</th>\n",
       "      <td>2.250000</td>\n",
       "      <td>79.783500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1104.333333</td>\n",
       "      <td>2012.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 number  orbital_period      mass  \\\n",
       "method                                                              \n",
       "Astrometry                     1.000000      631.180000       NaN   \n",
       "Eclipse Timing Variations      1.666667     4751.644444  5.125000   \n",
       "Imaging                        1.315789   118247.737500       NaN   \n",
       "Microlensing                   1.173913     3153.571429       NaN   \n",
       "Orbital Brightness Modulation  1.666667        0.709307       NaN   \n",
       "Pulsar Timing                  2.200000     7343.021201       NaN   \n",
       "Pulsation Timing Variations    1.000000     1170.000000       NaN   \n",
       "Radial Velocity                1.721519      823.354680  2.630699   \n",
       "Transit                        1.954660       21.102073  1.470000   \n",
       "Transit Timing Variations      2.250000       79.783500       NaN   \n",
       "\n",
       "                                  distance         year  \n",
       "method                                                   \n",
       "Astrometry                       17.875000  2011.500000  \n",
       "Eclipse Timing Variations       315.360000  2010.000000  \n",
       "Imaging                          67.715937  2009.131579  \n",
       "Microlensing                   4144.000000  2009.782609  \n",
       "Orbital Brightness Modulation  1180.000000  2011.666667  \n",
       "Pulsar Timing                  1200.000000  1998.400000  \n",
       "Pulsation Timing Variations            NaN  2007.000000  \n",
       "Radial Velocity                  51.600208  2007.518987  \n",
       "Transit                         599.298080  2011.236776  \n",
       "Transit Timing Variations      1104.333333  2012.500000  "
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets.groupby('method').mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "year   \n",
       "1989  0    Radial Velocity\n",
       "1992  0      Pulsar Timing\n",
       "1994  0      Pulsar Timing\n",
       "1995  0    Radial Velocity\n",
       "1996  0    Radial Velocity\n",
       "1997  0    Radial Velocity\n",
       "1998  0    Radial Velocity\n",
       "1999  0    Radial Velocity\n",
       "2000  0    Radial Velocity\n",
       "2001  0    Radial Velocity\n",
       "2002  0    Radial Velocity\n",
       "2003  0    Radial Velocity\n",
       "2004  0    Radial Velocity\n",
       "2005  0    Radial Velocity\n",
       "2006  0    Radial Velocity\n",
       "2007  0    Radial Velocity\n",
       "2008  0    Radial Velocity\n",
       "2009  0    Radial Velocity\n",
       "2010  0            Transit\n",
       "2011  0    Radial Velocity\n",
       "2012  0            Transit\n",
       "2013  0            Transit\n",
       "2014  0            Transit\n",
       "Name: method, dtype: object"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets.groupby('year')['method'].apply(pd.Series.mode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>method</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Astrometry</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2011.500000</td>\n",
       "      <td>2.121320</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>2010.75</td>\n",
       "      <td>2011.5</td>\n",
       "      <td>2012.25</td>\n",
       "      <td>2013.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Eclipse Timing Variations</th>\n",
       "      <td>9.0</td>\n",
       "      <td>2010.000000</td>\n",
       "      <td>1.414214</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>2009.00</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>2011.00</td>\n",
       "      <td>2012.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Imaging</th>\n",
       "      <td>38.0</td>\n",
       "      <td>2009.131579</td>\n",
       "      <td>2.781901</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>2008.00</td>\n",
       "      <td>2009.0</td>\n",
       "      <td>2011.00</td>\n",
       "      <td>2013.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Microlensing</th>\n",
       "      <td>23.0</td>\n",
       "      <td>2009.782609</td>\n",
       "      <td>2.859697</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>2008.00</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>2012.00</td>\n",
       "      <td>2013.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Orbital Brightness Modulation</th>\n",
       "      <td>3.0</td>\n",
       "      <td>2011.666667</td>\n",
       "      <td>1.154701</td>\n",
       "      <td>2011.0</td>\n",
       "      <td>2011.00</td>\n",
       "      <td>2011.0</td>\n",
       "      <td>2012.00</td>\n",
       "      <td>2013.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pulsar Timing</th>\n",
       "      <td>5.0</td>\n",
       "      <td>1998.400000</td>\n",
       "      <td>8.384510</td>\n",
       "      <td>1992.0</td>\n",
       "      <td>1992.00</td>\n",
       "      <td>1994.0</td>\n",
       "      <td>2003.00</td>\n",
       "      <td>2011.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pulsation Timing Variations</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2007.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>2007.00</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>2007.00</td>\n",
       "      <td>2007.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Radial Velocity</th>\n",
       "      <td>553.0</td>\n",
       "      <td>2007.518987</td>\n",
       "      <td>4.249052</td>\n",
       "      <td>1989.0</td>\n",
       "      <td>2005.00</td>\n",
       "      <td>2009.0</td>\n",
       "      <td>2011.00</td>\n",
       "      <td>2014.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Transit</th>\n",
       "      <td>397.0</td>\n",
       "      <td>2011.236776</td>\n",
       "      <td>2.077867</td>\n",
       "      <td>2002.0</td>\n",
       "      <td>2010.00</td>\n",
       "      <td>2012.0</td>\n",
       "      <td>2013.00</td>\n",
       "      <td>2014.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Transit Timing Variations</th>\n",
       "      <td>4.0</td>\n",
       "      <td>2012.500000</td>\n",
       "      <td>1.290994</td>\n",
       "      <td>2011.0</td>\n",
       "      <td>2011.75</td>\n",
       "      <td>2012.5</td>\n",
       "      <td>2013.25</td>\n",
       "      <td>2014.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               count         mean       std     min      25%  \\\n",
       "method                                                                         \n",
       "Astrometry                       2.0  2011.500000  2.121320  2010.0  2010.75   \n",
       "Eclipse Timing Variations        9.0  2010.000000  1.414214  2008.0  2009.00   \n",
       "Imaging                         38.0  2009.131579  2.781901  2004.0  2008.00   \n",
       "Microlensing                    23.0  2009.782609  2.859697  2004.0  2008.00   \n",
       "Orbital Brightness Modulation    3.0  2011.666667  1.154701  2011.0  2011.00   \n",
       "Pulsar Timing                    5.0  1998.400000  8.384510  1992.0  1992.00   \n",
       "Pulsation Timing Variations      1.0  2007.000000       NaN  2007.0  2007.00   \n",
       "Radial Velocity                553.0  2007.518987  4.249052  1989.0  2005.00   \n",
       "Transit                        397.0  2011.236776  2.077867  2002.0  2010.00   \n",
       "Transit Timing Variations        4.0  2012.500000  1.290994  2011.0  2011.75   \n",
       "\n",
       "                                  50%      75%     max  \n",
       "method                                                  \n",
       "Astrometry                     2011.5  2012.25  2013.0  \n",
       "Eclipse Timing Variations      2010.0  2011.00  2012.0  \n",
       "Imaging                        2009.0  2011.00  2013.0  \n",
       "Microlensing                   2010.0  2012.00  2013.0  \n",
       "Orbital Brightness Modulation  2011.0  2012.00  2013.0  \n",
       "Pulsar Timing                  1994.0  2003.00  2011.0  \n",
       "Pulsation Timing Variations    2007.0  2007.00  2007.0  \n",
       "Radial Velocity                2009.0  2011.00  2014.0  \n",
       "Transit                        2012.0  2013.00  2014.0  \n",
       "Transit Timing Variations      2012.5  2013.25  2014.0  "
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "planets.groupby('method')['year'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pivot Tables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have seen how the GroupBy abstraction lets us explore relationships within a dataset.\n",
    "A pivot table is a similar operation that is commonly seen in spreadsheets and\n",
    "other programs that operate on tabular data. The pivot table takes simple columnwise\n",
    "data as input, and groups the entries into a two-dimensional table that provides\n",
    "a multidimensional summarization of the data. The difference between pivot tables\n",
    "and GroupBy can sometimes cause confusion; it helps me to think of pivot tables as\n",
    "essentially a multidimensional version of GroupBy aggregation. That is, you splitapply-\n",
    "combine, but both the split and the combine happen across not a onedimensional\n",
    "index, but across a two-dimensional grid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>embarked</th>\n",
       "      <th>class</th>\n",
       "      <th>who</th>\n",
       "      <th>adult_male</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alive</th>\n",
       "      <th>alone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "      <td>C</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "      <td>S</td>\n",
       "      <td>First</td>\n",
       "      <td>woman</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>yes</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>S</td>\n",
       "      <td>Third</td>\n",
       "      <td>man</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>no</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   survived  pclass     sex   age  sibsp  parch     fare embarked  class  \\\n",
       "0         0       3    male  22.0      1      0   7.2500        S  Third   \n",
       "1         1       1  female  38.0      1      0  71.2833        C  First   \n",
       "2         1       3  female  26.0      0      0   7.9250        S  Third   \n",
       "3         1       1  female  35.0      1      0  53.1000        S  First   \n",
       "4         0       3    male  35.0      0      0   8.0500        S  Third   \n",
       "\n",
       "     who  adult_male deck  embark_town alive  alone  \n",
       "0    man        True  NaN  Southampton    no  False  \n",
       "1  woman       False    C    Cherbourg   yes  False  \n",
       "2  woman       False  NaN  Southampton   yes   True  \n",
       "3  woman       False    C  Southampton   yes  False  \n",
       "4    man        True  NaN  Southampton    no   True  "
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic = sns.load_dataset('titanic')\n",
    "titanic.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>class</th>\n",
       "      <th>First</th>\n",
       "      <th>Second</th>\n",
       "      <th>Third</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>0.968085</td>\n",
       "      <td>0.921053</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>0.368852</td>\n",
       "      <td>0.157407</td>\n",
       "      <td>0.135447</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "class      First    Second     Third\n",
       "sex                                 \n",
       "female  0.968085  0.921053  0.500000\n",
       "male    0.368852  0.157407  0.135447"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.pivot_table('survived', index='sex', columns='class')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "# call signature as of Pandas 0.18\n",
    "DataFrame.pivot_table(data, values=None, index=None, columns=None,\n",
    "                        aggfunc='mean', fill_value=None, margins=False,\n",
    "                        dropna=True, margins_name='All')```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">fare</th>\n",
       "      <th colspan=\"3\" halign=\"left\">survived</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>class</th>\n",
       "      <th>First</th>\n",
       "      <th>Second</th>\n",
       "      <th>Third</th>\n",
       "      <th>First</th>\n",
       "      <th>Second</th>\n",
       "      <th>Third</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>106.125798</td>\n",
       "      <td>21.970121</td>\n",
       "      <td>16.118810</td>\n",
       "      <td>91</td>\n",
       "      <td>70</td>\n",
       "      <td>72</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>67.226127</td>\n",
       "      <td>19.741782</td>\n",
       "      <td>12.661633</td>\n",
       "      <td>45</td>\n",
       "      <td>17</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              fare                       survived             \n",
       "class        First     Second      Third    First Second Third\n",
       "sex                                                           \n",
       "female  106.125798  21.970121  16.118810       91     70    72\n",
       "male     67.226127  19.741782  12.661633       45     17    47"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.pivot_table(index='sex', columns='class',\n",
    "                        aggfunc={'survived':sum, 'fare':'mean'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>class</th>\n",
       "      <th>First</th>\n",
       "      <th>Second</th>\n",
       "      <th>Third</th>\n",
       "      <th>All</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>female</th>\n",
       "      <td>0.968085</td>\n",
       "      <td>0.921053</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.742038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>0.368852</td>\n",
       "      <td>0.157407</td>\n",
       "      <td>0.135447</td>\n",
       "      <td>0.188908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>All</th>\n",
       "      <td>0.629630</td>\n",
       "      <td>0.472826</td>\n",
       "      <td>0.242363</td>\n",
       "      <td>0.383838</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "class      First    Second     Third       All\n",
       "sex                                           \n",
       "female  0.968085  0.921053  0.500000  0.742038\n",
       "male    0.368852  0.157407  0.135447  0.188908\n",
       "All     0.629630  0.472826  0.242363  0.383838"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic.pivot_table('survived', index='sex', columns='class', margins=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tidy Data:\n",
    "https://www.ibm.com/developerworks/community/blogs/jfp/entry/Tidy_Data_In_Python?lang=en"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}