Pre-Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PYSPARK NLP MODELLING "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summary: \n",
    "********\n",
    "- Building a model that can detect sentiment using PySpark.\n",
    "- Dataset being used is \"Sentiment140\" which contains info about 1.6 million tweets\n",
    "- More info on the dataset can be found from the link >>\n",
    "http://help.sentiment140.com/for-students/<br>\n",
    "The dataset can be downloaded from the below link.<br>\n",
    "http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By looking at the description of the dataset from the link, the information on each field can be found.\n",
    "\n",
    "0 - polarity of the tweet (0 = negative, 2 = neutral, 4 = positive)<br>\n",
    "1 - id of the tweet<br>\n",
    "2 - date of the tweet (Sat May 16 23:58:44 UTC 2009)<br>\n",
    "3 - query. If there is no query, then this value is NO_QUERY.<br>\n",
    "4 - UserID<br>\n",
    "5 - text of the tweet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd  \n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Assigning column names\n",
    "cols = ['sentiment','id','date','query_string','user','text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"/Users/arun/Downloads/CS777/trainingandtestdata/training.1600000.processed.noemoticon.csv\",header=None, names=cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>id</th>\n",
       "      <th>date</th>\n",
       "      <th>query_string</th>\n",
       "      <th>user</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810369</td>\n",
       "      <td>Mon Apr 06 22:19:45 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>_TheSpecialOne_</td>\n",
       "      <td>@switchfoot http://twitpic.com/2y1zl - Awww, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810672</td>\n",
       "      <td>Mon Apr 06 22:19:49 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>scotthamilton</td>\n",
       "      <td>is upset that he can't update his Facebook by ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1467810917</td>\n",
       "      <td>Mon Apr 06 22:19:53 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>mattycus</td>\n",
       "      <td>@Kenichan I dived many times for the ball. Man...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1467811184</td>\n",
       "      <td>Mon Apr 06 22:19:57 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>ElleCTF</td>\n",
       "      <td>my whole body feels itchy and like its on fire</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1467811193</td>\n",
       "      <td>Mon Apr 06 22:19:57 PDT 2009</td>\n",
       "      <td>NO_QUERY</td>\n",
       "      <td>Karoli</td>\n",
       "      <td>@nationwideclass no, it's not behaving at all....</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sentiment          id                          date query_string  \\\n",
       "0          0  1467810369  Mon Apr 06 22:19:45 PDT 2009     NO_QUERY   \n",
       "1          0  1467810672  Mon Apr 06 22:19:49 PDT 2009     NO_QUERY   \n",
       "2          0  1467810917  Mon Apr 06 22:19:53 PDT 2009     NO_QUERY   \n",
       "3          0  1467811184  Mon Apr 06 22:19:57 PDT 2009     NO_QUERY   \n",
       "4          0  1467811193  Mon Apr 06 22:19:57 PDT 2009     NO_QUERY   \n",
       "\n",
       "              user                                               text  \n",
       "0  _TheSpecialOne_  @switchfoot http://twitpic.com/2y1zl - Awww, t...  \n",
       "1    scotthamilton  is upset that he can't update his Facebook by ...  \n",
       "2         mattycus  @Kenichan I dived many times for the ball. Man...  \n",
       "3          ElleCTF    my whole body feels itchy and like its on fire   \n",
       "4           Karoli  @nationwideclass no, it's not behaving at all....  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1600000 entries, 0 to 1599999\n",
      "Data columns (total 6 columns):\n",
      "sentiment       1600000 non-null int64\n",
      "id              1600000 non-null int64\n",
      "date            1600000 non-null object\n",
      "query_string    1600000 non-null object\n",
      "user            1600000 non-null object\n",
      "text            1600000 non-null object\n",
      "dtypes: int64(2), object(4)\n",
      "memory usage: 73.2+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4    800000\n",
       "0    800000\n",
       "Name: sentiment, dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sentiment.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inference:\n",
    "*****\n",
    "- 1/2 of the dataset has positive sentiment values\n",
    "- The other 1/2 of the dataset has negative sentiment values\n",
    "- No neutral tweets\n",
    "- No skewness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NO_QUERY    1600000\n",
       "Name: query_string, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.query_string.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#dropping the unecessary values\n",
    "df.drop(['id','date','query_string','user'],axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>@switchfoot http://twitpic.com/2y1zl - Awww, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>is upset that he can't update his Facebook by ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>@Kenichan I dived many times for the ball. Man...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>my whole body feels itchy and like its on fire</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>@nationwideclass no, it's not behaving at all....</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sentiment                                               text\n",
       "0          0  @switchfoot http://twitpic.com/2y1zl - Awww, t...\n",
       "1          0  is upset that he can't update his Facebook by ...\n",
       "2          0  @Kenichan I dived many times for the ball. Man...\n",
       "3          0    my whole body feels itchy and like its on fire \n",
       "4          0  @nationwideclass no, it's not behaving at all...."
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Negative class is from 0 - 799999th index,\n",
    "- Positive class entries start from 800000 to the end of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>@switchfoot http://twitpic.com/2y1zl - Awww, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>is upset that he can't update his Facebook by ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>@Kenichan I dived many times for the ball. Man...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>my whole body feels itchy and like its on fire</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>@nationwideclass no, it's not behaving at all....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>@Kwesidei not the whole crew</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>Need a hug</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>@LOLTrish hey  long time no see! Yes.. Rains a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>@Tatiana_K nope they didn't have it</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>@twittera que me muera ?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sentiment                                               text\n",
       "0          0  @switchfoot http://twitpic.com/2y1zl - Awww, t...\n",
       "1          0  is upset that he can't update his Facebook by ...\n",
       "2          0  @Kenichan I dived many times for the ball. Man...\n",
       "3          0    my whole body feels itchy and like its on fire \n",
       "4          0  @nationwideclass no, it's not behaving at all....\n",
       "5          0                      @Kwesidei not the whole crew \n",
       "6          0                                        Need a hug \n",
       "7          0  @LOLTrish hey  long time no see! Yes.. Rains a...\n",
       "8          0               @Tatiana_K nope they didn't have it \n",
       "9          0                          @twittera que me muera ? "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.sentiment == 0].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>800000</th>\n",
       "      <td>4</td>\n",
       "      <td>I LOVE @Health4UandPets u guys r the best!!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800001</th>\n",
       "      <td>4</td>\n",
       "      <td>im meeting up with one of my besties tonight! ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800002</th>\n",
       "      <td>4</td>\n",
       "      <td>@DaRealSunisaKim Thanks for the Twitter add, S...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800003</th>\n",
       "      <td>4</td>\n",
       "      <td>Being sick can be really cheap when it hurts t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800004</th>\n",
       "      <td>4</td>\n",
       "      <td>@LovesBrooklyn2 he has that effect on everyone</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800005</th>\n",
       "      <td>4</td>\n",
       "      <td>@ProductOfFear You can tell him that I just bu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800006</th>\n",
       "      <td>4</td>\n",
       "      <td>@r_keith_hill Thans for your response. Ihad al...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800007</th>\n",
       "      <td>4</td>\n",
       "      <td>@KeepinUpWKris I am so jealous, hope you had a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800008</th>\n",
       "      <td>4</td>\n",
       "      <td>@tommcfly ah, congrats mr fletcher for finally...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800009</th>\n",
       "      <td>4</td>\n",
       "      <td>@e4VoIP I RESPONDED  Stupid cat is helping me ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        sentiment                                               text\n",
       "800000          4       I LOVE @Health4UandPets u guys r the best!! \n",
       "800001          4  im meeting up with one of my besties tonight! ...\n",
       "800002          4  @DaRealSunisaKim Thanks for the Twitter add, S...\n",
       "800003          4  Being sick can be really cheap when it hurts t...\n",
       "800004          4    @LovesBrooklyn2 he has that effect on everyone \n",
       "800005          4  @ProductOfFear You can tell him that I just bu...\n",
       "800006          4  @r_keith_hill Thans for your response. Ihad al...\n",
       "800007          4  @KeepinUpWKris I am so jealous, hope you had a...\n",
       "800008          4  @tommcfly ah, congrats mr fletcher for finally...\n",
       "800009          4  @e4VoIP I RESPONDED  Stupid cat is helping me ..."
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.sentiment == 4].head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([     0,      1,      2,      3,      4,      5,      6,      7,\n",
       "                 8,      9,\n",
       "            ...\n",
       "            799990, 799991, 799992, 799993, 799994, 799995, 799996, 799997,\n",
       "            799998, 799999],\n",
       "           dtype='int64', length=800000)"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.sentiment == 0].index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([ 800000,  800001,  800002,  800003,  800004,  800005,  800006,\n",
       "             800007,  800008,  800009,\n",
       "            ...\n",
       "            1599990, 1599991, 1599992, 1599993, 1599994, 1599995, 1599996,\n",
       "            1599997, 1599998, 1599999],\n",
       "           dtype='int64', length=800000)"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.sentiment == 4].index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#One Hot Encoding\n",
    "df['sentiment'] = df['sentiment'].map({0: 0, 4: 1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    800000\n",
       "0    800000\n",
       "Name: sentiment, dtype: int64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sentiment.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a way of sanity check, let's look at the length of the string in text column in each entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAArwAAAJWCAYAAAC6ZVSnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3X98XVWd7//XSkOLJG0KWGmhYulMkVDHcRCqJGpA7+1DQNA6qL34LcWZYVB+WWKHfr+oxdtR75RhUloQQWGAKlD91keBToThe0WiNFyjMqMiBw1DEaktINikJy3FNuv7x/lhkp6UpE1ysvd5PR+PPPY5e6/PPuvwx3m8u1h7rRBjRJIkSUqrqnJ3QJIkSRpNBl5JkiSlmoFXkiRJqWbglSRJUqoZeCVJkpRqBl5JkiSlmoFXkiRJqWbglSRJUqoZeCVJkpRqBl5JkiSlWnW5OzAWurq63D9ZkiQpRerq6sJQ2zrCK0mSpFQz8EqSJCnVDLySJElKNQOvJEmSUs3AK0mSpFQz8EpSSnR2dtLZ2VnubkjSuGPglSRJUqoZeCVJkpRqBl5JkiSlmoFXkiRJqWbglSRJUqoZeCVJkpRqBl5JkiSlmoFXkiRJqWbglSRJUqoZeCVJkpRqBl5JkiSlmoFXkiRJqWbglSRJUqoZeCVJkpRq1eXugCTp4GUyGdavX09PTw+zZs2iqamJ+vr6cndLksYFA68kJVhbWxsrV66kvb19n2sNDQ0sW7aMpqamMvRMksaPEGMsdx9GXVdXV/q/pKSKs3btWpYsWUJvby+1tbWcdtppTJs2jd27d3PfffeRzWapqqpi9erVLFq0qNzdlaQRVVdXF4ba1sArSQnU1tbGggUL6O3tpbm5mSuuuIJt27YBMGfOHHbs2MGqVatoaWmhqqqKDRs2ONIrKVWGE3h9aE2SEmjlypXFsLt8+XImT57c7/rkyZNZvnw5zc3N9Pb2cs0115Spp5JUfo7wSlLCZDIZTj31VGpra8lkMsWw29nZCeRGeAu6u7s58cQTyWazPProoz7IJik1HOGVpBRra2sD4JxzzimG3Uwmw7p167j11lu56aabyGQyAEyZMoWzzz67X50kVRpXaZCkhNmxYwcAM2bMGNIqDTNmzOhXJ0mVxsArSQlTGNV95JFHWLVq1aCrNLS3t7NgwQJOPvnkfnWSVGmcwytJCVOYw1vwWqs0FDiHV1KauCzZAAZeSWlz7LHH0t3dzbx583jwwQeB0g+tzZ8/n46ODqZMmcKzzz5blr5K0mjwoTVJSrFMJkN3dzcAHR0drFixovi+oLu7mxUrVtDR0VF8X3iQTZIqjXN4JSlhCqstvOMd7+DHP/4xLS0tfO1rX6Opqak4h3fjxo3FndZOPvlkOjo6aGtrc0qDpIrkCK8kJUxhtYV3vetdbNiwgcbGRrLZLK2trdx+++3cfffdZLNZGhsb2bBhA+9617v61UlSpXGEV5ISprDawtatW2lqaqKpqYlMJsP69evp6elh1qxZNDU1FUdz161b169OkiqND61JUsK405ok+dCaJKVafX09DQ0NZLNZVq1atd+21113XXF6g2FXUqUy8EpSAi1btoyqqipaWlr2u0pDS0sLVVVVXHnllWXqqSSVn1MaJCmh1q5dy5IlS4o7rQ22SsPq1atZtGhRubsrSSPKjScGMPBKSqu2tjauueYaNm3atM+1xsZGrrzySpqamsrQM0kaXQbeAQy8ktLu5ptv5pZbbmHnzp1MnTqV888/n4suuqjc3ZKkUWPgHcDAKymt1qxZw7XXXrvPHF6AKVOmsHTpUi6//PIy9EySRpeBdwADr6Q0+tSnPsXdd99dfD9z5kyOOOIIstksTz/9dPH8eeedx4033liOLkrSqBlO4HXjCUlKoDVr1hTD7rx587jjjjvIZrNAbh3erVu3snjxYjo6Orjrrrs44YQTHOmVVLEc4ZWkBDr22GPp7u5m3rx5PPjgg0DpjSfmz59PR0cHU6ZM4dlnny1LXyVpNLjxhCSlWGtra3HO7h133LHftrfddhuQW5e3tbV11PsmSeORgVeSEmbdunUAzJ49mxkzZuy37THHHMNxxx3Xr06SKo2BV5ISpjC6O23atCG1L7QrtZKDJFUCA68kJcyUKVMAePHFF4fUvtCuUCdJlcbAK0kJs3DhQgCefvpptm7dut+2W7ZsYfPmzf3qJKnSGHglKWHOOuus4mjt4sWL99v2E5/4BJAb3T3rrLNGvW+SNB4ZeCUpgZYuXQpAR0cH8+fPZ8uWLf2ub9mypbgkWd/2klSJXIdXkhKq1E5rhx9+ONlstjiNAdxpTVI6jfk6vCGElSGE74UQfhtC2BVCeDmE8B8hhKtDCEcOaDsrhBD38zfoujkhhMUhhI4QQjaE0BVCeDiE8IGR+A6SlDRf/epXWbFiRXF6w3PPPccvfvGLYtidMmUKK1asMOxKqngjMsIbQngVeAx4AngBqAHeCZwM/A54Z4zxt/m2s4DNwM+Ae0rc7vEY4/oSn3Et8BngOWA9MBFYCBwBXBZjvGGw/jnCKyntWltbueWWW8hms0yfPp2FCxc6Z1dSqg1nhLd6hD5zSozxlYEnQwhfAq4C/h/g4gGX/zPG+IWh3DyE0EAu7P4XcEqM8Q/58/8M/BS4NoTwbzHGZw74G0hSgs2ePZuTTjqJnp4eZs2axezZs8vdJUkaN0Yk8JYKu3nfJhd45wxyfag+mT9+qRB285/7TAjhK8DngU8AVx/k50hSorS1tbFy5Ura29v3udbQ0MCyZctoamoqQ88kafwY7VUazs4ff17i2tEhhItCCFflj2/dz33emz8+UOLa/QPaSFJFWLt2LQsWLKC9vZ3a2lo+8IEP8IlPfILzzjuP2tpa2tvbWbBgAd/4xjfK3VVJKqsRXaUhhLAUqAXqyM3ffRe5sPvfYowv5tvMIjeHt5SHgcUxxmf73LMGyALZGOPkEp/5euBF4IUY41GlbjrYHN7Ozs6hfC1JGnc6Ojq47LLL6O3t5YILLuCCCy6gpqameL2np4fbb7+d22+/naqqKq6//nrmzZtXxh5L0oGZM6f0RIExX6Whj6XkphUsIRd2HwDmF8Ju3k7gH4G3A4fn/5qA7wOnAd/Lh9yCuvyxa5DPLJyfOgL9l6REuOWWW4ph95JLLukXdgFqamq45JJLuOCCC+jt7eXWW28tU08lqfxGZR3eEMJRQAPwT8Bk4AMxxsdeo6YaeAR4B7Akxrg6f/5oYAuwJcY4s0TdIcCrwKsxxkml7u0qDZLSJJPJcOqpp1JbW0smk2Hy5Nz//Cr8X6u+oyHd3d2ceOKJZLNZHn30Uerr68vSZ0kaaeUc4QUgxvh8jHEDMB84Elg7hJo9wC35t+/pc6kwgltHaYXz2w+gq5KUOG1tbQCcc845xbA7mClTpnD22Wf3q5OkSjOqD63FGH9Dbm3eufm5tq+lMPWh+P/mYow95EZ4a0MIM0rUFIYyfn0wfZWkpNixYwcAM2aU+kncV6FdoU6SKs1or9IAcHT+uHcIbd+ZPz494PxD+eP7S9ScMaCNJKVaYVR369atQ2pfaPdao8GSlFYHHXhDCMeHEPaZbhBCqMpvPPEGoL3PZhEnhRD2+dwQwvuAK/Jvvzng8k3542dDCIf3qZkFXALsBm47yK8iSYlQWFf3vvvue81R2+7ubjZu3NivTpIqzUiM8J4JbAsh/H8hhK+FEP5XCOFfgU5ym05sAy7s074F+G0I4f8NIazK/30P+N/AJODzMcZ+K6jn37cAfwb8PF/zFeAn5LYWXuoua5IqRX19PQ0NDWSzWVatWrXfttdddx3ZbJbGxkYfWJNUsQ56lYYQwlvI7YT2LmAmueXBesjNqW0F1sQYX+7T/m+BBcBbgNcDhwDPA48CN8QYf7ifz7qA3IjuiUAv8BjwzzHGf9tfH12lQVLatLW1sWDBAnp7e2lubmbJkiU8//zzQG6Vhu7ubq677jpaWlqoqqpiw4YNjvBKSpXhrNIwKsuSjTcGXklptHbtWpYsWUJvby+1tbU0NTUxbdo0du/ezcaNG8lms1RVVbF69WoWLVpU7u5K0ogy8A5g4JWUVm1tbVxzzTVs2rRpn2uNjY1ceeWVjuxKSiUD7wAGXklpl8lkWL9+PT09PcyaNYumpibn7EpKNQPvAAZeSZWg1E5rkpRWZd9pTZIkSRovDLySJElKNQOvJEmSUs3AK0mSpFQz8EqSJCnVDLySJElKNQOvJEmSUs3AK0mSpFQz8EqSJCnVDLySJElKtepyd0CSdPAymQzr16+np6eHWbNm0dTURH19fbm7JUnjgoFXkhKsra2NlStX0t7evs+1hoYGli1bRlNTUxl6JknjR4gxlrsPo66rqyv9X1JSxVm7di1Lliyht7eX2tpaTjvtNKZNm8bu3bu57777yGazVFVVsXr1ahYtWlTu7krSiKqrqwtDbescXklKoLa2tmLYbW5u5p577uHNb34zkyZN4q1vfSv33HMPzc3N9Pb28ulPf5q2trZyd1mSysYRXklKoDPPPJP29nbOPfdcfve73w06peHoo49m/fr1NDY20traWoaeStLoGM4Ir4FXkhImk8lw6qmnMmnSJF599VVijINOaQghMHHiRHbv3s2jjz7qg2ySUsMpDZKUYoXpCbt37ybGSHNzM5lMhquvvpqLL76YG2+8kUwmQ3NzMzFGdu/e3a9OkiqNqzRIUsLs2LGj+Lq5uZnly5eXXJZs+fLlALS0tOxTJ0mVxMArSQmzc+dOAKqrqznllFOK83kHamho4LLLLqO6upo9e/YU6ySp0hh4JSmhent7Oe+88wadw9ve3s6jjz5KCEOe5iZJqWTglaSEOeyww4Bc4IXctIYrrriCbdu2ATBnzhxWrlzJqlWraGlpofBwcqFOkiqND61JUsJMnjx5n3MDV9wptQJPqTpJqgSO8EpSwrzxjW8svq6qqqKlpYWvfe1rNDU1Fac0bNy4sbjTWmEkuG+dJFUSA68kJcxvf/vb4usPf/jDbN26lU2bNu2zsURjYyMzZsxg/fr1+9RJUiUx8EpSwhSWFwshsH79epqbm/nCF77AAw88UFyW7O1vfzv3338/LS0thBCIMbosmaSK5RxeSUqYwlzcefPmFac0LFiwgCeffJJXXnmFn/3sZyxYsICWlhaqqqo45ZRT+tVJUqVxhFeSEqapqQmAX/7yl9x5553ccMMNg05puOSSS/j7v//7fnWSVGkMvJKUMPX19TQ0NNDe3s6Pf/xjWltbS+60Vl9fz4oVK8hmszQ2NlJfX1/urktSWYRSS9ekTVdXV/q/pKSK0tbWxoIFC+jt7aW5uZklS5bw/PPPA7l1eLu7u7nuuuuK0xo2bNjgCK+kVKmrqxvyrjoGXklKqLVr17JkyRJ6e3t53etex3HHHcekSZMIIZDJZNi1axdVVVWsXr2aRYsWlbu7kjSihhN4ndIgSQl1/vnns337dq699lq6u7t54okn+l2fMmUKS5cuNexKqniO8EpSQvUd4T300EOZOnUqIQSqq6t58cUXeeWVVxzhlZRaTmkYwMArKW36zuE95phj2LJlyz5tCuedwyspjYYTeF2HV5ISaOXKlcUtg0uF3b7ne3t7ueaaa8asb5I03jjCK0kJk8lkOPXUU/udq6mp4fTTT2fatGns3r2be++9l56enn5tHn30UZcmk5QajvBKUoq1tbX1e9/c3MyTTz7J1VdfzcUXX8yNN97Ik08+SXNz837rJKlSGHglKWGefvrp4utLL72U5cuX77Nt8OTJk1m+fDmXXHJJyTpJqiQGXklKmF//+tcAhBBYtmzZftsuW7aMEEK/OkmqNAZeSUqYV199dUzrJCnpDLySlDC1tbUAxBhZtWrVftted911FB5OLtRJUqUx8EpSwsydO7f4uqWlhRUrVtDd3d2vTXd3NytWrKClpaVknSRVEpclk6SEKbUsWW1tLU1NTcVlyTZu3Eg2m+3XxmXJJKWJO60NYOCVlDYNDQ088cQTAMycOZPnnntunzZ9z8+dO5dNmzaNaR8laTQZeAcw8EpKm7a2Nj70oQ8V5+e+7nWvY/bs2UycOJEQAplMhl27dgG51RzuuecetxaWlCoG3gEMvJLSaO3atXz6059mf7/jIQTWrFnDokWLxrBnkjT63GlNkirA+eefzz333ENjY2PJ642Njdxzzz2GXUkVzxFeSUqBTCbD+vXr6enpYdasWTQ1NfmAmqRUc0rDAAZeSZWgs7MTgDlz5pS5J5I0+oYTeKtHsyOSpLHhCK8kDW5EAm8IYSVwMnA88HpgF/Ab4B7ghhjjSyVqGoDPAe8EXgd0Av8KXB9j3DvI53wAWAr8FTAB+CVwY4zxjpH4HpKUNG1tbaxcuZL29vZ9rjU0NLBs2TJXZ5BU8UZkSkMI4VXgMeAJ4AWghlyQPRn4HfDOGONv+7T/IPAd4BXgW8DLwNnAm4H1McaPlPiMS4HrgZfyNa8C5wIzgX+JMS4drH9OaZCURmvXrmXJkiX09vYO2qaqqorVq1f74Jqk1BnzObwhhENjjK+UOP8l4CrgqzHGi/PnpgBPAXVAY4zxJ4V7AA8BpwL/I8a4rs99ZgFPAj3A22OMz+TPHw78GPgzoCHG+Gip/hl4JaXNwHV4a2pqOP3004s7rd1777309PQArsMrKZ3GfFmyUmE379v5Y98nKM4FpgHrCmG3zz0+l3/7qQH3+RtgErnpEc/0qfkD8OX8208eUOclKYGuuuqqYthtbm7mySef5Oqrr+biiy/mxhtv5Mknn6S5uRmAGCOf/exny9ldSSqr0X5o7ez88ed9zr03f3ygRPsfADuBhhDCpBjj7iHU3D+gjSSlWiaT4Ze//CUAl156KcuXL6e1tZVbbrmFbDbL9OnTWbhwIcuXL2f37t185Stf4fHHHyeTyfggm6SKNKKBN4SwFKglN13hZOBd5MLuP/Vp9ub88dcD62OMe0IIm4G5wGwgM4SarSGEHmBmCOGwGOPOofa3sISPJCXJTTfdBMCECRP44x//yMyZM8lms/3abNy4kdraWj7+8Y8zYcIE9u7dy0033cSll15aji5L0gEbiaUWR3qEdylwVJ/3DwAXxBhf7HOuLn/sGuQehfNTh1lTk2835MArSUn01FNPAXDYYYdx8803F8/PnDmTI444gpdffpnnnnuObDbLzTffzOTJk9mxY0exTpIqzYgG3hjjdIAQwlFAA7mR3f8IIXwgxvjYSH7WSHBxdklJVFNTA8COHTsAmDdvHp/73Odoa2srrsN74okn8sUvfpGOjo5iu5qaGn/3JFWkUZnDG2N8HtgQQniM3DSEtcBb8pcLo7R1pWr7nN/e51wXufV968gtSzZYzWAjwJKUGnPnzuXBBx8Ecv9wr66u5pxzztmnXUNDA3/+539eHNmdO3fumPZTksaLUX1oLcb4mxDCE8DbQgivjzH+HvgVf9qk4qd924cQqoHjgD3A030u/Ypc4D0eeHRAzQxy0xmeG878XUlKqunTpxdfd3Z20tnZSW1tLaeddlpxWbL77rtvn80o+tZJUiUZkWXJXsPR+WNh97SH8sf3l2j7HuAwoL3PCg2vVXPGgDaSlGqPPPJIv/fz5s2jo6Oj37JkHR0dzJs3b791klQpDnqEN4RwPPB8jLFrwPkq4B+BN5ALsH/IX1oPrAQWhhCuH7DxxBfzbb464GNuA64ELg0h3DZg44mr8m1uOtjvIklJ0N3d3e99R0cHp5xyyj4jvIWNJwark6RKMRJTGs4E/lcI4RFgM7k5tkcBTeSWFtsGXFhoHGPsDiFcSC74PhxCWEdua+FzyG8tTG7rYPrUbA4h/AOwBvhJCKHU1sIld1mTpLQJIfR7HWOkp6eH1tbWkm0LG1T0rZOkSjISUxr+N3Arud3TPgz8A/DX5ELs/wTmxhif6FsQY7yHXCD+Qb7tZcAfgWZgYSyx33GM8XpyofiXwPnA35ML0xfEGJeOwPeQpER44xvfWHx966230tjYWLJdY2Mjt9xyS8k6SaokBz3CG2N8HBj2SuYxxk3kRoeHU7MR2Djcz5KkNDn22GOLr2+66SYefPBBMpkM69evLy5L1tTURH19PfPnzy9ZJ0mVZLS3FpYkjbDJkycXX3d0dDB//nxuu+02Fi5cCOSWKtuyZQvz58+no6OjZJ0kVRIDryQlTFNTEwDV1dXs2bOHjo4O5s6dy8yZMzn88MPJZrNs3ry52L7QrlAnSZVmLJYlkySNoPr6ehoaGtizZw/vfve7mTJlCgDPPfccv/jFL4phd8qUKbz73e9mz549NDY2Ul9fX85uS1LZhBLPh6VOV1dX+r+kpIrS1tbGggUL6O3tpbm5mfr6eu666y6y2SzTp0/nnHPOIZPJ0NLSQlVVFRs2bHCEV1Kq1NXVDXnpGQOvJCXU2rVrWbJkCb29vdTW1tLU1FRch3fjxo1ks1mqqqpYvXo1ixYtKnd3JWlEGXgHMPBKSqu2tjauueYaNm3atM+1xsZGrrzySkd2JaWSgXcAA6+ktBtsWTJJSisD7wAGXkmVoLOzE8gtSyZJaTecwOuyZJKUAq2trdxyyy3Fh9YWLlzIWWedVe5uSdK44AivJCXYmjVruPbaa+nu7t7n2pQpU1i6dCmXX355GXomSaPLKQ0DGHglpdGnPvUp7r777uL7mTNncsQRR5DNZnn66aeL58877zxuvPHGcnRRkkaNUxokKeXWrFlTDLvz5s3jjjvuIJvNArk5vFu3bmXx4sV0dHRw1113ccIJJzjSK6liOcIrSQl07LHH0t3dzbx583jwwQeB0g+tzZ8/n46ODqZMmcKzzz5blr5K0mgYzgivWwtLUsK0trYW5+zecccd+2172223AdDd3U1ra+uo902SxiMDryQlzLp16wCYPXs2M2bMAHIh+Morr+Tiiy9m0aJFxXB7zDHHcNxxx/Wrk6RK4xxeSUqYwujutGnTBl2lYePGjcVVGqZNm8bmzZtLruQgSZXAwCtJCTNlyhQAfvnLX/KjH/2oeH7gKg3d3d0sX76c2trafnWSVGmc0iBJCbNw4UKA4qoM8+bNI5PJsGHDBm699VYee+wxMpkM8+bN69euUCdJlcZVGiQpgaZOnQpATU0NW7ZsAUqv0nDMMcfQ09MDwPbt28e4l5I0elyHV5JSrO9qCz09PcyfP5+rrrqKH/7wh/T09DBr1ixOOOEEvvzlLxfDbqHO7YYlVSJHeCUpYRYtWsTGjRupra0tTlfYn0K7s88+m2984xtj0ENJGn2uwytJKdZ3lYYQ9v97H0Jg2rRp/eokqdI4pUGSEqaw2sLmzZsBaG5u5sQTT+TOO+8km80yffp0PvjBD/LEE0/Q0tJSbOcqDZIqlYFXkhJm4cKFbNy4EYALL7yQ5cuXk8lkOOmkk4pzeOfOncu5557Ljh07+PrXv16sk6RKZOCVpISZPXt28fXDDz/MmWeeSXt7+z7tGhoaePHFF0vWSVIlMfBKUsK0tbUVX3d2dhaXIxtoYAhua2ujvr5+VPsmSeORgVeSEmbHjh0lzx922GHU1NSwa9eukqs3DFYnSWln4JWkhJk8eXK/9xMnTuTVV19l586d7Ny5c5/zg9VJUqVwWTJJSpg3vvGNxdcXXnghL7zwAnfeeSenn346p5xyCmeffTZ33nknL7zwAhdeeGHJOkmqJI7wSlLC/OQnPym+njRpEgBnnXUWxx9/PNB/a+GJEyf2q3OnNUmVyMArSQnz+OOPF1/fcMMNTJw4kSVLlvRr093dzXXXXcdXvvKVknWSVEkMvJKUMIXd1Y4++mi2bdtGS0sLX/va12hqamLatGns3r2bjRs3ks1mqaqq4qijjmLr1q2vuSubJKWVc3glKWHmzp0LUJy729jYSDabpbW1ldtvv527776bbDZLY2Mjd955Z3Et3kKdJFUaA68kJcxHP/pRAPbs2cO6deuIMZZsF2Nk3bp17Nmzp1+dJFWaMNgPZZp0dXWl/0tKqigNDQ088cQTxfc1NTWcfvrpxSkN9957Lz09PcXrc+fOZdOmTeXoqiSNirq6uiHP03KEV5ISaOHChf3exxj3+evrYx/72Fh2T5LGFUd4JSmBzjzzzH22Dt6fxsZGWltbR7FHkjS2HOGVpBTLZDK0t7dTW1vL3XffTWNjY8l2jY2N3HXXXdTW1rJp0yYymcwY91SSxgeXJZOkhGlrawPgnHPO4YwzzuCMM84gk8mwfv16enp6mDVrFk1NTdTX1wNw9tlnc/fdd9PW1lY8J0mVxMArSQmzY8cOAGbMmFE8V19fX5zX23entb7tCnWSVGmc0iBJCTN58mQAtm7dOqT2hXaFOkmqNAZeSUqYpqYmAO67777XHLXt7u5m48aN/eokqdIYeCUpYerr62loaCCbzbJq1ar9tr3uuuuKu645f1dSpTLwSlICLVu2jKqqKlpaWlixYgXd3d39rnd3d7NixQpaWlqoqqriyiuvLFNPJan8XIdXkhJq7dq1LFmyhN7eXmpra2lqairutLZx40ay2SxVVVWsXr2aRYsWlbu7kjSihrMOr4FXkhKsra2Na665puS2wY2NjVx55ZXO3ZWUSm48IUkVZLCBi0oY0JCkoXCEV5ISauCUhtNOO604peG+++5zSoOkVHNKwwAGXklp09bWxoIFC+jt7aW5uZkrrriCbdu2AbmNJ3bs2MGqVauKD61t2LDBqQ2SUmU4gded1iQpgVauXFkMu8uXLy+5tfDy5csBaGlp4ZprrjHwSqpYBl5JSphMJkN7ezu1tbWccsopnHnmmbS3t+/TrqGhgcsuu4za2lo2bdpEJpNxLV5JFcnAK0kJ09bWBsDcuXP5+Mc/Pugc3vb2dv7P//k/nHzyyXR0dNDW1mbglVSRDnqVhhDCkSGEvwshbAghPBVC2BVC6AohPBJC+NsQQtWA9rNCCHE/f+v281mLQwgdIYRs/jMeDiF84GC/gyQlSWE74R/96EfFaQ2ZTIarr76aiy++mBtvvJFMJkNzczO9vb10dHT0q5OkSjMSI7wfAb4KbAW+DzwLHAV8GLgFOCOE8JG479NxPwPuKXG/x0t9SAhj5PIGAAAdLklEQVThWuAzwHPA14GJwEJgYwjhshjjDSPwXSRp3Js8eXLxdWEOL1B8aK3Qpu8c3oF1klRJDnqVhhDCe4EaoDXG2Nvn/HSgA3gjcG6M8Tv587OAzcAdMcYLhvgZDcAm4L+AU2KMf+hzr5/mP/+EGOMzpepdpUFSmrS2tvLxj38cyM3nnTFjRsmH1urr69myZQtz584F4M477+Sss84qZ9clacSM6SoNMcaHBjm/LYRwE/Al4DTgOwfxMZ/MH79UCLv5z3gmhPAV4PPAJ4CrD+IzJCkRfvvb3xZff/7zn+d3v/vdoA+tHX300SXrJKmSjPZDa3/MH/eUuHZ0COEi4EjgJeDRGOPPB7nPe/PHB0pcu59c4H0vBl5JFaDvXNz169cDUFNTw+mnn158aO3ee+/dJwQ7h1dSpRq1wBtCqAbOz78tFVT/e/6vb83DwOIY47N9ztUAxwDZGOPWEvfpzB+PH24fOzs7X7uRJI0zr7zyyj7nent7iTESY6Srq4ve3t6Sdf7uSUqaOXPmHPQ9RnOE95+AtwDfjTH+e5/zO4F/JPfA2tP5c28FvgCcDnwvhPC2GGNP/lpd/tg1yOcUzk8doX5L0rh2yimnFF+/733v4w9/+AOPPfYYra2t/dqddNJJTJ06lYceemifOkmqJKMSeEMIl5NbUeFJoN8G7jHGF4DlA0p+EEKYDzwCvAP4O2D1aPStr5H4F4MkjbU9e/40S2zXrl089dRTJds99dRTvOUtbym+f9Ob3uTvnqSKdNDr8A4UQriUXFh9Ajg9xvjyUOpijHvILWMG8J4+lwojuHWUVji/fZhdlaREKmw8AdDe3k53d3fJdt3d3f3m8fatk6RKMqKBN4SwBLie3Fq6p8cYt71GyUAv5o81hRP5qQ1bgNoQwowSNYXhil8P87MkKZEGe/hs6tSpHHXUURx55JHDqpOktBuxwBtCWAasAv6TXNh94QBu88788ekB5wtLn72/RM0ZA9pIUqoN3EBi4sSJAGzfvp3nn3+el156qd/5weokqVKMSOANIXye3ENqPwXeF2P8/X7anjRwu+H8+fcBV+TffnPA5Zvyx8+GEA7vUzMLuATYDdx2oP2XpCQJof9a66+++mrJdgPPD6yTpEpx0A+thRAWAyuAvcAPgctL/Kg+E2O8Pf+6BZgTQmgnt00w5FZpKKy1+/kYY7/FI2OM7SGEFqAZ+HkIYT25rYU/BhwBXDbYLmuSlDb333//AddddNFFI9wbSRr/RmKVhuPyxwnAkkHatAG3519/A1gAnEJuOsIhwPPAt4EbYow/LHWDGONnQgi/IDei+/dAL/AY8M8xxn87+K8hSckw2I5phx12GLW1tezcuZNsNjvkOklKuxBjLHcfRl1XV1f6v6SkivFXf/VXbN68ud+56upqjjzySKqqqpgwYQLbtm3rt3wZwHHHHcd//Md/jGVXJWnU1NXVDXme1mhvLSxJGmFVVX96DGLSpEns2bOHPXv28Pzzz/drN2HCBKqrq9m9e/c+dZJUSQy8kpQwL7/8p+XNC2EWcsuSFQLwSy+9xN69e9m7d2/JOkmqJAZeSUqYgfNza2pqWLNmDU888QQ9PT3MmjWLadOmcfnll9PT0zNonSRVCufwSlLCTJ8+nVdeeWXYdYceeijbtg13PyBJGp+GM4fXCV2SlDC1tbVjWidJSWfglaSEOeqoo8a0TpKSzsArSQnz+te/ftBr+1uJYX91kpRmBl5JSpj9bRHc29t7QHWSlGYGXklKmL4rL4xFnSQlnYFXkhLm8ccfH9M6SUo6A68kJcyBLEl2MHWSlHQGXklKmAOdi+scXkmVysArSQkzYcKEMa2TpKQz8EpSwuxvJYbRqJOkpDPwSlLC7N27t+T5ww47jDe84Q2D7qg2WJ0kpV11uTsgSRoZO3fuZOfOneXuhiSNO47wSlLC+NCaJA2PgVeSEmbixIljWidJSWfglaSEOeSQQ8a0TpKSzsArSQkz2ENpo1UnSUln4JWkhDnyyCPHtE6Sks7AK0kJ8/LLL49pnSQlnYFXkhLmpZdeGtM6SUo6A68kJcyBbiDhxhOSKpWBV5ISxmXJJGl4DLySlDDHH3/8mNZJUtIZeCUpYY499tgxrZOkpDPwSlLC/P73vx/TOklKOgOvJCXMk08+OaZ1kpR0Bl5JSphdu3aNaZ0kJZ2BV5ISZs+ePWNaJ0lJZ+CVpIRxHV5JGh4DryQlTIxxTOskKekMvJIkSUo1A68kJUwIYUzrJCnpDLySlDCHHHLImNZJUtIZeCUpYRzhlaThMfBKUsL09vaOaZ0kJZ2BV5ISpqrqwH66D7ROkpLOXz9JShinNEjS8Bh4JSlhXn311TGtk6SkM/BKUsK48YQkDY+BV5ISxsArScNj4JWkhPGhNUkaHn/9JClhDLySNDz++klSwjilQZKGx8ArSQnjsmSSNDwGXklKGEd4JWl4DLySlDAGXkkaHgOvJCWMgVeShsfAK0kJ4xxeSRoeA68kSZJS7aADbwjhyBDC34UQNoQQngoh7AohdIUQHgkh/G0IoeRnhBAaQgjfDSG8nK/5eQhhSQhhwn4+6wMhhIfz98+GEH4UQlh8sN9BkpLEEV5JGp7qEbjHR4CvAluB7wPPAkcBHwZuAc4IIXwk9pk8FkL4IPAd4BXgW8DLwNnAKqAxf89+QgiXAtcDLwHfBF4FzgVuDyH8RYxx6Qh8F0ka9yZMmMDevXsPqE6SKlE42IcYQgjvBWqA1hhjb5/z04EO4I3AuTHG7+TPTwGeAuqAxhjjT/LnDwUeAk4F/keMcV2fe80CngR6gLfHGJ/Jnz8c+DHwZ0BDjPHRUn3s6urySQ1JqfGmN72Jrq6uYdfV1dXxm9/8ZhR6JEljr66ubsj/2+qgpzTEGB+KMW7sG3bz57cBN+Xfntbn0rnANGBdIezm278CfC7/9lMDPuZvgEnADYWwm6/5A/Dl/NtPHtw3kaRk2LVr15jWSVLSjfZDa3/MH/f0Offe/PGBEu1/AOwEGkIIk4ZYc/+ANpKUan/84x9fu9EI1klS0o3EHN6SQgjVwPn5t32D6pvzx18PrIkx7gkhbAbmArOBzBBqtoYQeoCZIYTDYow7h9rHzs7OoTaVpHHjYNbh9XdPUtLMmTPnoO8xmiO8/wS8BfhujPHf+5yvyx8Hm4BWOD/1AGrqBrkuSZKkCjUqI7whhMuBz5B70GzRaHzGSBiJfzFI0lirrq5mz549r92wRJ2/e5Iq0YiP8OaXD1sNPAGcHmN8eUCT1xqNLZzffgA1w39sWZISZrApDVVVVRxyyCGDLj/m1sKSKtWIBt4QwhJya+U+Ti7sbivR7Ff54/El6quB48g95Pb0EGtmkFsW7bnhzN+VpKQabA3eEALV1dWDbjBxIGv3SlIajFjgDSEsI7dxxH+SC7svDNL0ofzx/SWuvQc4DGiPMe4eYs0ZA9pIUkXau3cvu3btOqDpDpKUZiMSeEMInyf3kNpPgffFGH+/n+brgd8DC0MIJ/e5x6HAF/Nvvzqg5jZgN3BpfhOKQs3hwFX5tzchSZIkDTASO60tBm4H9pKbzlBqHu0zMcbb+9R8iFzwfQVYR25r4XPILT+2HvhoHNCxEMJlwBpyWwt/iz9tLTwT+Jf9bS3sTmuS0mTq1Kn93ldVVdHb27tPu1Lnt2/fvk87SUqi4ey0NhKB9wvA1a/RrC3GeNqAukbgs+S2Ej6U3HbD/wqsiTGWnGgWQjgbWAqcRG50+glyu6/dsb8PN/BKGk8GBtYkMTBLGi/GNPAmgYFX0nhi4JWkgzecwDtqO61Jkko72NB48803s2zZsmHXrVy5kosuuuigPluSksgRXklKoAMZJXZ0VlKaDGeEdzS3FpYkjZLFixePantJShNHeCUpoebPn09HR8drtnvHO97Bv//7v49BjyRp7DjCK0kV4MEHH2Tx4sWDbiU8YcIEFi9ebNiVVPEc4ZWkFOj7INvcuXM5//zzfUBNUqq5LNkABl5JlaDwIJsPp0mqBE5pkCRJkvIMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdVGJPCGEM4NIVwfQvhhCKE7hBBDCN8cpO2s/PXB/tbt53MWhxA6QgjZEEJXCOHhEMIHRuI7SJIkKZ2qR+g+nwP+EsgCzwEnDKHmZ8A9Jc4/XqpxCOFa4DP5+38dmAgsBDaGEC6LMd5wAP2WJElSyo1U4L2CXBB9CmgCvj+Emv+MMX5hKDcPITSQC7v/BZwSY/xD/vw/Az8Frg0h/FuM8Znhd12SJElpNiJTGmKM348xdsYY40jcr4RP5o9fKoTd/Oc+A3wFmAR8YpQ+W5IkSQlWzofWjg4hXBRCuCp/fOt+2r43f3ygxLX7B7SRJEmSikZqSsOB+O/5v6IQwsPA4hjjs33O1QDHANkY49YS9+nMH48fpX5KkiQpwcoReHcC/0jugbWn8+feCnwBOB34XgjhbTHGnvy1uvyxa5D7Fc5PHW5HOjs7X7uRJCWMv22S0mTOnDkHfY8xn9IQY3whxrg8xvhYjHF7/u8HwHzgR8CfA3831v2SJElSOpVzSkM/McY9IYRbgHcA7wFW5y8VRnDrShb+6fz24X7mSPyLQZLGG3/bJKm/8bbT2ov5Y03hRH5qwxagNoQwo0RN4Zf916PcN0mSJCXQeAu878wfnx5w/qH88f0las4Y0EaSJEkqGvPAG0I4KYSwz+eGEN5HbgMLgIHbEt+UP342hHB4n5pZwCXAbuC2Ee+sJEmSEm9E5vCGED4EfCj/dnr+eGoI4fb869/HGJfmX7cAc0II7eR2Z4PcKg2FdXQ/H2Ns73v/GGN7CKEFaAZ+HkJYT25r4Y8BRwCXucuaJEmSSgkjsTlaCOELwNX7afKbGOOsfNu/BRYAbwFeDxwCPA88CtwQY/zhfj7nAnIjuicCvcBjwD/HGP9tf/3r6uoarR3gJGncmDo1tzrj9u3DfoZXkhKnrq4uDLXtiATe8c7AK6kSGHglVZLhBN7x9tCaJEmSNKIMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVDPwSpIkKdUMvJIkSUo1A68kSZJSzcArSZKkVKsudwckaTz56Ec/yoMPPljubhyUqVOnlrsLB2T+/Pl8+9vfLnc3JKXQiIzwhhDODSFcH0L4YQihO4QQQwjffI2ahhDCd0MIL4cQdoUQfh5CWBJCmLCfmg+EEB4OIXSFELIhhB+FEBaPxHeQJCDxYTfJ/G8vabSM1Ajv54C/BLLAc8AJ+2scQvgg8B3gFeBbwMvA2cAqoBH4SImaS4HrgZeAbwKvAucCt4cQ/iLGuHSEvosksX379nJ3Ydg6OzsBmDNnTpl7MnxJHZWWlAwjNYf3CuB4YArwqf01DCFMAb4O7AVOizH+bYzxH4C3AY8C54YQFg6omQVcSy4YnxxjvCTGeAXwVuC/gM+EEE4doe8iSZKkFBmRwBtj/H6MsTPGGIfQ/FxgGrAuxviTPvd4hdxIMewbmv8GmATcEGN8pk/NH4Av599+8gC7L0mSpBQrxyoN780fHyhx7QfATqAhhDBpiDX3D2gjSZIkFZVjlYY354+/HnghxrgnhLAZmAvMBjJDqNkaQugBZoYQDosx7hxqRwrz3SRpoCT/Pth3SWkyEs8llGOEty5/7BrkeuF83ycYhlpTN8h1SZIkVaiKXoc3iU8ySxobSfx9SPIqDQVJ7ruk8ascI7yvNRpbON93TaCh1gw2AixJkqQKVY7A+6v88fiBF0II1cBxwB7g6SHWzABqgOeGM39XkiRJlaEcgfeh/PH9Ja69BzgMaI8x7h5izRkD2kiSJElF5Qi864HfAwtDCCcXToYQDgW+mH/71QE1twG7gUvzm1AUag4Hrsq/vWmU+itJkqQEG5GH1kIIHwI+lH87PX88NYRwe/717wtb/8YYu0MIF5ILvg+HENaR20HtHHLLj60nt91wUYxxcwjhH4A1wE9CCN/iT1sLzwT+Jcb46Eh8F0mSJKXLSK3S8DZg8YBzs/N/AL8BlhYuxBjvCSE0AZ8F/ho4FHgKaAbWlNqxLcZ4fQjhmfx9zic3Ov0E8LkY4x0j9D0kSZKUMmFouwEnW1dXV/q/pKQRMXVqbgnw7du3v0bL8SfJy5Il+b+7pPKoq6sLQ21bjjm8kiRJ0pgx8EqSJCnVDLySJElKNQOvJEmSUs3AK0mSpFQz8EqSJCnVDLySJElKNQOvJEmSUs3AK0mSpFQz8EqSJCnVDLySJElKNQOvJEmSUs3AK0mSpFQz8EqSJCnVDLySJElKNQOvJEmSUs3AK0mSpFQz8EqSJCnVDLySJElKNQOvJEmSUs3AK0mSpFQz8EqSJCnVDLySJElKNQOvJEmSUs3AK0mSpFQz8EqSJCnVDLySJElKNQOvJEmSUs3AK0mSpFQz8EqSJCnVDLySJElKtRBjLHcfRl1XV1f6v6SkETF16lT2fPBt5e5GRaq+9z/Zvn17ubshKSHq6urCUNs6witJkqRUqy53ByRpvEnqSGNnZycAc+bMKXNPhm/q1Knl7oKkFHOEV5IkSalm4JUkSVKqGXglSZKUagZeSZIkpZqBV5IkSalm4JUkSVKqGXglSZKUagZeSZIkpZqBV5IkSalm4JUkSVKqGXglSZKUagZeSZIkpZqBV5IkSalm4JUkSVKqGXglSZKUagZeSZIkpZqBV5IkSalm4JUkSVKqlS3whhCeCSHEQf62DVLTEEL4bgjh5RDCrhDCz0MIS0IIE8a6/5IkSUqG6jJ/fhdwXYnz2YEnQggfBL4DvAJ8C3gZOBtYBTQCHxm9bkqSJCmpyh14t8cYv/BajUIIU4CvA3uB02KMP8mf/zzwEHBuCGFhjHHdaHZWkiRJyZOUObznAtOAdYWwCxBjfAX4XP7tp8rRMUmSJI1v5R7hnRRC+L+AY4Ee4OfAD2KMewe0e2/++ECJe/wA2Ak0hBAmxRh3j1pvJUmSlDjlDrzTgW8MOLc5hPCJGGNbn3Nvzh9/PfAGMcY9IYTNwFxgNpAZ6od3dnYOs7uSKkWSfx/su6Q0mTNnzkHfo5xTGm4D3kcu9NYAfwHcDMwC7g8h/GWftnX5Y9cg9yqcnzry3ZQkSVKSlW2EN8b4Pwecehz4ZAghC3wG+AKwYDT7MBL/YpCUTkn8fSiMjiax7wVJ7ruk8Ws8PrR2U/74nj7nCiO4dZRWOL99VHokSZKkxBqPgffF/LGmz7lf5Y/HD2wcQqgGjgP2AE+PbtckSZKUNOMx8L4zf+wbXh/KH99fov17gMOAdldokCRJ0kBlCbwhhPoQQk2J87OAG/Jvv9nn0nrg98DCEMLJfdofCnwx//aro9JZSZIkJVq5Hlr7GPCZEMIPgN8AO4A/A84CDgW+C1xbaBxj7A4hXEgu+D4cQlhHbmvhc8gtWbae3HbDkiRJUj/lCrzfJxdU/wpoJDdfdzvwCLl1eb8RY4x9C2KM94QQmoDPAn9NLhg/BTQDawa2l6SDMXWqqxxKUlqUJfDmN5Voe82G+9ZtAs4c+R5JUs727dsNu2Uyf/78cndBUkqVe6c1SRp3tm9P5gqHhaCe1P5L0mgZj6s0SJIkSSPGwCtJkqRUM/BKkiQp1Qy8kiRJSjUDryRJklLNwCtJkqRUM/BKkiQp1Qy8kiRJSjUDryRJklLNwCtJkqRUM/BKkiQp1Qy8kiRJSjUDryRJklLNwCtJkqRUM/BKkiQp1Qy8kiRJSjUDryRJklLNwCtJkqRUM/BKkiQp1Qy8kiRJSjUDryRJklLNwCtJkqRUM/BKkiQp1Qy8kiRJSjUDryRJklLNwCtJkqRUM/BKkiQp1Qy8kiRJSjUDryRJklLNwCtJkqRUM/BKkiQp1Qy8kiRJSjUDryRJklLNwCtJkqRUM/BKkiQp1Qy8kiRJSjUDryRJklLNwCtJkqRUM/BKkiQp1Qy8kiRJSjUDryRJklLNwCtJkqRUM/BKkiQp1Qy8kiRJSjUDryRJklLNwCtJkqRUM/BKkiQp1Qy8kiRJSrXqcndAkirN1KlTE3v/7du3j9q9JWm0OMIrSZKkVHOEV5LG2GiNknZ2dgIwZ86cUbm/JCVVokZ4QwgzQwj/GkL4XQhhdwjhmRDCdSGEw8vdN0mSJI1PiRnhDSH8GdAOvAG4F3gSmAd8Gnh/CKExxvhSGbsoSZKkcShJI7w3kgu7l8cYPxRj/L9jjO8FVgFvBr5U1t5JkiRpXEpE4M2P7s4HngG+MuDy1UAPsCiEUDPGXZMkSdI4l4jAC5yePz4YY+zteyHGuAPYBBwGvHOsOyZJkqTxLSlzeN+cP/56kOud5EaAjwe+N9SbFp5olqQ08bdNUpqMxMozSRnhrcsfuwa5Xjg/uqu5S5IkKXGSMsI7KlyrUlKauA6vJJWWlBHewghu3SDXC+fd81KSJEn9JCXw/ip/PH6Q64XhjMHm+EqSJKlCJSXwfj9/nB9C6NfnEMJkoPH/b++OdWwIoDgO/0+ik9hWuYlEdBoJai+g8AJKVEJHgkfAC0h4Co1C4w0kEoVQUqygPYpdCoUIsTNz9vuaOzczxbndLydzZ5J8S/LqsAcDAGDdNhG83f02yfMku0lu/HL6QZLjSZ5299dDHg0AgJXb0p/Wrmf/1cKPqupSktdJzmf/Gb1vktxZcDYAAFZqExve5OeW91ySJ9kP3VtJTiV5mORCd39abjoAANZqSxvedPf7JFeXngMAgO3YzIYXAAD+huAFAGA0wQsAwGiCFwCA0QQvAACjVXcvPcN/t7e3N/9HAgAcITs7O/Wn19rwAgAwmuAFAGA0wQsAwGiCFwCA0QQvAACjHYmnNAAAcHTZ8AIAMJrgBQBgNMELAMBoghcAgNEEL8CGVdWVqnpcVS+r6nNVdVU9W3ougDU5tvQAAPyTu0nOJvmS5EOSM8uOA7A+NrwA23YzyekkJ5JcW3gWgFWy4QXYsO5+8eO4qpYcBWC1bHgBABhN8AIAMJrgBQBgNMELAMBoghcAgNEELwAAowleAABGE7wAAIzmxRMAG1ZVl5NcPvh68uDzYlU9OTj+2N23D30wgBWp7l56BgD+UlXdT3LvN5e86+7dw5kGYJ0ELwAAo7mHFwCA0QQvAACjCV4AAEYTvAAAjCZ4AQAYTfACADCa4AUAYDTBCwDAaIIXAIDRBC8AAKMJXgAARhO8AACMJngBABhN8AIAMJrgBQBgNMELAMBoghcAgNG+A4TuN+dKxjgTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 299,
       "width": 350
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#overall distribution of length of strings in each entry.\n",
    "df['pre_clean_len'] = [len(t) for t in df.text]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(5, 5))\n",
    "plt.boxplot(df.pre_clean_len)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inference:\n",
    "***\n",
    "- From the above box plot, some of the tweets are more than 140 chracters long."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>text</th>\n",
       "      <th>pre_clean_len</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>213</th>\n",
       "      <td>0</td>\n",
       "      <td>Awwh babs... you look so sad underneith that s...</td>\n",
       "      <td>142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>226</th>\n",
       "      <td>0</td>\n",
       "      <td>Tuesdayï¿½ll start with reflection ï¿½n then a...</td>\n",
       "      <td>141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>279</th>\n",
       "      <td>0</td>\n",
       "      <td>Whinging. My client&amp;amp;boss don't understand ...</td>\n",
       "      <td>145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>343</th>\n",
       "      <td>0</td>\n",
       "      <td>@TheLeagueSF Not Fun &amp;amp; Furious? The new ma...</td>\n",
       "      <td>145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>400</th>\n",
       "      <td>0</td>\n",
       "      <td>#3 woke up and was having an accident - &amp;quot;...</td>\n",
       "      <td>144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>464</th>\n",
       "      <td>0</td>\n",
       "      <td>My bathtub drain is fired: it haz 1 job 2 do, ...</td>\n",
       "      <td>146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>492</th>\n",
       "      <td>0</td>\n",
       "      <td>pears &amp;amp; Brie, bottle of Cabernet, and &amp;quo...</td>\n",
       "      <td>150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>747</th>\n",
       "      <td>0</td>\n",
       "      <td>Have an invite for &amp;quot;Healthy Dining&amp;quot; ...</td>\n",
       "      <td>141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>957</th>\n",
       "      <td>0</td>\n",
       "      <td>Damnit I was really digging this season of Rea...</td>\n",
       "      <td>141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1064</th>\n",
       "      <td>0</td>\n",
       "      <td>Why do I keep looking...I know that what I rea...</td>\n",
       "      <td>141</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      sentiment                                               text  \\\n",
       "213           0  Awwh babs... you look so sad underneith that s...   \n",
       "226           0  Tuesdayï¿½ll start with reflection ï¿½n then a...   \n",
       "279           0  Whinging. My client&amp;boss don't understand ...   \n",
       "343           0  @TheLeagueSF Not Fun &amp; Furious? The new ma...   \n",
       "400           0  #3 woke up and was having an accident - &quot;...   \n",
       "464           0  My bathtub drain is fired: it haz 1 job 2 do, ...   \n",
       "492           0  pears &amp; Brie, bottle of Cabernet, and &quo...   \n",
       "747           0  Have an invite for &quot;Healthy Dining&quot; ...   \n",
       "957           0  Damnit I was really digging this season of Rea...   \n",
       "1064          0  Why do I keep looking...I know that what I rea...   \n",
       "\n",
       "      pre_clean_len  \n",
       "213             142  \n",
       "226             141  \n",
       "279             145  \n",
       "343             145  \n",
       "400             144  \n",
       "464             146  \n",
       "492             150  \n",
       "747             141  \n",
       "957             141  \n",
       "1064            141  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.pre_clean_len > 140].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PRE-PROCESSING:\n",
    "*****\n",
    "- Replacing HTML characters like '&amp','&quot',etc\n",
    "- Removing @\n",
    "- Removing URL links\n",
    "- Replacing # with blanks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"pears &amp; Brie, bottle of Cabernet, and &quot;Win a Date With Tad Hamilton&quot;... oh gawwd my life flashed forward to when I'm 40 with my 75 cats \""
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.text[492]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pears & Brie, bottle of Cabernet, and \"Win a Date With Tad Hamilton\"... oh gawwd my life flashed forward to when I'm 40 with my 75 cats \n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "x = BeautifulSoup(df.text[492], 'lxml')\n",
    "print (x.get_text())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'@TheLeagueSF Not Fun &amp; Furious? The new mantra for the Bay 2 Breakers? It was getting 2 rambunctious;the city overreacted &amp; clamped down '"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.text[343]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' Not Fun &amp; Furious? The new mantra for the Bay 2 Breakers? It was getting 2 rambunctious;the city overreacted &amp; clamped down '"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "re.sub(r'@[A-Za-z0-9]+','',df.text[343])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"@switchfoot http://twitpic.com/2y1zl - Awww, that's a bummer.  You shoulda got David Carr of Third Day to do it. ;D\""
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.text[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"@switchfoot  - Awww, that's a bummer.  You shoulda got David Carr of Third Day to do it. ;D\""
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.sub('https?://[A-Za-z0-9./]+','',df.text[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"@machineplay I'm so sorry you're having to go through this. Again.  #therapyfail\""
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.text[175]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' machineplay I m so sorry you re having to go through this  Again    therapyfail'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.sub(\"[^a-zA-Z]\", \" \", df.text[175])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapper Function for Cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import WordPunctTokenizer\n",
    "tok = WordPunctTokenizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "pat1 = r'@[A-Za-z0-9]+'\n",
    "pat2 = r'https?://[A-Za-z0-9./]+'\n",
    "combined_pat = r'|'.join((pat1, pat2))\n",
    "\n",
    "def tweet_cleaner(text):\n",
    "    soup = BeautifulSoup(text, 'lxml')\n",
    "    souped = soup.get_text()\n",
    "    stripped = re.sub(combined_pat, '', souped)\n",
    "    try:\n",
    "        clean = stripped.decode(\"utf-8-sig\").replace(u\"\\ufffd\", \"?\")\n",
    "    except:\n",
    "        clean = stripped\n",
    "    letters_only = re.sub(\"[^a-zA-Z]\", \" \", clean)\n",
    "    lower_case = letters_only.lower()\n",
    "    # During the letters_only process two lines above, it has created unnecessay white spaces,\n",
    "    # I will tokenize and join together to remove unneccessary white spaces\n",
    "    words = tok.tokenize(lower_case)\n",
    "    return (\" \".join(words)).strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['awww that s a bummer you shoulda got david carr of third day to do it d',\n",
       " 'is upset that he can t update his facebook by texting it and might cry as a result school today also blah',\n",
       " 'i dived many times for the ball managed to save the rest go out of bounds',\n",
       " 'my whole body feels itchy and like its on fire',\n",
       " 'no it s not behaving at all i m mad why am i here because i can t see you all over there',\n",
       " 'not the whole crew',\n",
       " 'need a hug',\n",
       " 'hey long time no see yes rains a bit only a bit lol i m fine thanks how s you',\n",
       " 'k nope they didn t have it',\n",
       " 'que me muera',\n",
       " 'spring break in plain city it s snowing',\n",
       " 'i just re pierced my ears',\n",
       " 'i couldn t bear to watch it and i thought the ua loss was embarrassing',\n",
       " 'it it counts idk why i did either you never talk to me anymore',\n",
       " 'i would ve been the first but i didn t have a gun not really though zac snyder s just a doucheclown',\n",
       " 'i wish i got to watch it with you i miss you and how was the premiere',\n",
       " 'hollis death scene will hurt me severely to watch on film wry is directors cut not out now',\n",
       " 'about to file taxes',\n",
       " 'ahh ive always wanted to see rent love the soundtrack',\n",
       " 'oh dear were you drinking out of the forgotten table drinks',\n",
       " 'i was out most of the day so didn t get much done',\n",
       " 'one of my friend called me and asked to meet with her at mid valley today but i ve no time sigh',\n",
       " 'barista i baked you a cake but i ated it',\n",
       " 'this week is not going as i had hoped',\n",
       " 'blagh class at tomorrow',\n",
       " 'i hate when i have to call and wake people up',\n",
       " 'just going to cry myself to sleep after watching marley and me',\n",
       " 'im sad now miss lilly',\n",
       " 'ooooh lol that leslie and ok i won t do it again so leslie won t get mad again',\n",
       " 'meh almost lover is the exception this track gets me depressed every time',\n",
       " 'some hacked my account on aim now i have to make a new one',\n",
       " 'i want to go to promote gear and groove but unfornately no ride there i may b going to the one in anaheim in may though',\n",
       " 'thought sleeping in was an option tomorrow but realizing that it now is not evaluations in the morning and work in the afternoon',\n",
       " 'awe i love you too am here i miss you',\n",
       " 'i cry my asian eyes to sleep at night',\n",
       " 'ok i m sick and spent an hour sitting in the shower cause i was too sick to stand and held back the puke like a champ bed now',\n",
       " 'ill tell ya the story later not a good day and ill be workin for like three more hours',\n",
       " 'sorry bed time came here gmt',\n",
       " 'i don t either its depressing i don t think i even want to know about the kids in suitcases',\n",
       " 'bed class work gym or then class another day that s gonna fly by i miss my girlfriend',\n",
       " 'really don t feel like getting up today but got to study to for tomorrows practical exam',\n",
       " 'he s the reason for the teardrops on my guitar the only one who has enough of me to break my heart',\n",
       " 'sad sad sad i don t know why but i hate this feeling i wanna sleep and i still can t',\n",
       " 'awww i soo wish i was there to see you finally comfortable im sad that i missed it',\n",
       " 'falling asleep just heard about that tracy girl s body being found how sad my heart breaks for that family',\n",
       " 'yay i m happy for you with your job but that also means less time for me and you',\n",
       " 'just checked my user timeline on my blackberry it looks like the twanking is still happening are ppl still having probs w bgs and uids',\n",
       " 'oh man was ironing s fave top to wear to a meeting burnt it',\n",
       " 'is strangely sad about lilo and samro breaking up',\n",
       " 'oh i m so sorry i didn t think about that before retweeting',\n",
       " 'broadband plan a massive broken promise via www diigo com tautao still waiting for broadband we are',\n",
       " 'wow tons of replies from you may have to unfollow so i can see my friends tweets you re scrolling the feed a lot',\n",
       " 'our duck and chicken are taking wayyy too long to hatch',\n",
       " 'put vacation photos online a few yrs ago pc crashed and now i forget the name of the site',\n",
       " 'i need a hug',\n",
       " 'not sure what they are only that they are pos as much as i want to i dont think can trade away company assets sorry andy',\n",
       " 'i hate when that happens',\n",
       " 'i have a sad feeling that dallas is not going to show up i gotta say though you d think more shows would use music from the game mmm',\n",
       " 'ugh degrees tomorrow',\n",
       " 'where did u move to i thought u were already in sd hmmm random u found me glad to hear yer doing well',\n",
       " 'i miss my ps it s out of commission wutcha playing have you copped blood on the sand',\n",
       " 'just leaving the parking lot of work',\n",
       " 'the life is cool but not for me',\n",
       " 'sadly though i ve never gotten to experience the post coitus cigarette before and now i never will',\n",
       " 'i had such a nice day too bad the rain comes in tomorrow at am',\n",
       " 'too bad i won t be around i lost my job and can t even pay my phone bill lmao aw shucks',\n",
       " 'damm back to school tomorrow',\n",
       " 'mo jobs no money how in the hell is min wage here f n clams an hour',\n",
       " 'not forever see you soon',\n",
       " 'algonquin agreed i saw the failwhale allllll day today',\n",
       " 'oh haha dude i dont really look at em unless someone says hey i added you sorry i m so terrible at that i need a pop up',\n",
       " 'i m sure you re right i need to start working out with you and the nikster or jared at least',\n",
       " 'i really hate how people diss my bands trace is clearly not ugly',\n",
       " 'gym attire today was puma singlet adidas shorts and black business socks and leather shoes lucky did not run into any cute girls',\n",
       " 'why won t you show my location',\n",
       " 'no picnic my phone smells like citrus',\n",
       " 'my donkey is sensitive about such comments nevertheless he d and me d be glad to see your mug asap charger is still awol',\n",
       " 'no new csi tonight fml',\n",
       " 'i think my arms are sore from tennis',\n",
       " 'wonders why someone that u like so much can make you so unhappy in a split seccond depressed',\n",
       " 'sleep soon i just hate saying bye and see you tomorrow for the night',\n",
       " 'just got ur newsletter those fares really are unbelievable shame i already booked and paid for mine',\n",
       " 'missin the boo',\n",
       " 'me too itm',\n",
       " 'damn i don t have any chalk my chalkboard is useless',\n",
       " 'had a blast at the getty villa but hates that she s had a sore throat all day it s just getting worse too',\n",
       " 'hey missed ya at the meeting sup mama',\n",
       " 'my tummy hurts i wonder if the hypnosis has anything to do with it if so it s working i get it stop smoking',\n",
       " 'why is it always the fat ones',\n",
       " 'sorry babe my fam annoys me too thankfully they re asleep right now muahaha evil laugh',\n",
       " 'i should have paid more attention when we covered photoshop in my webpage design class in undergrad',\n",
       " 'wednesday my b day don t know what do',\n",
       " 'poor cameron the hills',\n",
       " 'pray for me please the ex is threatening to start sh at my our babies st birthday party what a jerk and i still have a headache',\n",
       " 'hmm do u really enjoy being with him if the problems are too constants u should think things more find someone ulike',\n",
       " 'strider is a sick little puppy',\n",
       " 'so rylee grace wana go steve s party or not sadly since its easter i wnt b able do much but ohh well',\n",
       " 'hey i actually won one of my bracket pools too bad it wasn t the one for money',\n",
       " 'you don t follow me either and i work for you',\n",
       " 'a bad nite for the favorite teams astros and spartans lose the nite out with t w was good']"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample = []\n",
    "for t in df.text[:100]:\n",
    "    sample.append(tweet_cleaner(t))\n",
    "\n",
    "sample"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}