Logistic_Regression_Consulting_Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression Consulting Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Binary Customer Churn\n",
    "\n",
    "A marketing agency has many customers that use their service to produce ads for the client/customer websites. They've noticed that they have quite a bit of churn in clients. They basically randomly assign account managers right now, but want you to create a machine learning model that will help predict which customers will churn (stop buying their service) so that they can correctly assign the customers most at risk to churn an account manager. Luckily they have some historical data.\n",
    "\n",
    "Create a classification algorithm that will help classify whether or not a customer churned. Then the company can test this against incoming data for future customers to predict which customers will churn and assign them an account manager.\n",
    "\n",
    "The data is saved as customer_churn.csv. Here are the fields and their definitions:\n",
    "\n",
    "    Name : Name of the latest contact at Company\n",
    "    Age: Customer Age\n",
    "    Total_Purchase: Total Ads Purchased\n",
    "    Account_Manager: Binary 0=No manager, 1= Account manager assigned  (NB. currently it is randomely assigned!)\n",
    "    Years: Totaly Years as a customer\n",
    "    Num_sites: Number of websites that use the service.\n",
    "    Onboard_date: Date that the name of the latest contact was onboarded\n",
    "    Location: Client HQ Address\n",
    "    Company: Name of Client Company\n",
    "    \n",
    "Once you've created the model and evaluated it, test out the model on some new data (you can think of this almost like a hold-out set) that your client has provided, saved under new_customers.csv. The client wants to know which customers are most likely to churn given this data (they don't have the label yet).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession\n",
    "spark = SparkSession.builder.appName('MLlibLogisticRegProject').getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = spark.read.csv('customer_churn.csv', header = True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(Names='Cameron Williams', Age=42.0, Total_Purchase=11066.8, Account_Manager=0, Years=7.22, Num_Sites=8.0, Onboard_date='2013-08-30 07:00:40', Location='10265 Elizabeth Mission Barkerburgh, AK 89518', Company='Harvey LLC', Churn=1)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Names: string (nullable = true)\n",
      " |-- Age: double (nullable = true)\n",
      " |-- Total_Purchase: double (nullable = true)\n",
      " |-- Account_Manager: integer (nullable = true)\n",
      " |-- Years: double (nullable = true)\n",
      " |-- Num_Sites: double (nullable = true)\n",
      " |-- Onboard_date: string (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      " |-- Company: string (nullable = true)\n",
      " |-- Churn: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+---+--------------+---------------+-----+---------+------------+--------+-------+-----+------+\n",
      "|Names|Age|Total_Purchase|Account_Manager|Years|Num_Sites|Onboard_date|Location|Company|Churn|Weight|\n",
      "+-----+---+--------------+---------------+-----+---------+------------+--------+-------+-----+------+\n",
      "|    0|  0|             0|              0|    0|        0|           0|       0|      0|    0|     0|\n",
      "+-----+---+--------------+---------------+-----+---------+------------+--------+-------+-----+------+\n",
      "\n",
      "+-----+---+--------------+---------------+-----+---------+------------+--------+-------+-----+------+\n",
      "|Names|Age|Total_Purchase|Account_Manager|Years|Num_Sites|Onboard_date|Location|Company|Churn|Weight|\n",
      "+-----+---+--------------+---------------+-----+---------+------------+--------+-------+-----+------+\n",
      "|    0|  0|             0|              0|    0|        0|           0|       0|      0|    0|     0|\n",
      "+-----+---+--------------+---------------+-----+---------+------------+--------+-------+-----+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Count null/nan values for all columns:\n",
    "from pyspark.sql.functions import isnan, isnull, when, count, col\n",
    "\n",
    "data.select([count(when(isnull(c), c)).alias(c) for c in data.columns]).show()\n",
    "data.select([count(when(isnan(c), c)).alias(c) for c in data.columns]).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring the feature importance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------+-----+\n",
      "|       Company|count|\n",
      "+--------------+-----+\n",
      "|    Wilson PLC|    3|\n",
      "|Anderson Group|    4|\n",
      "|  Williams PLC|    3|\n",
      "+--------------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df = data.groupby('Company').count()\n",
    "df.filter(df['count']>2).show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'Company' name does not seem important, since it has not been repeated frequently.\n",
    "\n",
    "'Location' is the same. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------+-----+\n",
      "|        Names|count|\n",
      "+-------------+-----+\n",
      "|Jennifer Wood|    2|\n",
      "+-------------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df = data.groupby('Names').count()\n",
    "df.filter(df['count']>1).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'Names' is not repeated, so not important.\n",
    "\n",
    "'Account_Manager' is assigned randomely, so not important\n",
    "\n",
    "#### In total, five features seem to be informative:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+-----------------+-----------------+-----------------+------------------+-------------------+\n",
      "|summary|              Age|   Total_Purchase|            Years|         Num_sites|       Onboard_date|\n",
      "+-------+-----------------+-----------------+-----------------+------------------+-------------------+\n",
      "|  count|              900|              900|              900|               900|                900|\n",
      "|   mean|41.81666666666667|10062.82403333334| 5.27315555555555| 8.587777777777777|               null|\n",
      "| stddev|6.127560416916251|2408.644531858096|1.274449013194616|1.7648355920350969|               null|\n",
      "|    min|             22.0|            100.0|              1.0|               3.0|2006-01-02 04:16:13|\n",
      "|    max|             65.0|         18026.01|             9.15|              14.0|2016-12-28 04:07:38|\n",
      "+-------+-----------------+-----------------+-----------------+------------------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.describe().select('summary','Age','Total_Purchase','Years',\n",
    "                       'Num_sites','Onboard_date',).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imbalanced classes:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----+-----+\n",
      "|Churn|count|\n",
      "+-----+-----+\n",
      "|    1|  150|\n",
      "|    0|  750|\n",
      "+-----+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.groupby('Churn').count().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classes are imbalance. 150 samples are available for class 1, while 750 classes for label 0.\n",
    "\n",
    "Ratio is 150/750 = 0.2.  So, 0.2 of samples belong to the class 1 and 0.8 of samples belong to the class 0.\n",
    "\n",
    "There are different methods to compensate for class imbalance issue. One way is to under sample the majority class. Another way is to assign weights (as **weightCol in LogisticRegression**) for each class to penalize the majority class by assigning less weight and boost the minority class by assigning higher weight. \n",
    "\n",
    "NB. weightCol should only affect the model training step. No effect on test time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import when\n",
    "\n",
    "ratio = 0.2\n",
    "def weightBalance(label):\n",
    "    return when(label == 1, 1-ratio).otherwise(ratio)\n",
    "\n",
    "data = data.withColumn('Weight', weightBalance(data['Churn']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess on 'Onboard_date' feature:\n",
    "\n",
    "First, convert from string to datetime.datetime format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+\n",
      "|       Onboard_date|\n",
      "+-------------------+\n",
      "|2013-08-30 07:00:40|\n",
      "|2013-08-13 00:38:46|\n",
      "|2016-06-29 06:20:07|\n",
      "|2014-04-22 12:43:12|\n",
      "|2016-01-19 15:31:15|\n",
      "|2009-03-03 23:13:37|\n",
      "|2016-12-05 03:35:43|\n",
      "|2006-03-09 14:50:20|\n",
      "|2011-09-29 05:47:23|\n",
      "|2006-03-28 15:42:45|\n",
      "|2016-11-13 13:13:01|\n",
      "|2015-05-28 12:14:03|\n",
      "|2011-02-16 08:10:47|\n",
      "|2012-11-22 05:35:03|\n",
      "|2015-03-28 02:13:44|\n",
      "|2015-07-22 08:38:40|\n",
      "|2006-09-03 06:13:55|\n",
      "|2006-10-22 04:42:38|\n",
      "|2015-10-07 00:27:10|\n",
      "|2014-11-06 23:47:14|\n",
      "+-------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.select('Onboard_date').show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import to_timestamp\n",
    "\n",
    "dfdate = data.withColumn('Date', to_timestamp(data['Onboard_date'],'yyyy-MM-dd HH:mm:ss'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Names: string (nullable = true)\n",
      " |-- Age: double (nullable = true)\n",
      " |-- Total_Purchase: double (nullable = true)\n",
      " |-- Account_Manager: integer (nullable = true)\n",
      " |-- Years: double (nullable = true)\n",
      " |-- Num_Sites: double (nullable = true)\n",
      " |-- Onboard_date: string (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      " |-- Company: string (nullable = true)\n",
      " |-- Churn: integer (nullable = true)\n",
      " |-- Weight: double (nullable = false)\n",
      " |-- Date: timestamp (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "dfdate.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, compute **delta (time difference) from today** to the Onboard_date ('Date' column):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "#dt = datetime.datetime.today()\n",
    "dt = datetime.datetime(2020, 7, 9, 11, 12, 37, 597168)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Row(Date=datetime.datetime(2013, 8, 30, 7, 0, 40))]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dfdate.select('Date').head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Register the python function as udf:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyspark.sql.functions as F\n",
    "dtDeltafn = F.udf(lambda x: (dt-x).days)\n",
    "dfDur = dfdate.withColumn('Duration_date', dtDeltafn(dfdate['Date']).cast('int'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------+----+--------------+-----+---------+------+-----+\n",
      "|Duration_date| Age|Total_Purchase|Years|Num_sites|Weight|label|\n",
      "+-------------+----+--------------+-----+---------+------+-----+\n",
      "|         2505|42.0|       11066.8| 7.22|      8.0|   0.8|    1|\n",
      "|         2522|41.0|      11916.22|  6.5|     11.0|   0.8|    1|\n",
      "|         1471|38.0|      12884.75| 6.67|     12.0|   0.8|    1|\n",
      "|         2269|42.0|       8010.76| 6.71|     10.0|   0.8|    1|\n",
      "|         1632|37.0|       9191.58| 5.56|      9.0|   0.8|    1|\n",
      "|         4145|48.0|      10356.02| 5.12|      8.0|   0.8|    1|\n",
      "|         1312|44.0|      11331.58| 5.23|     11.0|   0.8|    1|\n",
      "|         5235|32.0|       9885.12| 6.92|      9.0|   0.8|    1|\n",
      "|         3206|43.0|       14062.6| 5.46|     11.0|   0.8|    1|\n",
      "|         5216|40.0|       8066.94| 7.11|     11.0|   0.8|    1|\n",
      "|         1333|30.0|      11575.37| 5.22|      8.0|   0.8|    1|\n",
      "|         1868|45.0|       8771.02| 6.64|     11.0|   0.8|    1|\n",
      "|         3431|45.0|       8988.67| 4.84|     11.0|   0.8|    1|\n",
      "|         2786|40.0|       8283.32|  5.1|     13.0|   0.8|    1|\n",
      "|         1930|41.0|       6569.87|  4.3|     11.0|   0.8|    1|\n",
      "|         1814|38.0|      10494.82| 6.81|     12.0|   0.8|    1|\n",
      "|         5058|45.0|       8213.41| 7.35|     11.0|   0.8|    1|\n",
      "|         5009|43.0|      11226.88| 8.08|     12.0|   0.8|    1|\n",
      "|         1737|53.0|       5515.09| 6.85|      8.0|   0.8|    1|\n",
      "|         2071|46.0|        8046.4| 5.69|      8.0|   0.8|    1|\n",
      "+-------------+----+--------------+-----+---------+------+-----+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "featureColumns = ['Duration_date','Age','Total_Purchase','Years','Num_sites']\n",
    "df = dfDur.withColumnRenamed('Churn','label').select(featureColumns+['Weight','label'])\n",
    "\n",
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------------------+-----------------+-----------------+-----------------+------------------+-------------------+-------------------+\n",
      "|summary|     Duration_date|              Age|   Total_Purchase|            Years|         Num_sites|             Weight|              label|\n",
      "+-------+------------------+-----------------+-----------------+-----------------+------------------+-------------------+-------------------+\n",
      "|  count|               900|              900|              900|              900|               900|                900|                900|\n",
      "|   mean|3376.3077777777776|41.81666666666667|10062.82403333334| 5.27315555555555| 8.587777777777777|0.29999999999999083|0.16666666666666666|\n",
      "| stddev| 1171.897908168496|6.127560416916251|2408.644531858096|1.274449013194616|1.7648355920350969|0.22373112736634135| 0.3728852122772358|\n",
      "|    min|              1289|             22.0|            100.0|              1.0|               3.0|                0.2|                  0|\n",
      "|    max|              5302|             65.0|         18026.01|             9.15|              14.0|                0.8|                  1|\n",
      "+-------+------------------+-----------------+-----------------+-----------------+------------------+-------------------+-------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.describe().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### VectorAssembler:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import VectorAssembler\n",
    "assembler = VectorAssembler(inputCols = featureColumns,\n",
    "                            outputCol = 'featuresAssem')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalizing features:\n",
    "Normalizing each feature to have unit standard deviation and zero mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import StandardScaler\n",
    "scaler = StandardScaler(inputCol=\"featuresAssem\", outputCol=\"features\",\n",
    "                        withStd=True, withMean=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a LogisticRegression instance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DataFrame[Duration_date: int, Age: double, Total_Purchase: double, Years: double, Num_sites: double, Weight: double, label: int]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyspark.ml.classification import LogisticRegression\n",
    "logr = LogisticRegression(featuresCol='features',labelCol='label', weightCol='Weight')\n",
    "\n",
    "df.withColumnRenamed('Churn','label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipline creation and defining the stages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml import Pipeline\n",
    "pipeline = Pipeline(stages=[assembler,scaler,logr])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train/test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = df.randomSplit([0.7,.3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.evaluation import BinaryClassificationEvaluator\n",
    "from pyspark.ml.tuning import CrossValidator, ParamGridBuilder\n",
    "\n",
    "paramGrid = ParamGridBuilder() \\\n",
    "    .addGrid(logr.regParam, [0, 0.1, 0.01])\\\n",
    "    .build()\n",
    "\n",
    "crossval = CrossValidator(estimator=pipeline,\n",
    "                          estimatorParamMaps=paramGrid,\n",
    "                          evaluator=BinaryClassificationEvaluator(),\n",
    "                          numFolds=3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run cross-validation, and choose the best set of parameters.\n",
    "cvModel = crossval.fit(train)\n",
    "#cvModel includes the best model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transform the test data\n",
    "results = cvModel.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.811113804387347"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_eval = BinaryClassificationEvaluator(rawPredictionCol='prediction',\n",
    "                                       labelCol='label', metricName=\"areaUnderROC\" )\n",
    "AUC = my_eval.evaluate(results)\n",
    "AUC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To find the best set of params:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "bestPipeline = cvModel.bestModel\n",
    "bestLRModel = bestPipeline.stages[2]\n",
    "bestParams = bestLRModel.extractParamMap()\n",
    "\n",
    "#bestParams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict on a brand new unlabeled data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_newCust = spark.read.csv('new_customers.csv', header = True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- Names: string (nullable = true)\n",
      " |-- Age: double (nullable = true)\n",
      " |-- Total_Purchase: double (nullable = true)\n",
      " |-- Account_Manager: integer (nullable = true)\n",
      " |-- Years: double (nullable = true)\n",
      " |-- Num_Sites: double (nullable = true)\n",
      " |-- Onboard_date: string (nullable = true)\n",
      " |-- Location: string (nullable = true)\n",
      " |-- Company: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data_newCust.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfDate_newCust = data_newCust.withColumn('Date', to_timestamp(data_newCust['Onboard_date'],'yyyy-MM-dd HH:mm:ss'))\n",
    "\n",
    "dfDur_newCust = dfDate_newCust.withColumn('Duration_date', dtDeltafn(dfDate_newCust['Date']).cast('int'))\n",
    "\n",
    "df_newCust = dfDur_newCust.select(featureColumns+['Company'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_newCust = cvModel.transform(df_newCust)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------+--------------------+----------+\n",
      "|         Company|         probability|prediction|\n",
      "+----------------+--------------------+----------+\n",
      "|        King Ltd|[0.61001946597518...|       0.0|\n",
      "|   Cannon-Benson|[0.03661577428857...|       1.0|\n",
      "|Barron-Robertson|[0.09110822343468...|       1.0|\n",
      "|   Sexton-Golden|[0.04535616983946...|       1.0|\n",
      "|        Wood LLC|[0.49787560005344...|       1.0|\n",
      "|   Parks-Robbins|[0.25682973707624...|       1.0|\n",
      "+----------------+--------------------+----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "results_newCust.select('Company','probability','prediction').show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above results show that we should not assign an Acocunt Manager to King Ltd Company."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}