proc_17_108_analysis_02_HDBScan_Clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "Straightforward notebook. Apply HDBSCAN to 100,000 row subset and save the model. I've provided the data w/ vectorized sample on kaggle.\n",
    "\n",
    "Note that algorithmic clusters do not necessarily match up perfectly with human intuitions of 'clusters'. A more precise way of thinking about purpose of this notebook and next couple is that we are trying to separate comments similar enough to be from the same campaign, and comments that are not. Thus a campaign may be broken up into several smaller clusters.\n",
    "\n",
    "### Additional Notes\n",
    "\n",
    "* I did not re-run this on the entire 100,000 datapoints as fit_predict() takes a few hours to finish running.\n",
    "* Like most clustering algorithms, HDBSCAN does not scale well with dimensionality or sample size - I tried PCA on the doc vecs but it gave me very muddy results\n",
    "* My intuition is that for this step if one had train the word vectors on the corpus using word2vec or use another encoding method with fewer dimensions you might be able to scale up HDBSCAN better. The issue with other encoding methods, however, is that many require taking the entire corpus into memory, which is not feasible on my desktop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.preprocessing import normalize\n",
    "import hdbscan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6.2 s, sys: 188 ms, total: 6.38 s\n",
      "Wall time: 6.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "## import a sample of the comments we didn't cull out from the original, vectorized\n",
    "X = pd.read_csv('proc_17_108_copy_uniques_level0_unclustered_sample_vectorized.csv', index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_sample = X.sample(20000) #try a small subset to see it working\n",
    "# X_sample = X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### hdbscan doesn't support cosine distance so let's use angular distance instead (euclidean of normalized vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = X_sample.iloc[:,-300:]\n",
    "norm_data = normalize(data, norm='l2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "clusterer = hdbscan.HDBSCAN(min_cluster_size=10, prediction_data=True) # would want to scale min_cluster_size with your sample size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6min 51s, sys: 208 ms, total: 6min 51s\n",
      "Wall time: 6min 51s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# takes a few hours for all 100,000 data points\n",
    "dbscan = clusterer.fit_predict(norm_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dbscan.max() #make sure it's clustering -- can do some more checking by glomming back onto the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "joblib.dump(clusterer, 'level1-clusterer.pkl') "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}