unidad2/Notebook2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unidad 2: Taller de análisis de datos textuales\n",
    "\n",
    "<h1> Notebook 2 - Topic Modeling</h1>\n",
    "\n",
    "- El modelamiento de tópicos (__Topic Modeling__) es una técnica para extraer automáticamente los tópicos ocultos de grandes volúmenes de texto. Esta técnica busca responder a la pregunta: ¿cuáles son los tópicos que se hablan en un conjunto de textos?\n",
    "\n",
    "- __Latent Dirichlet Allocation__ (LDA) es un algoritmo popular para _topic modeling_ con buenas implementaciones en el paquete Gensim de Python. \n",
    "\n",
    "- Un desafío metodológico consiste en extraer __tópicos coherentes y significativos__. Esto depende en gran medida de la calidad del preprocesamiento del texto y de la estrategia de __encontrar el número óptimo de tópicos__. Este taller propone abordar algunas buenas prácticas para resolver estos problemas.\n",
    "\n",
    "El taller tiene 2 partes, una parte guiada que ilustra cómo utilizar LDA para extraer tópicos en un dataset de discusiones de foros. La segundo parte es un trabajo práctico que apunta a reproducir una metodología de extracción de tópicos con LDA.\n",
    "\n",
    "## Parte I - ¿Cómo aplicar LDA para extraer tópicos de un dataset de textos?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Introducción\n",
    "\n",
    "Una de las principales aplicaciones del procesamiento del lenguaje natural es extraer automáticamente los temas o tópicos que la gente está discutiendo de grandes volúmenes de texto. Algunos ejemplos de textos de gran tamaño podrían ser feeds de medios sociales, reseñas de clientes de hoteles, películas, etc., feedbacks de usuarios, noticias, correos electrónicos de quejas de clientes, etc.\n",
    "\n",
    "Saber de qué habla la gente y comprender sus problemas y opiniones es muy valioso para las empresas, los administradores y las campañas políticas. Y es realmente difícil leer manualmente volúmenes tan grandes y compilar los temas.\n",
    "\n",
    "Por lo tanto, se requiere un algoritmo automatizado que pueda leer los documentos de texto y generar automáticamente los tópicos tratados.\n",
    "\n",
    "En este taller, tomaremos un ejemplo real del conjunto de datos de '20 Newsgroups' (foros de discusiones en inglés) y usaremos el algoritmos LDA para extraer los tópicos discutidos.\n",
    "\n",
    "Usaremos el paquete Latent Dirichlet Allocation (LDA) de Gensim. También extraeremos el volumen y la contribución porcentual de cada tema para tener una idea de lo importante que es un tema."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Prerrequisitos y librerías necesarias - Descargar una lista de Stop-Word en inglés disponibles en el paquete NLTK\n",
    "\n",
    "Necesitaremos las palabras stop-words definidas en NLTK y también el modelo spacy para el procesamiento, en particular la lematización, del inglés."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /home/mvernier/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk; nltk.download('stopwords')\n",
    "\n",
    "import spacy\n",
    "\n",
    "# Initialize spacy 'en' model, keeping only tagger component (for efficiency)\n",
    "nlp = spacy.load('en_core_web_sm', disable=['parser', 'ner'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Los paquetes principales utilizados en este taller son _re_ (expresión regular), gensim, spacy y pyLDAvis. Además de esto también usaremos matplotlib, numpy y pandas para el manejo y visualización de datos.\n",
    "\n",
    "Importamos los paquetes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pprint import pprint\n",
    "\n",
    "# Gensim\n",
    "import gensim\n",
    "import gensim.corpora as corpora\n",
    "from gensim.utils import simple_preprocess\n",
    "from gensim.models import CoherenceModel\n",
    "\n",
    "# Plotting tools\n",
    "import pyLDAvis\n",
    "import pyLDAvis.gensim  # don't skip this\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\",category=DeprecationWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. ¿Qué hace LDA?\n",
    "\n",
    "El enfoque de LDA para el _Topic modeling_ es que considera cada documento como una colección de tópicos en una cierta proporción. Y cada tópico como una colección de palabras claves, de nuevo, en una cierta proporción.\n",
    "\n",
    "Una vez que uno proporciona el algoritmo con el número de temas, todo lo que hace es reorganizar la distribución de tópicos dentro de los documentos y la distribución de palabras claves dentro de los tópicos para obtener una buena composición de la distribución de tópicos y palabras clave.\n",
    "\n",
    "Cuando hablamos de __tópico__, ¿qué es realmente y cómo se representa?\n",
    "\n",
    "Un tópico no es más que un conjunto de palabras claves dominantes que son típicas de un tema dado. Con sólo mirar las palabras claves, puede identificar de qué se trata el tema.\n",
    "\n",
    "Por ejemplo, el tópico formado por las palabras: 'Sánchez', 'copa, 'ganar', 'Alexis', probablemente se refieren a un tópico que tiene que ver con el équipo chileno de fútbol."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Importar el dataset\n",
    "\n",
    "Para este ejercicio utilizaremos el conjunto de datos de los 20 foros en inglés '20-Newsgroup'. Esta versión del conjunto de datos contiene unos 11.000 mensajes de grupos de noticias de 20 temas diferentes. Está disponible como newsgroups.json.\n",
    "\n",
    "Esto se importa usando pandas.read_json y el conjunto de datos resultante tiene 3 columnas como se muestra.\n",
    "\n",
    "Para reducir el tiempo de proceso, reduciremos el dataset a 1.000 mensajes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['rec.autos' 'comp.sys.mac.hardware' 'rec.motorcycles' 'misc.forsale'\n",
      " 'comp.os.ms-windows.misc' 'alt.atheism' 'comp.graphics'\n",
      " 'rec.sport.baseball' 'rec.sport.hockey' 'sci.electronics' 'sci.space'\n",
      " 'talk.politics.misc' 'sci.med' 'talk.politics.mideast'\n",
      " 'soc.religion.christian' 'comp.windows.x' 'comp.sys.ibm.pc.hardware'\n",
      " 'talk.politics.guns' 'talk.religion.misc' 'sci.crypt']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>content</th>\n",
       "      <th>target</th>\n",
       "      <th>target_names</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>From: lerxst@wam.umd.edu (where's my thing)\\nS...</td>\n",
       "      <td>7</td>\n",
       "      <td>rec.autos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>From: guykuo@carson.u.washington.edu (Guy Kuo)...</td>\n",
       "      <td>4</td>\n",
       "      <td>comp.sys.mac.hardware</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>From: irwin@cmptrc.lonestar.org (Irwin Arnstei...</td>\n",
       "      <td>8</td>\n",
       "      <td>rec.motorcycles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>From: tchen@magnus.acs.ohio-state.edu (Tsung-K...</td>\n",
       "      <td>6</td>\n",
       "      <td>misc.forsale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000</th>\n",
       "      <td>From: dabl2@nlm.nih.gov (Don A.B. Lindbergh)\\n...</td>\n",
       "      <td>2</td>\n",
       "      <td>comp.os.ms-windows.misc</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                content  target  \\\n",
       "0     From: lerxst@wam.umd.edu (where's my thing)\\nS...       7   \n",
       "1     From: guykuo@carson.u.washington.edu (Guy Kuo)...       4   \n",
       "10    From: irwin@cmptrc.lonestar.org (Irwin Arnstei...       8   \n",
       "100   From: tchen@magnus.acs.ohio-state.edu (Tsung-K...       6   \n",
       "1000  From: dabl2@nlm.nih.gov (Don A.B. Lindbergh)\\n...       2   \n",
       "\n",
       "                 target_names  \n",
       "0                   rec.autos  \n",
       "1       comp.sys.mac.hardware  \n",
       "10            rec.motorcycles  \n",
       "100              misc.forsale  \n",
       "1000  comp.os.ms-windows.misc  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import Dataset\n",
    "df = pd.read_json('newsgroups.json')\n",
    "print(df.target_names.unique())\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=df.head(1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Preparar las palabras Stop-Words y filtrar emails\n",
    "\n",
    "Ya hemos descargado las palabras clave. Importémoslos y hagámoslos disponibles en la variable stop_words.\n",
    "\n",
    "Como pueden ver, en el dataset hay muchos correos electrónicos, líneas nuevas y espacios adicionales que distraen bastante. Deshagámonos de ellos usando expresiones regulares y stop-words específicos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NLTK Stop words\n",
    "from nltk.corpus import stopwords\n",
    "stop_words = stopwords.words('english')\n",
    "stop_words.extend(['from', 'subject', 're', 'edu', 'use'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['From: (wheres my thing) Subject: WHAT car is this!? Nntp-Posting-Host: '\n",
      " 'rac3.wam.umd.edu Organization: University of Maryland, College Park Lines: '\n",
      " '15 I was wondering if anyone out there could enlighten me on this car I saw '\n",
      " 'the other day. It was a 2-door sports car, looked to be from the late 60s/ '\n",
      " 'early 70s. It was called a Bricklin. The doors were really small. In '\n",
      " 'addition, the front bumper was separate from the rest of the body. This is '\n",
      " 'all I know. If anyone can tellme a model name, engine specs, years of '\n",
      " 'production, where this car is made, history, or whatever info you have on '\n",
      " 'this funky looking car, please e-mail. Thanks, - IL ---- brought to you by '\n",
      " 'your neighborhood Lerxst ---- ']\n"
     ]
    }
   ],
   "source": [
    "# Convert to list\n",
    "data = df.content.values.tolist()\n",
    "\n",
    "# Remove Emails\n",
    "data = [re.sub('\\S*@\\S*\\s?', '', sent) for sent in data]\n",
    "\n",
    "# Remove new line characters\n",
    "data = [re.sub('\\s+', ' ', sent) for sent in data]\n",
    "\n",
    "# Remove distracting single quotes\n",
    "data = [re.sub(\"\\'\", \"\", sent) for sent in data]\n",
    "\n",
    "pprint(data[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Después de eliminar los correos electrónicos y los espacios adicionales, el texto sigue pareciendo desordenado. No está listo para que el LDA lo consuma. Necesitamos tokenizar y lematizar cada frase.\n",
    "\n",
    "### 6. Tokenizar en palabras y limpieza\n",
    "\n",
    "Vamos a convertir cada frase en una lista de palabras, eliminando los signos de puntuación y los caracteres innecesarios por completo.\n",
    "\n",
    "El <code>simple_preprocess()</code> de Gensim es comodo para esto."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['from', 'wheres', 'my', 'thing', 'subject', 'what', 'car', 'is', 'this', 'nntp', 'posting', 'host', 'rac', 'wam', 'umd', 'edu', 'organization', 'university', 'of', 'maryland', 'college', 'park', 'lines', 'was', 'wondering', 'if', 'anyone', 'out', 'there', 'could', 'enlighten', 'me', 'on', 'this', 'car', 'saw', 'the', 'other', 'day', 'it', 'was', 'door', 'sports', 'car', 'looked', 'to', 'be', 'from', 'the', 'late', 'early', 'it', 'was', 'called', 'bricklin', 'the', 'doors', 'were', 'really', 'small', 'in', 'addition', 'the', 'front', 'bumper', 'was', 'separate', 'from', 'the', 'rest', 'of', 'the', 'body', 'this', 'is', 'all', 'know', 'if', 'anyone', 'can', 'tellme', 'model', 'name', 'engine', 'specs', 'years', 'of', 'production', 'where', 'this', 'car', 'is', 'made', 'history', 'or', 'whatever', 'info', 'you', 'have', 'on', 'this', 'funky', 'looking', 'car', 'please', 'mail', 'thanks', 'il', 'brought', 'to', 'you', 'by', 'your', 'neighborhood', 'lerxst']]\n"
     ]
    }
   ],
   "source": [
    "def sent_to_words(sentences):\n",
    "    for sentence in sentences:\n",
    "        yield(gensim.utils.simple_preprocess(str(sentence), deacc=True))  # deacc=True removes punctuations\n",
    "\n",
    "data_words = list(sent_to_words(data))\n",
    "\n",
    "print(data_words[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Creación de modelos Bigram y Trigram\n",
    "\n",
    "Los bigramas son dos palabras que aparecen frecuentemente juntas en el documento. Los trigramas son 3 palabras que ocurren con frecuencia.\n",
    "\n",
    "Algunos ejemplos en nuestro ejemplo son: 'front_bumper', 'oil_leak', 'maryland_college_park' etc.\n",
    "\n",
    "Gensim puede construir e implementar los bigrams, trigramas, cuadrantes y más. Los dos argumentos importantes para las Frases son min_count y threshold. Cuanto más altos sean los valores de estos parámetros, más difícil será que las palabras se combinen con bigrams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['from', 'wheres', 'my', 'thing', 'subject', 'what', 'car', 'is', 'this', 'nntp_posting_host', 'rac_wam_umd_edu', 'organization', 'university', 'of', 'maryland_college_park', 'lines', 'was', 'wondering', 'if', 'anyone', 'out', 'there', 'could', 'enlighten', 'me', 'on', 'this', 'car', 'saw', 'the', 'other', 'day', 'it', 'was', 'door', 'sports', 'car', 'looked', 'to', 'be', 'from', 'the', 'late', 'early', 'it', 'was', 'called', 'bricklin', 'the', 'doors', 'were', 'really', 'small', 'in', 'addition', 'the', 'front_bumper', 'was', 'separate', 'from', 'the', 'rest', 'of', 'the', 'body', 'this', 'is', 'all', 'know', 'if', 'anyone', 'can', 'tellme', 'model', 'name', 'engine', 'specs', 'years', 'of', 'production', 'where', 'this', 'car', 'is', 'made', 'history', 'or', 'whatever', 'info', 'you', 'have', 'on', 'this', 'funky', 'looking', 'car', 'please', 'mail', 'thanks', 'il', 'brought', 'to', 'you', 'by', 'your', 'neighborhood', 'lerxst']\n"
     ]
    }
   ],
   "source": [
    "# Build the bigram and trigram models\n",
    "bigram = gensim.models.Phrases(data_words, min_count=5, threshold=100) # higher threshold fewer phrases.\n",
    "trigram = gensim.models.Phrases(bigram[data_words], threshold=100)  \n",
    "\n",
    "# Faster way to get a sentence clubbed as a trigram/bigram\n",
    "bigram_mod = gensim.models.phrases.Phraser(bigram)\n",
    "trigram_mod = gensim.models.phrases.Phraser(trigram)\n",
    "\n",
    "# See trigram example\n",
    "print(trigram_mod[bigram_mod[data_words[0]]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Eliminar Stop-words, generar bigrams y lemmatizar\n",
    "\n",
    "El modelo bigrams está listo. Vamos a definir las funciones para eliminar los stop-words, hacer bigrams y lemmatización y llamarlas secuencialmente."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define functions for stopwords, bigrams, trigrams and lemmatization\n",
    "def remove_stopwords(texts):\n",
    "    return [[word for word in simple_preprocess(str(doc)) if word not in stop_words] for doc in texts]\n",
    "\n",
    "def make_bigrams(texts):\n",
    "    return [bigram_mod[doc] for doc in texts]\n",
    "\n",
    "#def make_trigrams(texts):\n",
    "#    return [trigram_mod[bigram_mod[doc]] for doc in texts]\n",
    "\n",
    "def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV']):\n",
    "    \"\"\"https://spacy.io/api/annotation\"\"\"\n",
    "    texts_out = []\n",
    "    for sent in texts:\n",
    "        doc = nlp(\" \".join(sent)) \n",
    "        texts_out.append([token.lemma_ for token in doc if token.pos_ in allowed_postags])\n",
    "    return texts_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Llamamos a la funciones en el orden:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['where', 's', 'thing', 'car', 'nntp', 'post', 'host', 'rac', 'wam', 'umd', 'organization', 'university', 'maryland', 'college', 'park', 'line', 'wonder', 'anyone', 'could', 'enlighten', 'car', 'see', 'day', 'door', 'sport', 'car', 'look', 'late', 'early', 'call', 'bricklin', 'door', 'really', 'small', 'addition', 'front', 'bumper', 'separate', 'rest', 'body', 'know', 'anyone', 'tellme', 'model', 'name', 'engine', 'specs', 'year', 'production', 'car', 'make', 'history', 'info', 'funky', 'look', 'car', 'mail', 'thank', 'bring', 'neighborhood', 'lerxst']]\n"
     ]
    }
   ],
   "source": [
    "# Remove Stop Words\n",
    "data_words_nostops = remove_stopwords(data_words)\n",
    "\n",
    "# Form Bigrams\n",
    "#data_words_bigrams = make_bigrams(data_words_nostops)\n",
    "\n",
    "# Do lemmatization keeping only noun, adj, vb, adv\n",
    "data_lemmatized = lemmatization(data_words_nostops, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV'])\n",
    "\n",
    "print(data_lemmatized[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Crear el diccionario y el corpus necesarios para el modelo de tópicos\n",
    "\n",
    "Las dos entradas principales al modelo de tópicos de LDA son el diccionario (id2word) y el corpus. Vamos a crearlos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[(0, 1), (1, 2), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 5), (8, 1), (9, 1), (10, 1), (11, 2), (12, 1), (13, 1), (14, 1), (15, 1), (16, 1), (17, 1), (18, 1), (19, 1), (20, 1), (21, 1), (22, 1), (23, 1), (24, 2), (25, 1), (26, 1), (27, 1), (28, 1), (29, 1), (30, 1), (31, 1), (32, 1), (33, 1), (34, 1), (35, 1), (36, 1), (37, 1), (38, 1), (39, 1), (40, 1), (41, 1), (42, 1), (43, 1), (44, 1), (45, 1), (46, 1), (47, 1), (48, 1), (49, 1), (50, 1), (51, 1), (52, 1), (53, 1)]]\n"
     ]
    }
   ],
   "source": [
    "# Create Dictionary\n",
    "id2word = corpora.Dictionary(data_lemmatized)\n",
    "\n",
    "# Create Corpus\n",
    "texts = data_lemmatized\n",
    "\n",
    "# Term Document Frequency\n",
    "corpus = [id2word.doc2bow(text) for text in texts]\n",
    "\n",
    "# View\n",
    "print(corpus[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gensim crea un identificador único para cada palabra del documento. El corpus producido que se muestra arriba es un mapeo de (word_id, word_frequency).\n",
    "\n",
    "Por ejemplo, (0, 1) arriba implica que la palabra id 0 ocurre una vez en el primer documento. Del mismo modo, la palabra id 1 aparece dos veces y así sucesivamente.\n",
    "\n",
    "Esto es usado como entrada por el modelo LDA.\n",
    "\n",
    "Si desea ver a qué palabra corresponde un id dado, pase el id como una llave al diccionario."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'addition'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2word[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "O bien, puede ver una forma legible por el ser humano del propio corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[('addition', 1),\n",
       "  ('anyone', 2),\n",
       "  ('body', 1),\n",
       "  ('bricklin', 1),\n",
       "  ('bring', 1),\n",
       "  ('bumper', 1),\n",
       "  ('call', 1),\n",
       "  ('car', 5),\n",
       "  ('college', 1),\n",
       "  ('could', 1),\n",
       "  ('day', 1),\n",
       "  ('door', 2),\n",
       "  ('early', 1),\n",
       "  ('engine', 1),\n",
       "  ('enlighten', 1),\n",
       "  ('front', 1),\n",
       "  ('funky', 1),\n",
       "  ('history', 1),\n",
       "  ('host', 1),\n",
       "  ('info', 1),\n",
       "  ('know', 1),\n",
       "  ('late', 1),\n",
       "  ('lerxst', 1),\n",
       "  ('line', 1),\n",
       "  ('look', 2),\n",
       "  ('mail', 1),\n",
       "  ('make', 1),\n",
       "  ('maryland', 1),\n",
       "  ('model', 1),\n",
       "  ('name', 1),\n",
       "  ('neighborhood', 1),\n",
       "  ('nntp', 1),\n",
       "  ('organization', 1),\n",
       "  ('park', 1),\n",
       "  ('post', 1),\n",
       "  ('production', 1),\n",
       "  ('rac', 1),\n",
       "  ('really', 1),\n",
       "  ('rest', 1),\n",
       "  ('s', 1),\n",
       "  ('see', 1),\n",
       "  ('separate', 1),\n",
       "  ('small', 1),\n",
       "  ('specs', 1),\n",
       "  ('sport', 1),\n",
       "  ('tellme', 1),\n",
       "  ('thank', 1),\n",
       "  ('thing', 1),\n",
       "  ('umd', 1),\n",
       "  ('university', 1),\n",
       "  ('wam', 1),\n",
       "  ('where', 1),\n",
       "  ('wonder', 1),\n",
       "  ('year', 1)]]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Human readable format of corpus (term-frequency)\n",
    "[[(id2word[id], freq) for id, freq in cp] for cp in corpus[:1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11. Construyendo el modelo de tópicos\n",
    "\n",
    "Tenemos todo lo necesario para entrenar el modelo LDA. Además del corpus y el diccionario, también debemos proporcionar el número de tópicos esperados (veremos en secciones ulteriores cómo estimar el mejor número de tópicos).\n",
    "\n",
    "_chunksize_ es el número de documentos que se utilizarán en cada chunk de entrenamiento. _update_every_ determina la frecuencia con la que se deben actualizar los parámetros del modelo y pasa es el número total de pases de entrenamiento.\n",
    "\n",
    "El proceso de entramiento del modelo de tópicos puede demorar unos minutos:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build LDA model\n",
    "lda_model = gensim.models.ldamodel.LdaModel(corpus=corpus,\n",
    "                                           id2word=id2word,\n",
    "                                           num_topics=10, \n",
    "                                           random_state=100,\n",
    "                                           update_every=1,\n",
    "                                           chunksize=100,\n",
    "                                           passes=10,\n",
    "                                           alpha='auto',\n",
    "                                           per_word_topics=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. Ver los tópicos del modelo LDA\n",
    "\n",
    "El modelo LDA anterior está construido con 10 tópicos diferentes donde cada tópico es una combinación de palabras claves y cada palabra clave contribuye con un cierto peso al tema.\n",
    "\n",
    "Puede ver las palabras clave para cada tema y el peso (importancia) de cada palabra clave usando lda_model.print_topics() como se muestra a continuación."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0,\n",
      "  '0.668*\"ax\" + 0.047*\"max\" + 0.008*\"contact\" + 0.005*\"roger\" + 0.004*\"wiring\" '\n",
      "  '+ 0.004*\"outlet\" + 0.003*\"yo\" + 0.003*\"excellent\" + 0.003*\"sj\" + '\n",
      "  '0.003*\"volt\" + 0.003*\"drain\" + 0.003*\"giz\" + 0.003*\"tb\" + 0.003*\"qax\" + '\n",
      "  '0.002*\"bhj\" + 0.002*\"strip\" + 0.002*\"ey\" + 0.002*\"tm\" + 0.002*\"metal\" + '\n",
      "  '0.002*\"org\"'),\n",
      " (1,\n",
      "  '0.013*\"say\" + 0.012*\"would\" + 0.012*\"not\" + 0.010*\"write\" + 0.009*\"people\" '\n",
      "  '+ 0.009*\"believe\" + 0.008*\"make\" + 0.008*\"may\" + 0.007*\"mean\" + '\n",
      "  '0.007*\"think\" + 0.007*\"point\" + 0.007*\"life\" + 0.007*\"question\" + '\n",
      "  '0.007*\"know\" + 0.006*\"see\" + 0.006*\"do\" + 0.006*\"give\" + 0.006*\"word\" + '\n",
      "  '0.005*\"claim\" + 0.005*\"way\"'),\n",
      " (2,\n",
      "  '0.011*\"state\" + 0.010*\"government\" + 0.009*\"armenian\" + '\n",
      "  '0.008*\"stephanopoulo\" + 0.008*\"gun\" + 0.008*\"people\" + 0.007*\"law\" + '\n",
      "  '0.007*\"right\" + 0.006*\"president\" + 0.006*\"american\" + 0.006*\"sale\" + '\n",
      "  '0.005*\"israel\" + 0.005*\"league\" + 0.005*\"country\" + 0.005*\"kill\" + '\n",
      "  '0.004*\"israeli\" + 0.004*\"house\" + 0.004*\"year\" + 0.004*\"war\" + '\n",
      "  '0.004*\"city\"'),\n",
      " (3,\n",
      "  '0.030*\"christian\" + 0.030*\"god\" + 0.016*\"jesus\" + 0.016*\"man\" + '\n",
      "  '0.014*\"bible\" + 0.012*\"jew\" + 0.011*\"church\" + 0.010*\"rise\" + 0.009*\"child\" '\n",
      "  '+ 0.009*\"christ\" + 0.009*\"verse\" + 0.008*\"love\" + 0.008*\"lord\" + '\n",
      "  '0.008*\"law\" + 0.007*\"davidian\" + 0.007*\"faith\" + 0.007*\"father\" + '\n",
      "  '0.007*\"gay\" + 0.007*\"religion\" + 0.007*\"jewish\"'),\n",
      " (4,\n",
      "  '0.024*\"image\" + 0.017*\"window\" + 0.017*\"server\" + 0.015*\"program\" + '\n",
      "  '0.013*\"circuit\" + 0.013*\"display\" + 0.011*\"code\" + 0.011*\"available\" + '\n",
      "  '0.009*\"directory\" + 0.009*\"motif\" + 0.009*\"object\" + 0.008*\"include\" + '\n",
      "  '0.008*\"format\" + 0.008*\"function\" + 0.008*\"version\" + 0.007*\"gif\" + '\n",
      "  '0.007*\"ftp\" + 0.007*\"error\" + 0.007*\"set\" + 0.007*\"sun\"'),\n",
      " (5,\n",
      "  '0.039*\"not\" + 0.024*\"do\" + 0.021*\"be\" + 0.017*\"go\" + 0.016*\"get\" + '\n",
      "  '0.014*\"would\" + 0.013*\"article\" + 0.013*\"write\" + 0.012*\"know\" + 0.010*\"s\" '\n",
      "  '+ 0.010*\"think\" + 0.010*\"time\" + 0.009*\"have\" + 0.009*\"say\" + 0.009*\"make\" '\n",
      "  '+ 0.009*\"see\" + 0.008*\"good\" + 0.008*\"well\" + 0.007*\"take\" + 0.007*\"come\"'),\n",
      " (6,\n",
      "  '0.024*\"translation\" + 0.015*\"award\" + 0.014*\"mary\" + 0.013*\"construction\" + '\n",
      "  '0.012*\"explicitly\" + 0.011*\"favor\" + 0.010*\"mainframe\" + 0.009*\"reporter\" + '\n",
      "  '0.009*\"gross\" + 0.008*\"clinic\" + 0.008*\"cock\" + 0.008*\"infection\" + '\n",
      "  '0.007*\"oklahoma\" + 0.007*\"huntsville\" + 0.007*\"genetic\" + 0.007*\"roby\" + '\n",
      "  '0.006*\"infinite\" + 0.006*\"disorder\" + 0.006*\"safely\" + 0.006*\"china\"'),\n",
      " (7,\n",
      "  '0.018*\"game\" + 0.017*\"team\" + 0.012*\"play\" + 0.012*\"win\" + 0.012*\"year\" + '\n",
      "  '0.011*\"hockey\" + 0.010*\"player\" + 0.008*\"season\" + 0.007*\"nhl\" + '\n",
      "  '0.006*\"canada\" + 0.006*\"university\" + 0.005*\"line\" + 0.005*\"last\" + '\n",
      "  '0.005*\"new\" + 0.005*\"organization\" + 0.005*\"goal\" + 0.005*\"division\" + '\n",
      "  '0.005*\"fan\" + 0.004*\"score\" + 0.004*\"pittsburgh\"'),\n",
      " (8,\n",
      "  '0.029*\"card\" + 0.029*\"drive\" + 0.024*\"window\" + 0.014*\"problem\" + '\n",
      "  '0.013*\"driver\" + 0.012*\"run\" + 0.011*\"color\" + 0.011*\"disk\" + '\n",
      "  '0.010*\"memory\" + 0.010*\"video\" + 0.010*\"use\" + 0.009*\"board\" + '\n",
      "  '0.009*\"system\" + 0.009*\"mode\" + 0.009*\"pc\" + 0.008*\"bit\" + 0.008*\"hard\" + '\n",
      "  '0.008*\"speed\" + 0.008*\"monitor\" + 0.008*\"ram\"'),\n",
      " (9,\n",
      "  '0.026*\"line\" + 0.022*\"organization\" + 0.017*\"post\" + 0.013*\"university\" + '\n",
      "  '0.013*\"host\" + 0.012*\"nntp\" + 0.008*\"use\" + 0.008*\"new\" + '\n",
      "  '0.007*\"distribution\" + 0.007*\"write\" + 0.007*\"system\" + 0.006*\"space\" + '\n",
      "  '0.006*\"thank\" + 0.006*\"reply\" + 0.005*\"anyone\" + 0.005*\"also\" + '\n",
      "  '0.005*\"file\" + 0.005*\"need\" + 0.005*\"computer\" + 0.005*\"mail\"')]\n"
     ]
    }
   ],
   "source": [
    "# Print the Keyword in the 10 topics\n",
    "pprint(lda_model.print_topics(num_words=20))\n",
    "doc_lda = lda_model[corpus]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__¿Cómo interpretar esto?__\n",
    "\n",
    "El tópico 7 se representa como _0.018 \"game\" + 0.017 \"team\" + 0.012 \"play\" + 0.012 \"win\" + 0.011 \"year\" + 0.011 \"hockey\" + 0.010 \"player\", etc.\n",
    "\n",
    "Las ponderaciones reflejan la importancia de una palabra clave para ese tema.\n",
    "\n",
    "Mirando estas palabras clave, ¿puedes adivinar cuál podría ser este tema? --> campeonato de hockey norte-americano.\n",
    "\n",
    "Del mismo modo, ¿puede repasar las palabras claves de cada tópico evaluar cuál es el tópico?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13. Calcular las métricas de perplejidad y coherencia de los tópicos\n",
    "\n",
    "Las métricas de __perplejidad del modelo__ y la __coherencia de los tópicos__ proporcionan una medida para evaluar qué tan bueno es un modelo de tópico dado."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Perplexity:  -9.042218143011286\n",
      "\n",
      "Coherence Score:  0.6103054612984313\n"
     ]
    }
   ],
   "source": [
    "# Compute Perplexity\n",
    "print('\\nPerplexity: ', lda_model.log_perplexity(corpus))  # a measure of how good the model is. lower the better.\n",
    "\n",
    "# Compute Coherence Score\n",
    "coherence_model_lda = CoherenceModel(model=lda_model, texts=data_lemmatized, dictionary=id2word, coherence='c_v')\n",
    "coherence_lda = coherence_model_lda.get_coherence()\n",
    "print('\\nCoherence Score: ', coherence_lda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14. ¿Cómo encontrar el número óptimo de tópicos ?\n",
    "\n",
    "El enfoque clásico para encontrar el número óptimo de tópicos consiste en construir muchos modelos LDA con diferentes valores de número de tópicos (k) y elegir el que dé el valor de coherencia más alto.\n",
    "\n",
    "La elección de una \"k\" que marca el final de un rápido crecimiento de la coherencia temática suele ofrecer temas significativos e interpretables. Escoger un valor aún mayor puede a veces proporcionar subtópicos más granulares.\n",
    "\n",
    "Si ve que las mismas palabras clave se repiten en varios temas, es probable que sea una señal de que la 'k' es demasiado grande.\n",
    "\n",
    "Los valores de compute_coherence_values() (ver abajo) entrena múltiples modelos LDA y proporciona los modelos y sus correspondientes puntuaciones de coherencia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_coherence_values(dictionary, corpus, texts, limit, start=2, step=1):\n",
    "    \"\"\"\n",
    "    Compute c_v coherence for various number of topics\n",
    "\n",
    "    Parameters:\n",
    "    ----------\n",
    "    dictionary : Gensim dictionary\n",
    "    corpus : Gensim corpus\n",
    "    texts : List of input texts\n",
    "    limit : Max num of topics\n",
    "\n",
    "    Returns:\n",
    "    -------\n",
    "    model_list : List of LDA topic models\n",
    "    coherence_values : Coherence values corresponding to the LDA model with respective number of topics\n",
    "    \"\"\"\n",
    "    coherence_values = []\n",
    "    model_list = []\n",
    "    for num_topics in range(start, limit, step):\n",
    "        model = gensim.models.ldamodel.LdaModel(corpus=corpus,\n",
    "                                           id2word=id2word,\n",
    "                                           num_topics=num_topics, \n",
    "                                           random_state=100,\n",
    "                                           update_every=1,\n",
    "                                           chunksize=100,\n",
    "                                           passes=3,\n",
    "                                           alpha='auto',\n",
    "                                           per_word_topics=True)\n",
    "        model_list.append(model)\n",
    "        coherencemodel = CoherenceModel(model=model, texts=texts, dictionary=dictionary, coherence='c_v')\n",
    "        coherence_values.append(coherencemodel.get_coherence())\n",
    "\n",
    "    return model_list, coherence_values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Entrenar los distintos modelos puede necesitar varios minutos:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Can take a long time to run.\n",
    "model_list, coherence_values = compute_coherence_values(dictionary=id2word, corpus=corpus, texts=data_lemmatized, start=2, limit=10, step=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VfX9x/HXJ4sACTsQICRh7x2WAwUrolVcqAxR21prK2rVLu2w1dqh1toq/VWrrYIMARcKroo4WWFvDAEyQAhhBAgh6/P7497QGDJuQk7OHZ/n43Ef5J57xhuM+eSc8z2fr6gqxhhjTHXC3A5gjDHG/1mxMMYYUyMrFsYYY2pkxcIYY0yNrFgYY4ypkRULY4wxNbJiYYwxpkZWLIwxxtTIioUxxpgaRbgdoL60adNGk5OT3Y5hjDEBZc2aNYdUNa6m9YKmWCQnJ5Oamup2DGOMCSgisteX9ewylDHGmBpZsTDGGFMjKxbGGGNqFDT3LCpTVFREVlYWBQUFbkepUnR0NAkJCURGRrodxRhjqhTUxSIrK4vY2FiSk5MREbfjnEVVyc3NJSsri86dO7sdxxhjqhTUl6EKCgpo3bq1XxYKABGhdevWfn3mY4wxEOTFAvDbQlHG3/MZYwyEQLEwxphgtmjDPt5an+34cRwtFiIyXkR2iEiaiPyiks9vE5EcEVnvfd3uXZ4kImu9y7aIyJ1O5jTGmECUeTifh17fxOwVGZSWqqPHcuwGt4iEAzOAS4EsYLWILFLVrRVWfVVVp1dYth8YpaqnRSQG2Ozddp9TeY0xJpCUlCoPLNgAwF9uHEhYmLOXtJ08sxgOpKlquqoWAvOAq33ZUFULVfW0920jAvxy2cyZMxkwYAADBw5k2rRpbscxpsEVFpdSUFTidoyg8uLn6azafZjfTuhLp1ZNHD+ek0NnOwKZ5d5nASMqWe96ERkN7ATuU9VMABHpBCwGugE/Pdezit+9vYWt+/LOZRdn6dOhGQ9f1bfadbZs2cLvf/97vvzyS9q0acPhw4frNYMxgeDeeevYsi+Pt6dfQPMm9kzRudq2P48n39/J+L7xXD+kY4Mc0+3f2N8GklV1APAh8HLZB6qa6V3eDbhVRNpV3FhE7hCRVBFJzcnJabDQtbF06VJuuOEG2rRpA0CrVq1cTmRMw9qbe5L3tnxNxuF8fv7aRlSdvbYe7E4Xl3Dfq+tp1jiSP1zXv8FGVDp5ZpENdCr3PsG77AxVzS339gXg8Yo7UdV9IrIZuBBYWOGz54HnAVJSUqr9DqzpDMAY44xXVuwlTITvXpDMvz7bzSsrM5g2MsntWAHrqQ92sv3r4/zntmG0ahrVYMd18sxiNdBdRDqLSBQwCVhUfgURaV/u7QRgm3d5gog09n7dErgA2OFgVseMHTuWBQsWkJvrqYt2GcqEkoKiEuanZnFZ33Y8eHlvLu4Zx6PvbGXb/vq9JBwqVqTn8vxn6UwdkciYXm0b9NiOFQtVLQamA+/jKQLzVXWLiDwiIhO8q93jHRq7AbgHuM27vDew0rv8E+BJVd3kVFYn9e3bl1/+8pdcdNFFDBw4kPvvv9/tSMY0mEUb9nHsVBHTRiYTFiY8ecNAWjSOZPqcteQXFrsdL6DkFRTxwPwNJLVqwi+/3bvBjy/Bcv0wJSVFK05+tG3bNnr3bvh/1NoKlJzG1IaqMuHZLygoKuGD+0afubb+Zdohpr64khuGJvD4xIEupwwc989fz1vr97HwzlEMTmxZb/sVkTWqmlLTem7f4DbGBKn1mUfZlH2MaaOSvnET9rxubZg+phvzU7Ma5MnjYLBk035eX5vNXWO61WuhqA0rFsYYR8xasZemUeFcO/jsoZ33XtKdlKSWPPT6JvYcOulCusBxMK+Ah97YxICE5tw9tptrOYK+WPj7ZTZ/z2dMXRw+Wcg7G/dz3ZAEYqPPfq4iIjyMv00eTER4GNPnruV0sT2wVxlV5acLN1JQVMJfbxpEZLh7P7KDulhER0eTm5vrtz+Qy+aziI6OdjuKMfXq1dWZFBaXMm1U1UNkO7ZozOMTB7A5O4/H3wvIwY6Oe2VlBp/szOGhK3rTNS7G1SxBPflRQkICWVlZ+OsDe/C/mfKMCRYlpcrslXsZ0bkVPdrFVrvuZX3jue28ZF78fDfndW3NJb3PevY2ZKXnnOCxxVsZ3SPOL55LCepiERkZaTPQGdPAlu04SNaRUzx4uW8j/H5xeS9W7T7MTxZs4N17RxPf3M60i0pKue/V9URHhvPExAF+Me9NUF+GMsY0vJnL99I2thHj+vp2lhAdGc4zUwZzuriUe+eto8ThVtuBYMbHaWzIOsYfru1Pu2b+UTytWBhj6s2eQyf5ZGcOk4cn1upmbNe4GB69uh8rdx/m2aVpDib0f+szj/LM0jSuG9yRK/q3r3mDBmLFwhhTb2av3EtEmDBlRGKtt71+aALXDe7I3z7aycr03Jo3CEL5hcXc9+p64ptF89ur/aufnRULY0y9OFVY1gcqvs6XTh65ph9JrZty77z1HD5ZWM8J/d8fl2xnT+5JnrxhIM0qGXLsJisWxph68XZZH6hqhsvWJKZRBM9MHszhk4X8dMEGvx327oSPdxxk1oq93H5BZ0Z1be12nLNYsTDGnDNVZeaKPfRoF8OIzuc2Z0u/js158IpefLT9IP/5Yk/9BPRzR04W8rOFG+nZLpYHxvV0O06lrFgYY87Z+syjbM7OY9rIpHoZ5nnbecl8q3db/vjuNjZlHauHhP5LVXnojU0czS/krzcNIjoy3O1IlbJiYYw5Z7OWe/tADamfB0xFhCcmDqR100bcPXctJ04HbzvzN9Zl8+7mr3lgXE/6dGjmdpwqWbEwxpyT3BOnz/SBimlUf8/5tmwaxd8nDybjcD6/emNTUN6/yDqSz8NvbWF4ciu+f2EXt+NUy4qFMeaczE/NorCk+j5QdTW8cyt+/K0evLl+H6+tDa525iWlyv3zN6DAX24cSHiY+09pV8fRYiEi40Vkh4ikicgvKvn8NhHJEZH13tft3uWDRGS5dxa9jSJyk5M5jTF1U1KqvLJiLyO71NwHqq7uGtONkV1a8es3N7Mr54Qjx3DDi5+ns2r3YR6+qg+dWjVxO06NHCsWIhIOzAAuB/oAk0WkTyWrvqqqg7yvF7zL8oFbVLUvMB54WkRaOJXVGFM3H28/SPbRU9wyKtmxY4SHCX+bNJjGUeFMn7OOgqLAb2e+bX8eT76/k8v6tmPi0MBoJOrkmcVwIE1V01W1EJgHXO3Lhqq6U1W/8n69DzgIxDmW1BhTJ7NW7KVds0Zc2sfZbrHtmkXz5A0D2LY/jz8u2ebosZx2uriE+15dT7PGkfzh2v5+0STQF04Wi45AZrn3Wd5lFV3vvdS0UEQ6VfxQRIYDUcAuZ2IaY+qirn2g6mpsr3bcfkFnXl6+l/c2f+348Zzy1Ac72f71cR6f2J/WMY3cjuMzt29wvw0kq+oA4EPg5fIfikh7YBbwHVUtrbixiNwhIqkikurPc1YYE4xeWeHpAzV5eO37QNXVz8b3on/H5vxs4QayjuQ32HHry4r0XJ7/LJ0pIxIZ2yuw5u5wslhkA+XPFBK8y85Q1VxVPe19+wIwtOwzEWkGLAZ+qaorKjuAqj6vqimqmhIXZ1epjGkonj5QmefUB6ouoiLCeGbyYEoV7p23nuKSs36H9Ft5BUU8MH8DSa2a8Ktv+zbXhz9xslisBrqLSGcRiQImAYvKr+A9cygzAdjmXR4FvAHMVNWFDmY0xtTB2xv2kVdQ7Mhw2Zokt2nKY9f2Y83eIzz9368a/Ph19btFW/k6r4C/3jSIJlGBN++cY8VCVYuB6cD7eIrAfFXdIiKPiMgE72r3eIfHbgDuAW7zLr8RGA3cVm5Y7SCnshpjfFeffaDq6upBHbkxJYEZy9L4Iu2QKxlq491N+3ltbRZ3jenG4MSWbsepEwmWpyJTUlI0NTXV7RjGBL21GUe47h9f8ug1/VydGzq/sJirnvmcvIJi3r33Qtr46c3ig3kFXPb0p3Rq1YTXfnhegwwGqA0RWaOqKTWt51+pjTF+75Xle4lpFMG1gysb3NhwmkRF8OyUIRw75bkXUOqH07GqKj97bSOnikr4602D/K5Q1EbgJjfGNLj/9YHqWK99oOqqd/tm/ObKPnyyM4cXPk93O85ZZq/MYNmOHB66ojdd42LcjnNOrFgYY3z2amqmpw+Ui5efKpo6IpHL+8Xz+Hs7WJ951O04Z6TnnOCxxdsY3SPOr/696sqKhTHGJyWlyuwVGYzq0pruDvWBqgsR4U/XDaBds2junruWvIIityNRXFLKffM3EBURxhMTBwTMU9rVsWJhjPFJWR8oN4bL1qR5k0j+Pnkw+44W8OBr7rczn/HxLjZkHuUP1/Zv0OdQnGTFwhjjk5kN1AeqroYmteSBcT1YvGk/81Zn1ryBQzZkHuXvS7/i2sEd+faA9jVvECCsWBhjarT70Ek+3ZnDlOFJfj2i587RXbmgWxt+u2gLOw8cb/Dj5xcWc9+r62kX24jfTujb4Md3kv/+VzfG+I3ZZ/pAndXr06+EhQlP3TSQ2OgIps9Zy6nChm1n/scl20k/dJInbxxI88aRDXpsp1mxMMZU60wfqH7xtA2A6+9tY6N56sZB7Dxwgkfe2dpgx/14x0FmrdjL7Rd05ryubRrsuA3FioUxplqLNmSTV1DMLQE0/HN0jzjuvKgrc1dl8M7GfY4f78jJQn62cCM92sXwk8t6On48N1ixMMZUSVWZuXwvPdvFMtylPlB19cC4Hgzq1IIHX9tE5mHn2pmrKg+9sYmj+YX89aZBREeGO3YsN1mxMMZUaV3mUbbsy+PmUUkB96xAZLinnTkC0+euo8ihduZvrMvm3c1fc/+lPenbobkjx/AHViyMMVWa5Sd9oOqqU6sm/Pn6AWzIPMqT7++o9/1nHcnn4be2MDy5FXeM7lLv+/cnViyMMZU6dOI0izfu53o/6QNVV1f0b8/UEYk892k6y3YcrLf9lpYqD8zfgAJ/uXEg4WGBdeZVW1YsjDGVmu/tA3VzAN3Yrsqvr+xDz3axPDB/AwfzCuplny9+vpuVuw/z8FV96NSqSb3s059ZsTDGnMVf+0DVVXRkOM9OGczJwmJ+/Op6Ss6xnfn2r/N44v0dXNa3HROHJtRTSv9mxcIYc5al3j5Qt/hhH6i66t4ult9N6MuXu3L55ye76ryf08Ul/Hjeepo1juQP1/YPuBv/deVosRCR8SKyQ0TSROQXlXx+m4jklJs69fZyn70nIkdF5B0nMwK8uS6bE6eLnT6MMQFjlp/3gaqrG1M6cdXADjz14U5S9xyu0z6e+nAn278+zuMT+9PaT2fnc4JjxUJEwoEZwOVAH2CyiPSpZNVXVXWQ9/VCueVPANOcylcm7eAJHliwgXvmrjvnU1NjgkH5PlARftwHqi5EhMeu7UfHFo25d956juYX1mr7lem5PP9pOlNGJDK2V3AV0po4+Z0wHEhT1XRVLQTmAVf7urGqfgQ43gmsW9sYfntVH5ZuP8gfl2xz+nDG+L1XAqQPVF01i47kmcmDOZBXwM9f2+hzO/PjBUXcP38DSa2a8Msrejuc0v84WSw6AuX7BGd5l1V0vYhsFJGFIlKr704RuUNEUkUkNScnp85Bp41K5tZRSbzw+W7mrcqo836MCXSnCktYkJrJ+ADpA1VXAzu14Ofje/H+lgO8smKvT9v87u2t7D92iqduGkTTAB5KXFdun2O+DSSr6gDgQ+Dl2mysqs+raoqqpsTFxZ1TkF9f2YfRPeL41Zub+XLXoXPalzGBqqwPVDBMA1qT713QmYt7xvHo4m1s2Xes2nXf27yfhWuymD6mG0MSWzZQQv/iZLHIBsqfKSR4l52hqrmqetr79gVgqIN5qhURHsazUwaT3KYpP3xlLbsPnXQrijGuCOQ+UHURFiY8ecNAWjSO5O656zhZxSCXg8cLePD1TfTv2Jy7L+newCn9h5PFYjXQXUQ6i0gUMAlYVH4FESk/jdQEwNWbBs2iI3nx1hTCBL730mqO5bs/l68xDWVthqcP1LQA7ANVV21iGvH0TYPYfegkDy/actbnqsrPFm4kv7CEv940yK8nfnKaT39zEWksIrXqu6uqxcB04H08RWC+qm4RkUdEZIJ3tXtEZIuIbADuAW4rd8zPgAXAJSKSJSKX1eb4dZXUuinPTUsh80g+P5qzxrHmY8b4m1dWBHYfqLo6r1sb7h7TjYVrsnhjXdY3Ppu9MoNlO3J46IredGsb41JC/1BjsRCRq4D1wHve94NEZFH1W3mo6hJV7aGqXVX1Me+y36jqIu/XD6pqX1UdqKpjVHV7uW0vVNU4VW2sqgmq+n5d/oJ1MbxzK/5wbX++SMvlt4u2uD75uzFOK98HKhRv3t5zSXeGJbfkV29sPnMJOj3nBI8t3saF3duExD2cmvhyZvFbPMNgjwKo6nqgs4OZ/MINKZ34wUVdmL0yg5e+3ON2HGMc9epqTx+oaUH0xHZtRISH8bdJg4kID+PuuWs9c2nP30BURBhPTBxIWJA3CfSFL8WiSFUrDhUIiV+1f35ZLy7t045H39nKx/XYrdIYf1JSqsxZmcF5XVvTrW3g94Gqqw4tGvPExAFszs7jyr9/zobMozx2bT/imwfvEOLa8KVYbBGRKUC4iHQXkWeALx3O5RfCwoSnbxpEz/hm3D1nHTsPOP6MoDENLhj7QNXVuL7x3HZeMumHTnLNoA5cOaCD25H8hi/F4m6gL3AamAMcA37sZCh/0rRRBC/emkLjqHC+9/Jqck+crnkjYwLIzOV7iG8Wzbd6h1b7iqo8eEUvnpg4gN9f29/tKH6l2mLh7e/0iKr+UlWHeV+/UtX6aQgfIDq0aMy/bknhYN5pfjBrDaeLS9yOZEy9SM85wWdfHWLKiMSg6wNVV40iwrkhpVNAT/jkhGq/O1S1BLiggbL4tUGdWvCXGweSuvcID76+yUZImaAwe2UGEWHCpCDtA2Xqjy+lc513qOwC4Mxjzar6umOp/NSVAzqw6+BJ/vrfnXRrG8OPLu7mdiRj6uwbfaBi7SauqZ4vxSIayAXGllumQMgVC4B7LunGrpwTPP7eDrq0iWF8v3i3IxlTJ2+t9/SBumVUsttRTACosVio6ncaIkigEBEenziAjMP53PfqehJajqJfx+ZuxzKmVsr6QPWKj2VYcmg2xjO148sT3Aki8oaIHPS+XhOR0Jh0tgrRkeE8f8tQWjaJ5PaXUzlQTxPAG9NQ1mYcZev+0OoDZc6NL8Mf/oOnAWAH7+tt77KQ1jY2mhdvG0ZeQRHfn5nKqUIbIWUCx6zle4htFME1g0KrD5SpO1+KRZyq/kdVi72vl4BzmzwiSPRu34y/TxrMpuxj/GTBBkptWlYTAA6dOM2STV9z/dCEkOwDZerGl2KRKyI3i0i493UznhveBvhWn3Y8eHkvFm/az9P/3el2HGNqVNYH6mZrjmdqwZdi8V3gRuBrYD8wEbCb3uV8/8Iu3JiSwN+XpvHW+uyaNzDGJd/sAxXaLbdN7fgyGmovnomJTBVEhN9f05+9ufn8dOFGElo2YWiSjTAx/uejbQfIPnqKX1/Z2+0oJsD4MhrqZRFpUe59SxH5t7OxAk9URBj/vHko7ZtH84NZqWQdyXc7kjFnmbViL+2bWx8oU3u+XIYaoKpHy96o6hFgsC87F5HxIrJDRNJE5BeVfH6biOSIyHrv6/Zyn90qIl95X7f6cjy3tWwaxYu3DuN0cSm3v5zKiSrm9DXGDWf6QA23PlCm9nz5jgkTkTPXVESkFT5cvvI2IZwBXA70ASaLSJ9KVn1VVQd5Xy+UO8bDwAg8Ey89XD6DP+vWNoZ/TB3CVwdPcO/cdZTYCCnjJ15ZkUFkuHCT9YEydeBLsfgLsFxEHhWR3+OZy+JxH7YbDqSparqqFgLzgKt9zHUZ8KGqHvaeyXwIjPdxW9dd2D2O317Vh4+2H+RP725zO44x5BcWs2BNJuP7tbc+UKZOaiwWqjoTuA44gGdE1HWqOsuHfXcEMsu9z/Iuq+h6EdkoIgtFpOxXHl+39VvTRiVz66gk/vXZbl5dneF2HBPiFq3fx/GCYpvgyNSZLze4uwK7VPVZYDPwrfI3vM/R20Cyqg7Ac/bwcm02FpE7RCRVRFJzcnLqKVL9+fWVfRjdI45fvrGZ5bvs0RTjjvJ9oFJslJ6pI18uQ70GlIhIN+A5oBOeGfNqku1dt0yCd9kZqpqrqmVTz70ADPV1W+/2z6tqiqqmxMX530PlEeFhPDtlMMltmvLD2WvYc+hkzRsZU8/WZhyxPlDmnPlSLEpVtRjPpahnVfWnQHsftlsNdBeRziISBUzC02PqDBEpv58JQNkF/veBcd5hui2Bcd5lAadZdCQv3pqCAN99eTXH8ovcjmRCzKzle60PlDlnvhSLIhGZDNwCvONdFlnTRt4CMx3PD/ltwHxV3SIij4hI2UN+94jIFhHZANwD3Obd9jDwKJ6CsxrP1K6Hff9r+Zek1k15bloKmYfzuWvOWopKSt2OZEKE9YEy9UVqmh7UO9z1TmC5qs4Vkc7Ajar654YI6KuUlBRNTU11O0a1FqRm8tOFG7l5ZCKPXt3PLgkYx834OI0n3t/Bf++/yNp7mEqJyBpVTalpPV/afWzF81t/2fvdgF8VikBxQ0on0nJO8Nwn6XSLi+G28zu7HckEseKSUmav2Mv53awPlDl39hhnA/v5Zb24tE87HnlnK8t2HHQ7jgliS7cfZN+xAqaNTHY7igkCViwaWFiY8PRNg+gZ34y756zjqwPH3Y5kgtT/+kC1dTuKCQI+FwsRaeJkkFDStFEEL96aQnRUON99eTW5J07XvJExtWB9oEx98+WhvPNEZCuw3ft+oIj8w/FkQa5Di8b865YUDuad5s5X1nC6ODinZbXeWO4o6wM1aXii21FMkPDlV46/4unVlAugqhuA0U6GChWDOrXgyRsGsnrPER56fTM1jUwLFBm5+bzwWTo3PrecHr96l3c37Xc7Ukgp6wN1eb/2xMU2cjuOCRI+DbxW1cwKwzyD89dgF1w1sAO7ck7w9H+/olvbGH54cVe3I9WaqrI5O48Ptn7Nh1sPsP1rz32YXvGxtGoaxQuf7+by/r48x2nqw1vePlDTrA+UqUe+FItMETkPUBGJBO7lf09am3pw7yXd2ZVzksff306XuKZc1jfe7Ug1KiopZWX64TMFYv+xAsIEhiW34lff7s24PvEktm7Cvz5N57El29j+dR694pu5HTvoqSqzrA+UcYAvxeJO4G94ur5mAx8AdzkZKtSICE9MHEDm4Xx+PG89C+4cRb+Ozd2OdZbjBUV8sjOHD7ceYOn2gxwvKCY6MozR3eN4YFxPxvZqS6umUd/Y5vqhCTzxwQ7mrMzgkav7uZQ8dJT1gfrDtf3toU9Tr3x5KO8QMLUBsoS06Mhwnr9lKNc8+wXfn5nKW3edT9tm7s87cDCvgA+3HeCDLQdYviuXwpJSWjWNYnzfeMb1jeeCbm1oHBVe5fatmkbx7f7teX1tNj8f38taTjhsZlkfqMEd3I5igowvM969DNxbNrWqt7HfX1T1u06HCzVtY6N54dZhTPznl3x/Ziqv/mAU0ZFV/yB2gqqyK+cEH2z1FIj1mZ4ZdZNaN+HW85K4tE88Q5NaEh7m+2+tU0ck8sa6bN7esM9G5zgo5/hplmzaz9QRSTSJsqJs6pcv31FnzcEtIj7NwW1qr0+HZvxt0mDumJXKAws28MykwYTV4gdzXZSUKuszj/DBlgN8uPUA6d5W6gMTmvOTcT0Y1zee7m1j6nxZY2hSS3q0i2H2ygwrFg6an5pJUYnajW3jCF+KRZiItPROb+rzHNym7i7t045fjO/FH9/dTte4GO6/tEe9H6OgqIQv0g7x4dYD/HfbAQ6dKCQyXBjZpTXfuaAzl/ZuR3zz+rkMJiJMHZHEw4u2sDHrKAMS6mvuLFOmrA/UBd3a0DXO+kCZ+ufLD/2yObgXAAJMBB5zNJXhjtFdSDt4gr9/9BVd45pydT3MRXA0v5Cl2w/ywZYDfPpVDvmFJcQ0iuDinnGM6xvPxT3jaBZdY/f5Orl2SEf+9O525qzMsGLhgI+8faB+c1Vft6OYIOXLDe6ZIrIGGONddJ23E61xkIjw2LX92Xs4n58u3EinVk0Yklj7oZBZR/L50Hv/YdWew5SUKu2aNeK6IR25tE88I7u0olGE8/dFmkVHMmFgB95av4+Hvt3bsaIUql5ZsZcO1gfKOMjXy0nbgSNl64tIoqpmOJbKABAVEcY/bx7KNTO+4I6Za3hr+vl0bNG42m1Ula37884UiK378wDo0S6GOy/qwrg+8fTv2Nzx+yCVmToykVdTM3lzXTa3jEpu8OMHq7I+UD8Z18P6QBnH+DIa6m7gYeAAnie3BVBggA/bjsfzjEY48IKq/qmK9a4HFgLDVDXVOw3rc0AKUIpnNNYyX/5CwaZV0yj+fVsK1/7jS7730moW/vA8YioMPy0uKWXVnsNnblBnHz2FCKQkteSXV/Tm0j7tSG7T1KW/wf8MSGhB/47Nmb0ig2kjbT7o+jJrxV4iw4WbhtngAeMcX84s7gV6qmpubXYsIuHADOBSIAtYLSKLKl7CEpFY7zFWllv8fQBV7S8ibYF3RWSYqobkfKTd2sYyY8oQvvPSan48bx3PTUuhoKiET70PyH20/SDHThXRKCKMC7vHce8l3Rnbuy1tYvyvL9CUEYk8+Pom1mYcYWhSK7fjBLz8wmIWrsmyPlDGcT61+wCO1WHfw4E0VU0HEJF5wNVAxfsdj+KZee+n5Zb1AZYCqOpBETmK5yxjVR1yBIXRPeJ4+Ko+/OatLVz5zOfsyjlBYXEpLZpE8q3e7bi0TztG92jj9+PrJwzswGOLtzF7RYYVi3pQ1gfqFhsuaxzmy0+WdGCZiCwGzky8oKpP1bBdRzyFpkwWMKL8CiIyBOikqotFpHyx2ABMEJG5QCdgqPfPkC0WALeMSib7yCk+3HqAm0ckMa5vO1KSWgbUdeqmjSK4dnBHXk3N5NdX9qFlhfYgxneqyszle+ndvhnx860DAAAbRUlEQVRDrQ+UcZgvxSLD+4ryvuqFiIQBTwG3VfLxv4HeQCqwF/iSSjrdisgdwB0AiYmhcb32wSt68+AVvd2OcU6mjEhk1oq9vLY2i9sv7OJ2nIC1Zu8RtlkfKNNAfBk6+zvwzJSnqvm12Hc2nrOBMgneZWVigX54zloA4oFFIjJBVVOB+8pWFJEvgZ2VZHseeB4gJSUlOCaDCAFlvwnPWZnB9y7obD/o6mjWir3ERlsfKNMwfJkpb1QdZ8pbDXQXkc7e0U2TgEVlH6rqMVVto6rJqpoMrAAmeEdDNRGRpt7jXQoU27MdwWXK8ETSD51keXqtxk0Yr4PHC1iyaT8Thyb4/X0qExx8udj9NHWYKU9Vi4HpwPt45r+Yr6pbROQREZlQw+ZtgbUisg34OTDNh5wmgHx7QHuaN45k9kp7XKcuXvxsNyWlyq32vIppII7OlKeqS4AlFZb9pop1Ly739R6gpy/HMIEpOjKciUMTePnLPeQcP23DPmvhyMlCZq3Yy4SBHfzi+RkTGnw5s/jGTHki8hNspjxTD6aMSKS4VJmfmlnzyuaM/3y5h/zCEn40ppvbUUwI8aVY3IlnZryymfIGYTPlmXrQNS6GUV1aM3dVBqWlNj7BF3kFRbz0xW7G942nR7tYt+OYEFJtsfA+hT1NVaeqajtVbauqN9f2aW5jqjJlRCJZR07x6Vc5bkcJCLOW7yWvoJjpY+2swjSsaouFqpYAUxooiwlBl/WNp3XTKLvR7YP8wmJe/Hw3F/eM88s52k1w8+Uy1Oci8qyIXCgiQ8pejiczISEqIowbh3Xio20H2H/slNtx/NrcVZkcPlnI3XZWYVzgS7EYBPQFHsEzEdJfgCedDGVCy+RhiSgwb5Xd6K7K6eISnv90FyO7tLKeWsYVvjzBPaamdYw5F4mtmzC6exzzVmdw99huAdXrqqEsXJPFgbzTPHXjILejmBDlyxPc7UTkRRF51/u+j4h8z/loJpRMGZHIgbzTLN1+0O0ofqeopJT/W7aLwYktOK9ra7fjmBDly69wL+F5CrusAc1O4MdOBTKh6ZJebWnXrJHd6K7EW+v3kXXkFNPHdLM+WsY1vhSLNqo6H8+MdWVtPHx6gtsYX0WEhzFpWCKffpVDRm5t+lUGt5JS5R/L0ujdvhlje9n82sY9vhSLkyLSGs9UqojISOo2GZIx1Zo0vBMCzF1tZxdl3t28n/Sck9w91s4qjLt8KRb34+kW21VEvgBmAnc7msqEpPbNG3NJ73YsSM2ksDgkZ9D9BlXl2aVpdI1ryvi+8W7HMSGuxmKhqmuBi4DzgB8AfVV1o9PBTGiaMiKRQycK+WDr125Hcd1H2w6y/evj3DWmG2FhdlZh3OXrGMXhwEBgCDBZRG5xLpIJZaO7x5HQsjGzV4T2pShV5ZmP0+jUqjETBtrkRsZ9vgydnYXnIbwLgGHeV4rDuUyICg8TJg9PZHl6LmkHT7gdxzWfpx1iQ+ZRfniRPXdi/IMv81mkAH1U1dqCmgZxY0on/vrhTuauyuDXV/ZxO44rnl2aRnyzaK4f2tHtKMYAvl2G2oxnfmxjGkRcbCMu6xfPwjVZFBSF3ijtVbsPs3L3Ye4Y3YVGEeFuxzEGqKZYiMjbIrIIaANsFZH3RWRR2cuXnYvIeBHZISJpIvKLata7XkRURFK87yNF5GUR2SQi20Tkwdr+xUxgmzo8kWOniliyab/bURrcsx+n0bppFJOHJ7odxZgzqrsMdU7NAr1zYcwALgWygNUiskhVt1ZYLxa4F1hZbvENQCNV7S8iTfAUq7ne6VZNCBjVtTVd2jRl9soMrhuS4HacBrMx6yif7szh5+N70TjKziqM/6jyzEJVPyl7AduBWO9rm3dZTYYDaaqarqqFwDzg6krWexT4M1BQ/vBAUxGJABoDhUCeL38hExxEhCkjElmz9wjb9ofOf/pnl6bRLDqCm0faWYXxL76MhroRWIXnt/0bgZUiMtGHfXcEyveczvIuK7/vIUAnVV1cYduFwElgP5ABPKmqhyvJdoeIpIpIak6OzbQWbK4fkkBURBhzQqRf1I6vj/PB1gN85/zOxEZHuh3HmG/w5Qb3L4Fhqnqrqt6C54zh1+d6YBEJA54CHqjk4+F4+k91ADoDD4hIl4orqerzqpqiqilxcXHnGsn4mZZNo7iyf3veWJfNydPFbsdx3IyP02gaFc53zk92O4oxZ/GlWISpavm+0bk+bpcNdCr3PsG7rEws0A9YJiJ7gJHAIu9N7inAe6pa5D32F9izHSFpyohETpwu5u0N+9yO4qjdh07yzsZ93DwqiRZNotyOY8xZfPmh/553JNRtInIbsBh414ftVgPdRaSziEQBk/D0mAJAVY+pahtVTVbVZGAFMEFVU/FcehoLICJN8RSS7bX4e5kgMTSpJT3bxQZ96/L/W5ZGZHgYt19w1gm0MX7Bl95QPwWeAwZ4X8+r6s982K4YmI5nLoxtwHxV3SIij4jIhBo2nwHEiMgWPEXnP9aPKjSJCFNHJrIp+xgbs466HccRWUfyeX1tNpOHJxIX28jtOMZUqsqhsyLSDWinql+o6uvA697lF4hIV1XdVdPOVXUJsKTCst9Use7F5b4+geeGujFcM7gjf1yyndkrMhgwsYXbcerdc5+kIwI/uMjOKoz/qu7M4mkqH656zPuZMQ2iWXQkVw/qwKIN+8grKHI7Tr06mFfAq6mZTByaQPvmjd2OY0yVqisW7VR1U8WF3mXJjiUyphJTRiRyqqiEN9dl17xyAPnXZ+kUl5Ry50Vd3Y5iTLWqKxbVne/br0CmQQ1IaEH/js2ZvSKDYOlpefhkIa+syODqQR1Jat3U7TjGVKu6YpEqIt+vuFBEbgfWOBfJmMpNHZHIjgPHWbP3iNtR6sV/vthNQXEJP7rYziqM/6uuWPwY+I6ILBORv3hfnwDfw9PLyZgGddXADsQ2igiKYbTHThXx0hd7GN83nu7tYt2OY0yNqusNdUBVzwN+B+zxvn6nqqNU1ea8NA2uaaMIrh3SkcWb9nPkZKHbcc7JrOV7OH66mLvGdHM7ijE+8eU5i49V9Rnva2lDhDKmKlNGJFJYXMpra7PcjlJn+YXFvPj5bsb2aku/js3djmOMT2y+RhNQesU3Y2hSS2avDNwb3XNWZnAkv8jOKkxAsWJhAs7UEYnsPnSS5bty3Y5SawVFJTz3aTrndW3N0KSWbscxxmdWLEzAuaJ/e1o0iQzIG90L1mSRc/w008faWYUJLFYsTMCJjgxn4pAE3t/yNQePF9S8gZ8oKinln8t2MSSxBaO6tHY7jjG1YsXCBKTJIxIpLlUWpAbOje4312WTffQUd4/tjoi4HceYWrFiYQJS17gYRnVpzdxVGZSU+v+N7pJS5R/LdtG3QzMu7mkTdZnAY8XCBKypIxPJOnKKT7/y/yl1F2/az+5DJ5k+ppudVZiAZMXCBKxxfeJpExPF7BX+faO7tFSZsTSNbm1juKxvvNtxjKkTKxYmYEVFhHFjSieWbj/AvqOn3I5Tpf9uO8COA8e5a0xXwsLsrMIEJkeLhYiMF5EdIpImIr+oZr3rRUS9828jIlNFZH25V6mIDHIyqwlMk4cnosCrqzPdjlIpVeXZj9NIbNWEqwZ0cDuOMXXmWLEQkXA806NeDvQBJotIn0rWi8XTmHBl2TJVna2qg1R1EDAN2K2q653KagJXp1ZNGN09jnmrMyguKXU7zlk+++oQG7OO8aOLuxIRbifyJnA5+d07HEhT1XRVLQTmAVdXst6jwJ+BqgbMT/Zua0ylpo5I5EDeaT7aftDtKGd5dmka7ZtHc92QBLejGHNOnCwWHYHy1wayvMvOEJEhQCdVXVzNfm4C5tZ/PBMsxvZqS3yzaL97ontlei6r9hzmB6O7EBVhZxUmsLn2HSwiYcBTwAPVrDMCyFfVzVV8foeIpIpIak6O/w+fNM6ICA9j0vBOfLozh4zcfLfjnPHsx2m0iYli0vBEt6MYc86cLBbZQKdy7xO8y8rEAv2AZSKyBxgJLCq7ye01iWrOKlT1eVVNUdWUuDh70CmUTRqWSHiYMHe1f5xdrM88ymdfHeL2C7sQHRnudhxjzpmTxWI10F1EOotIFJ4f/IvKPlTVY6raRlWTVTUZWAFMUNVUOHPmcSN2v8L4IL55NGN7tWX+6kwKi92/0f3s0jSaN47k5pFJbkcxpl44VixUtRiYDrwPbAPmq+oWEXlERCb4sIvRQKaqpjuV0QSXqSMSyT1ZyPtb3J3Icdv+PP677QDfOT+ZmEYRrmYxpr44+p2sqkuAJRWW/aaKdS+u8H4ZnktTxvhkdPc4Elo2ZvbKvVw10L1nGmZ8nEZMowhuOy/ZtQzG1DcbomGCRliYMGVEIivSD5N28IQrGXblnGDxpv1MG5VEiyZRrmQwxglWLExQuWFoJyLDhbmr3LnR/X/LdtEoIozvXdDZleMb4xQrFiaoxMU2YlzfeBauyaKgqKRBj515OJ831mUzeXgibWIaNeixjXGaFQsTdKaOSOTYqSIWb9zfoMf95ye7CBfhjtFdGvS4xjQEKxYm6Izq0poubZoye+XeBjvmgbwCFqRmcf3QBNo3b9xgxzWmoVixMEFHxHOje23GUbbtz2uQYz7/aTolqvzwoq4NcjxjGpoVCxOUJg5NICoijDkN0C8q98Rp5qzM4OqBHUhs3cTx4xnjBisWJii1aBLFlf3b88a6bE6eLnb0WP/+YjcFxSX8aIydVZjgZcXCBK2pIxM5cbqYRRv2OXaMY6eKmPnlXq7o155ubWMdO44xbrNiYYLWkMSW9IqP5ZUVe1FVR44x88s9HD9dbGcVJuhZsTBBS0SYOiKRLfvy2Jh1rN73f/J0MS9+sZtLerWlb4fm9b5/Y/yJFQsT1K4Z3JEmUeGO3OievXIvR/OLuGtst3rftzH+xoqFCWqx0ZFMGNiBRRv2cexUUb3tt6CohH99tpvzu7VmSGLLetuvMf7KioUJelNHJHGqqIQ312XXvLKP5qdmknP8NNPHdK+3fRrjz6xYmKDXP6E5AxKaM3tl/dzoLiwu5Z/LdpGS1JKRXVrVQ0Jj/J8VCxMSpo5IZOeBE6TuPXLO+3pzXTb7jhVw19huiEg9pDPG/zlaLERkvIjsEJE0EflFNetdLyJafv5tERkgIstFZIuIbBKRaCezmuB21cAOxDaKOOcb3cUlpfxjWRr9Ojbj4h4277sJHY4VCxEJB2YAlwN9gMki0qeS9WKBe4GV5ZZFAK8Ad6pqX+BioP7uTpqQ0yQqguuGdGTxpv0cPllY5/0s3rSfPbn5TB9jZxUmtDh5ZjEcSFPVdFUtBOYBV1ey3qPAn4GCcsvGARtVdQOAquaqasNOTmCCzpQRSRQWl/Lamqw6bV9aqsz4OI3ubWMY1ye+ntMZ49+cLBYdgcxy77O8y84QkSFAJ1VdXGHbHoCKyPsislZEfuZgThMiesbHkpLUkjmrMigtrf2N7g+2HmDngRNMH9uNsDA7qzChxbUb3CISBjwFPFDJxxHABcBU75/XisgllezjDhFJFZHUnJwcR/Oa4DB1ZCK7D51keXpurbZTVZ79+CuSWjfh2/3bO5TOGP/lZLHIBjqVe5/gXVYmFugHLBORPcBIYJH3JncW8KmqHlLVfGAJMKTiAVT1eVVNUdWUuDi72Whqdnm/9rRoElnrG92f7Mxhc3YeP7q4KxHhNojQhB4nv+tXA91FpLOIRAGTgEVlH6rqMVVto6rJqpoMrAAmqGoq8D7QX0SaeG92XwRsdTCrCRHRkeHcMDSB97d8zcHjBTVvgOes4pmlaXRoHs21gxMcTmiMf3KsWKhqMTAdzw/+bcB8Vd0iIo+IyIQatj2C5xLVamA9sLaS+xrG1Mnk4YkUlyoLUn270b0i/TBr9h7hBxd1JSrCzipMaIpwcuequgTPJaTyy35TxboXV3j/Cp7hs8bUqy5xMZzXtTVzVmZw50VdCa/hZvWMj9NoE9OIm4Z1qnY9Y4KZ/ZpkQtLUEUlkHz3FpzurHxixLuMIn6cd4o7RnYmODG+gdMb4HysWJiRd2qcdbWIaMbuGG90zPk6jRZNIpo5IaqBkxvgnKxYmJEVFhHHTsASWbj/AvqOnKl1ny75j/HfbQb57fmeaNnL0iq0xfs+KhQlZk4YlosC81ZmVfv6Pj3cR0yiCW0clN2guY/yRFQsTsjq1asJFPeKYtyqDopLSb3yWdvAESzbv55ZRSTRvEulSQmP8hxULE9Kmjkji4PHTfLTt4DeW/2NZGo0iwvjeBZ1dSmaMf7FiYULamJ5xtG8ezZxV/7vRnZGbz1vr9zFleBKtYxq5mM4Y/2HFwoS0iPAwJg1L5NOdOWTk5gPwz093ES7CHaO7uJzOGP9hxcKEvJuGdSI8TJizKoP9x06xMDWLG1ISiG9u820ZU8bGA5qQF988mkt6tWVBaiYnThdRosqdF3V1O5YxfsXOLIwBpo5MIvdkIa+syOCaQR3p1KqJ25GM8StWLIwBLuzWhk6tGiMCPxpjZxXGVGSXoYwBwsKER67uR0ZuPl3jYtyOY4zfsWJhjNeYnm3djmCM37LLUMYYY2pkxcIYY0yNrFgYY4ypkaPFQkTGi8gOEUkTkV9Us971IqIikuJ9nywip0Rkvff1TydzGmOMqZ5jN7hFJByYAVwKZAGrRWSRqm6tsF4scC+wssIudqnqIKfyGWOM8Z2TZxbDgTRVTVfVQmAecHUl6z0K/BkocDCLMcaYc+BksegIlJ9VJsu77AwRGQJ0UtXFlWzfWUTWicgnInJhZQcQkTtEJFVEUnNyqp9L2RhjTN25doNbRMKAp4AHKvl4P5CoqoOB+4E5ItKs4kqq+ryqpqhqSlxcnLOBjTEmhDn5UF420Knc+wTvsjKxQD9gmYgAxAOLRGSCqqYCpwFUdY2I7AJ6AKlVHWzNmjWHRGTvOeRtAxw6h+0bUiBlhcDKG0hZIbDyBlJWCKy855I1yZeVRFXruP8adiwSAewELsFTJFYDU1R1SxXrLwN+oqqpIhIHHFbVEhHpAnwG9FfVw46E9Rw/VVVTnNp/fQqkrBBYeQMpKwRW3kDKCoGVtyGyOnZmoarFIjIdeB8IB/6tqltE5BEgVVUXVbP5aOARESkCSoE7nSwUxhhjqudobyhVXQIsqbDsN1Wse3G5r18DXnMymzHGGN/ZE9z/87zbAWohkLJCYOUNpKwQWHkDKSsEVl7Hszp2z8IYY0zwsDMLY4wxNQrpYiEinUTkYxHZKiJbRORetzNVR0SiRWSViGzw5v2d25lqIiLh3ocr33E7S01EZI+IbPL2I6tymLY/EJEWIrJQRLaLyDYRGeV2pqqISM9yfd7Wi0ieiPzY7VxVEZH7vP9/bRaRuSIS7XamqojIvd6cW5z+Nw3py1Ai0h5or6prvT2q1gDXVOxf5S/E80BKU1U9ISKRwOfAvaq6wuVoVRKR+4EUoJmqXul2nuqIyB4gRVX9fmy9iLwMfKaqL4hIFNBEVY+6nasm3p5x2cAIVT2X56IcISId8fx/1UdVT4nIfGCJqr7kbrKziUg/PG2UhgOFwHt4Ro6mOXG8kD6zUNX9qrrW+/VxYBsVWpL4E/U44X0b6X35bbUXkQTg28ALbmcJJiLSHM/w8hcBVLUwEAqF1yV4moT6XaEoJwJo7H1WrAmwz+U8VekNrFTVfFUtBj4BrnPqYCFdLMoTkWRgMGd3v/Ur3ss664GDwIeq6s95nwZ+hudZmUCgwAciskZE7nA7TDU6AznAf7yX+F4QkaZuh/LRJGCu2yGqoqrZwJNABp62Q8dU9QN3U1VpM3ChiLQWkSbAFXyza0a9smIBiEgMnuc6fqyqeW7nqY6qlnhbtycAw72non5HRK4EDqrqGrez1MIFqjoEuBy4S0RGux2oChHAEOD/vP3TTgJVzhfjL7yXyyYAC9zOUhURaYmnO3ZnoAPQVERudjdV5VR1G56O3R/guQS1Hihx6nghXyy81/5fA2ar6utu5/GV97LDx8B4t7NU4Xxggvc+wDxgrIi84m6k6nl/q0RVDwJv4LkW7I+ygKxyZ5UL8RQPf3c5sFZVD7gdpBrfAnarao6qFgGvA+e5nKlKqvqiqg5V1dHAETwtlhwR0sXCe8P4RWCbqj7ldp6aiEiciLTwft0Yz8RS291NVTlVfVBVE1Q1Gc+lh6Wq6pe/oQGISFPvIAe8l3TG4TnN9zuq+jWQKSI9vYsuAfxyUEYFk/HjS1BeGcBIEWni/flwCZ57mX5JRNp6/0zEc79ijlPHcrTdRwA4H5gGbPLeBwB4yNumxB+1B172jigJA+arqt8PSQ0Q7YA3vB2QI4A5qvqeu5GqdTcw23tpJx34jst5quUtwJcCP3A7S3VUdaWILATWAsXAOvz7Se7XRKQ1UATc5eRAh5AeOmuMMcY3IX0ZyhhjjG+sWBhjjKmRFQtjjDE1smJhjDGmRlYsjDHG1MiKhQlJIqIi8pdy738iIr+t52N8p1yn1cJyHW3/VId9dRKRV+sznzG1YUNnTUgSkQI8vX+GqeohEfkJEKOqv3XoeHsIkI62xlTGzixMqCrG87DVfRU/EJGXRGRiufcnvH9eLCKfiMhbIpIuIn8SkaneOUY2iUhXXw8uIm1EZJGIbBSRL8t6fInI70XkZRFZISJfich3vcu7lT04KiIRIvJX7zwGG0XkR97lT4hnbpaNIvLnc/nHMaaiUH+C24S2GcBGEXm8FtsMxNMa+jCeJ6dfUNXh4pk4627A1wloHsXTXnqCiIwDXsIz7wdAfzz9iJoBa0VkcYVtf4inyd1AVS0RkVYi0g5P19G+qqplbWGMqS92ZmFClrfD8Ezgnlpstto7D8ppYBeejp8Am4DkWuznAmCWN8cHQIdybcbfVNUCb0PDT4FhFbb9FvBPVS3xbn8YT/EqBf4lItfi6URrTL2xYmFC3dPA94Dy80EU4/1/Q0TCgKhyn50u93Vpufel1N+ZesUbiTXeWPR2SE0B3gSuASqejRhzTqxYmJDm/a18Pp6CUWYPMNT79QQ8MxLWt8+AqQAi8i0gW1XLzgauEZFGIhIHXAhUnA/8Q+BOb0NJvJehYvFMXfsOnvswgx3IbEKY3bMwBv4CTC/3/l/AWyKyAc+kMk5c0vkN8G8R2Qic4JtdYzfjmSKzNfCwqh4oa5/u9RzQHc/9lmLg/4B3gNdFpBGeXwLvdyCzCWE2dNYYPyIivwcOqerTbmcxpjy7DGWMMaZGdmZhjDGmRnZmYYwxpkZWLIwxxtTIioUxxpgaWbEwxhhTIysWxhhjamTFwhhjTI3+H+1egghQG1s7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Show graph\n",
    "limit=10; start=2; step=1;\n",
    "x = range(start, limit, step)\n",
    "plt.plot(x, coherence_values)\n",
    "plt.xlabel(\"Num Topics\")\n",
    "plt.ylabel(\"Coherence score\")\n",
    "plt.legend((\"coherence_values\"), loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num Topics = 2  has Coherence Value of 0.5107\n",
      "Num Topics = 3  has Coherence Value of 0.4903\n",
      "Num Topics = 4  has Coherence Value of 0.4956\n",
      "Num Topics = 5  has Coherence Value of 0.4533\n",
      "Num Topics = 6  has Coherence Value of 0.4888\n",
      "Num Topics = 7  has Coherence Value of 0.53\n",
      "Num Topics = 8  has Coherence Value of 0.5066\n",
      "Num Topics = 9  has Coherence Value of 0.5313\n"
     ]
    }
   ],
   "source": [
    "# Print the coherence scores\n",
    "for m, cv in zip(x, coherence_values):\n",
    "    print(\"Num Topics =\", m, \" has Coherence Value of\", round(cv, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Si el puntaje de coherencia parece seguir aumentando, puede tener más sentido elegir el modelo que dio el puntaje más alto antes de aplanarse. Este es exactamente el caso aquí.\n",
    "\n",
    "Así que para los pasos siguientes voy a elegir el modelo con 20 temas en sí."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(0,\n",
      "  '0.015*\"key\" + 0.009*\"ripem\" + 0.005*\"public\" + 0.004*\"get\" + 0.004*\"write\" '\n",
      "  '+ 0.004*\"rsa\" + 0.004*\"line\" + 0.004*\"car\" + 0.004*\"organization\" + '\n",
      "  '0.004*\"pgp\"'),\n",
      " (1,\n",
      "  '0.008*\"state\" + 0.007*\"law\" + 0.006*\"right\" + 0.006*\"people\" + '\n",
      "  '0.006*\"government\" + 0.005*\"israel\" + 0.005*\"gun\" + 0.004*\"say\" + '\n",
      "  '0.004*\"israeli\" + 0.004*\"turkish\"'),\n",
      " (2,\n",
      "  '0.017*\"not\" + 0.010*\"do\" + 0.008*\"write\" + 0.008*\"would\" + 0.008*\"say\" + '\n",
      "  '0.008*\"think\" + 0.008*\"be\" + 0.008*\"god\" + 0.007*\"people\" + 0.006*\"line\"'),\n",
      " (3,\n",
      "  '0.028*\"_\" + 0.007*\"cx\" + 0.007*\"c\" + 0.005*\"mv\" + 0.004*\"gm\" + 0.004*\"md\" + '\n",
      "  '0.003*\"det\" + 0.003*\"scx\" + 0.002*\"tor\" + 0.002*\"ed\"'),\n",
      " (4,\n",
      "  '0.015*\"line\" + 0.013*\"organization\" + 0.008*\"post\" + 0.008*\"university\" + '\n",
      "  '0.007*\"host\" + 0.007*\"nntp\" + 0.006*\"write\" + 0.006*\"not\" + 0.006*\"get\" + '\n",
      "  '0.006*\"run\"')]\n"
     ]
    }
   ],
   "source": [
    "# Select the model and print the topics\n",
    "optimal_model = model_list[3]\n",
    "model_topics = optimal_model.show_topics(formatted=False)\n",
    "pprint(optimal_model.print_topics(num_words=10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 16. Encontrar el tópico dominante en cada frase o documento\n",
    "\n",
    "Una de las aplicaciones prácticas del modelo de tópicos es determinar de qué tópico trata un documento dado.\n",
    "\n",
    "Para encontrar eso, encontramos el número del tópico que tiene el porcentaje más alto de contribución en ese documento.\n",
    "\n",
    "La función format_topics_sentences() agrega esta información en una tabla presentable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Document_No</th>\n",
       "      <th>Dominant_Topic</th>\n",
       "      <th>Topic_Perc_Contrib</th>\n",
       "      <th>Keywords</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5422</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>From: (wheres my thing) Subject: WHAT car is t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.9321</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>From: (Guy Kuo) Subject: SI Clock Poll - Final...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6085</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>From: (Irwin Arnstein) Subject: Re: Recommenda...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.9908</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>From: (Tsung-Kun Chen) Subject: ** Software fo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.6702</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>From: (Don A.B. Lindbergh) Subject: Diamond SS...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.7441</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>From: (Robert Loper) Subject: Re: SHO and SC N...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.9378</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>From: (Kim Richard Man) Subject: SyQuest 44M c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.9885</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>From: (Kirtley Wilson) Subject: Mirosoft Offic...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.6934</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>Subject: Re: Dont more innocents die without t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.5050</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>From: (Jon Livesey) Subject: Re: Genocide is C...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Document_No  Dominant_Topic  Topic_Perc_Contrib  \\\n",
       "0            0             0.0              0.5422   \n",
       "1            1             4.0              0.9321   \n",
       "2            2             0.0              0.6085   \n",
       "3            3             4.0              0.9908   \n",
       "4            4             4.0              0.6702   \n",
       "5            5             4.0              0.7441   \n",
       "6            6             4.0              0.9378   \n",
       "7            7             4.0              0.9885   \n",
       "8            8             2.0              0.6934   \n",
       "9            9             2.0              0.5050   \n",
       "\n",
       "                                            Keywords  \\\n",
       "0  key, ripem, public, get, write, rsa, line, car...   \n",
       "1  line, organization, post, university, host, nn...   \n",
       "2  key, ripem, public, get, write, rsa, line, car...   \n",
       "3  line, organization, post, university, host, nn...   \n",
       "4  line, organization, post, university, host, nn...   \n",
       "5  line, organization, post, university, host, nn...   \n",
       "6  line, organization, post, university, host, nn...   \n",
       "7  line, organization, post, university, host, nn...   \n",
       "8  not, do, write, would, say, think, be, god, pe...   \n",
       "9  not, do, write, would, say, think, be, god, pe...   \n",
       "\n",
       "                                                Text  \n",
       "0  From: (wheres my thing) Subject: WHAT car is t...  \n",
       "1  From: (Guy Kuo) Subject: SI Clock Poll - Final...  \n",
       "2  From: (Irwin Arnstein) Subject: Re: Recommenda...  \n",
       "3  From: (Tsung-Kun Chen) Subject: ** Software fo...  \n",
       "4  From: (Don A.B. Lindbergh) Subject: Diamond SS...  \n",
       "5  From: (Robert Loper) Subject: Re: SHO and SC N...  \n",
       "6  From: (Kim Richard Man) Subject: SyQuest 44M c...  \n",
       "7  From: (Kirtley Wilson) Subject: Mirosoft Offic...  \n",
       "8  Subject: Re: Dont more innocents die without t...  \n",
       "9  From: (Jon Livesey) Subject: Re: Genocide is C...  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def format_topics_sentences(ldamodel=lda_model, corpus=corpus, texts=data):\n",
    "    # Init output\n",
    "    sent_topics_df = pd.DataFrame()\n",
    "\n",
    "    # Get main topic in each document\n",
    "    for i, row in enumerate(ldamodel[corpus]):\n",
    "        #row = sorted(row, key=lambda x: (x[1]), reverse=True)\n",
    "        row = sorted(row[0], key=lambda x: (x[1]), reverse=True)\n",
    "        # Get the Dominant topic, Perc Contribution and Keywords for each document\n",
    "        for j, (topic_num, prop_topic) in enumerate(row):\n",
    "            if j == 0:  # => dominant topic\n",
    "                wp = ldamodel.show_topic(topic_num)\n",
    "                topic_keywords = \", \".join([word for word, prop in wp])\n",
    "                sent_topics_df = sent_topics_df.append(pd.Series([int(topic_num), round(prop_topic,4), topic_keywords]), ignore_index=True)\n",
    "            else:\n",
    "                break\n",
    "    sent_topics_df.columns = ['Dominant_Topic', 'Perc_Contribution', 'Topic_Keywords']\n",
    "\n",
    "    # Add original text to the end of the output\n",
    "    contents = pd.Series(texts)\n",
    "    sent_topics_df = pd.concat([sent_topics_df, contents], axis=1)\n",
    "    return(sent_topics_df)\n",
    "\n",
    "\n",
    "df_topic_sents_keywords = format_topics_sentences(ldamodel=optimal_model, corpus=corpus, texts=data)\n",
    "\n",
    "# Format\n",
    "df_dominant_topic = df_topic_sents_keywords.reset_index()\n",
    "df_dominant_topic.columns = ['Document_No', 'Dominant_Topic', 'Topic_Perc_Contrib', 'Keywords', 'Text']\n",
    "\n",
    "# Show\n",
    "df_dominant_topic.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 17. Encontrar el documento más representativo para cada tópico\n",
    "\n",
    "A veces las palabras claves del tópico pueden no ser suficientes para entender de qué se trata el tema. Por lo tanto, para ayudar a comprender el tópico, se puede encontrar los documentos a los que un tópico dado ha contribuido más y deducir el tema leyendo ese documento."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Topic_Num</th>\n",
       "      <th>Topic_Perc_Contrib</th>\n",
       "      <th>Keywords</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.9258</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>From: (michael mchugh) Subject: 45 rpm Singles...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.9071</td>\n",
       "      <td>state, law, right, people, government, israel,...</td>\n",
       "      <td>From: (Serdar Argic) Subject: Letter to Presid...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.9898</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>From: (Amy Mossman) Subject: Re: Is MSG sensit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.9926</td>\n",
       "      <td>_, cx, c, mv, gm, md, det, scx, tor, ed</td>\n",
       "      <td>From: (Clive Mitchell) Subject: Dataproducts L...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.9916</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>From: (Daniel M. Coleman) Subject: Re: Do the ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Topic_Num  Topic_Perc_Contrib  \\\n",
       "0        0.0              0.9258   \n",
       "1        1.0              0.9071   \n",
       "2        2.0              0.9898   \n",
       "3        3.0              0.9926   \n",
       "4        4.0              0.9916   \n",
       "\n",
       "                                            Keywords  \\\n",
       "0  key, ripem, public, get, write, rsa, line, car...   \n",
       "1  state, law, right, people, government, israel,...   \n",
       "2  not, do, write, would, say, think, be, god, pe...   \n",
       "3            _, cx, c, mv, gm, md, det, scx, tor, ed   \n",
       "4  line, organization, post, university, host, nn...   \n",
       "\n",
       "                                                Text  \n",
       "0  From: (michael mchugh) Subject: 45 rpm Singles...  \n",
       "1  From: (Serdar Argic) Subject: Letter to Presid...  \n",
       "2  From: (Amy Mossman) Subject: Re: Is MSG sensit...  \n",
       "3  From: (Clive Mitchell) Subject: Dataproducts L...  \n",
       "4  From: (Daniel M. Coleman) Subject: Re: Do the ...  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Group top 5 sentences under each topic\n",
    "sent_topics_sorteddf_mallet = pd.DataFrame()\n",
    "\n",
    "sent_topics_outdf_grpd = df_topic_sents_keywords.groupby('Dominant_Topic')\n",
    "\n",
    "for i, grp in sent_topics_outdf_grpd:\n",
    "    sent_topics_sorteddf_mallet = pd.concat([sent_topics_sorteddf_mallet, \n",
    "                                             grp.sort_values(['Perc_Contribution'], ascending=[0]).head(1)], \n",
    "                                            axis=0)\n",
    "\n",
    "# Reset Index    \n",
    "sent_topics_sorteddf_mallet.reset_index(drop=True, inplace=True)\n",
    "\n",
    "# Format\n",
    "sent_topics_sorteddf_mallet.columns = ['Topic_Num', \"Topic_Perc_Contrib\", \"Keywords\", \"Text\"]\n",
    "\n",
    "# Show\n",
    "sent_topics_sorteddf_mallet.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 18. Distribución de tópicos en el dataset\n",
    "\n",
    "Finalmente, queremos entender el volumen y la distribución de los tópicos para juzgar la amplitud de la discusión. La siguiente tabla expone esa información."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Dominant_Topic</th>\n",
       "      <th>Topic_Keywords</th>\n",
       "      <th>Num_Documents</th>\n",
       "      <th>Perc_Documents</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>125.0</td>\n",
       "      <td>0.125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>86.0</td>\n",
       "      <td>0.086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>288.0</td>\n",
       "      <td>0.288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>487.0</td>\n",
       "      <td>0.487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1.0</td>\n",
       "      <td>state, law, right, people, government, israel,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>1.0</td>\n",
       "      <td>state, law, right, people, government, israel,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>1.0</td>\n",
       "      <td>state, law, right, people, government, israel,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>970</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>971</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>972</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>973</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>974</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>975</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>977</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>978</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>979</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>980</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>981</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>982</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>983</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>984</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>985</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>986</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>987</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>988</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>989</th>\n",
       "      <td>1.0</td>\n",
       "      <td>state, law, right, people, government, israel,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>990</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>991</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>992</th>\n",
       "      <td>2.0</td>\n",
       "      <td>not, do, write, would, say, think, be, god, pe...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>993</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>994</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>1.0</td>\n",
       "      <td>state, law, right, people, government, israel,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>4.0</td>\n",
       "      <td>line, organization, post, university, host, nn...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>0.0</td>\n",
       "      <td>key, ripem, public, get, write, rsa, line, car...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Dominant_Topic                                     Topic_Keywords  \\\n",
       "0               0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "1               4.0  line, organization, post, university, host, nn...   \n",
       "2               0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "3               4.0  line, organization, post, university, host, nn...   \n",
       "4               4.0  line, organization, post, university, host, nn...   \n",
       "5               4.0  line, organization, post, university, host, nn...   \n",
       "6               4.0  line, organization, post, university, host, nn...   \n",
       "7               4.0  line, organization, post, university, host, nn...   \n",
       "8               2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "9               2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "10              4.0  line, organization, post, university, host, nn...   \n",
       "11              0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "12              0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "13              0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "14              0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "15              0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "16              2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "17              1.0  state, law, right, people, government, israel,...   \n",
       "18              4.0  line, organization, post, university, host, nn...   \n",
       "19              4.0  line, organization, post, university, host, nn...   \n",
       "20              2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "21              4.0  line, organization, post, university, host, nn...   \n",
       "22              4.0  line, organization, post, university, host, nn...   \n",
       "23              1.0  state, law, right, people, government, israel,...   \n",
       "24              0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "25              2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "26              1.0  state, law, right, people, government, israel,...   \n",
       "27              2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "28              4.0  line, organization, post, university, host, nn...   \n",
       "29              4.0  line, organization, post, university, host, nn...   \n",
       "..              ...                                                ...   \n",
       "970             2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "971             4.0  line, organization, post, university, host, nn...   \n",
       "972             2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "973             2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "974             2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "975             4.0  line, organization, post, university, host, nn...   \n",
       "976             4.0  line, organization, post, university, host, nn...   \n",
       "977             0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "978             2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "979             2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "980             4.0  line, organization, post, university, host, nn...   \n",
       "981             4.0  line, organization, post, university, host, nn...   \n",
       "982             2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "983             0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "984             2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "985             4.0  line, organization, post, university, host, nn...   \n",
       "986             4.0  line, organization, post, university, host, nn...   \n",
       "987             4.0  line, organization, post, university, host, nn...   \n",
       "988             2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "989             1.0  state, law, right, people, government, israel,...   \n",
       "990             4.0  line, organization, post, university, host, nn...   \n",
       "991             4.0  line, organization, post, university, host, nn...   \n",
       "992             2.0  not, do, write, would, say, think, be, god, pe...   \n",
       "993             0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "994             4.0  line, organization, post, university, host, nn...   \n",
       "995             4.0  line, organization, post, university, host, nn...   \n",
       "996             4.0  line, organization, post, university, host, nn...   \n",
       "997             1.0  state, law, right, people, government, israel,...   \n",
       "998             4.0  line, organization, post, university, host, nn...   \n",
       "999             0.0  key, ripem, public, get, write, rsa, line, car...   \n",
       "\n",
       "     Num_Documents  Perc_Documents  \n",
       "0            125.0           0.125  \n",
       "1             86.0           0.086  \n",
       "2            288.0           0.288  \n",
       "3             14.0           0.014  \n",
       "4            487.0           0.487  \n",
       "5              NaN             NaN  \n",
       "6              NaN             NaN  \n",
       "7              NaN             NaN  \n",
       "8              NaN             NaN  \n",
       "9              NaN             NaN  \n",
       "10             NaN             NaN  \n",
       "11             NaN             NaN  \n",
       "12             NaN             NaN  \n",
       "13             NaN             NaN  \n",
       "14             NaN             NaN  \n",
       "15             NaN             NaN  \n",
       "16             NaN             NaN  \n",
       "17             NaN             NaN  \n",
       "18             NaN             NaN  \n",
       "19             NaN             NaN  \n",
       "20             NaN             NaN  \n",
       "21             NaN             NaN  \n",
       "22             NaN             NaN  \n",
       "23             NaN             NaN  \n",
       "24             NaN             NaN  \n",
       "25             NaN             NaN  \n",
       "26             NaN             NaN  \n",
       "27             NaN             NaN  \n",
       "28             NaN             NaN  \n",
       "29             NaN             NaN  \n",
       "..             ...             ...  \n",
       "970            NaN             NaN  \n",
       "971            NaN             NaN  \n",
       "972            NaN             NaN  \n",
       "973            NaN             NaN  \n",
       "974            NaN             NaN  \n",
       "975            NaN             NaN  \n",
       "976            NaN             NaN  \n",
       "977            NaN             NaN  \n",
       "978            NaN             NaN  \n",
       "979            NaN             NaN  \n",
       "980            NaN             NaN  \n",
       "981            NaN             NaN  \n",
       "982            NaN             NaN  \n",
       "983            NaN             NaN  \n",
       "984            NaN             NaN  \n",
       "985            NaN             NaN  \n",
       "986            NaN             NaN  \n",
       "987            NaN             NaN  \n",
       "988            NaN             NaN  \n",
       "989            NaN             NaN  \n",
       "990            NaN             NaN  \n",
       "991            NaN             NaN  \n",
       "992            NaN             NaN  \n",
       "993            NaN             NaN  \n",
       "994            NaN             NaN  \n",
       "995            NaN             NaN  \n",
       "996            NaN             NaN  \n",
       "997            NaN             NaN  \n",
       "998            NaN             NaN  \n",
       "999            NaN             NaN  \n",
       "\n",
       "[1000 rows x 4 columns]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of Documents for Each Topic\n",
    "topic_counts = df_topic_sents_keywords['Dominant_Topic'].value_counts()\n",
    "\n",
    "# Percentage of Documents for Each Topic\n",
    "topic_contribution = round(topic_counts/topic_counts.sum(), 4)\n",
    "\n",
    "# Topic Number and Keywords\n",
    "topic_num_keywords = df_topic_sents_keywords[['Dominant_Topic', 'Topic_Keywords']]\n",
    "\n",
    "# Concatenate Column wise\n",
    "df_dominant_topics = pd.concat([topic_num_keywords, topic_counts, topic_contribution], axis=1)\n",
    "\n",
    "# Change Column names\n",
    "df_dominant_topics.columns = ['Dominant_Topic', 'Topic_Keywords', 'Num_Documents', 'Perc_Documents']\n",
    "\n",
    "# Show\n",
    "df_dominant_topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II - Trabajo práctico\n",
    "\n",
    "La carpeta Drive siguiente contiene una serie de datasets: https://drive.google.com/drive/folders/14bt3oF4tibthkxQee0-a_esveoPwrZtd?usp=sharing\n",
    "\n",
    "Cada dataset contiene noticias de prensa publicadas por medios chilenos entre Febrero 2017 y Febrero 2018. Cada dataset fue generado buscando una palabra clave específica, por ejemplo :'feminismo', 'corrupción', 'educación', 'mapuche', 'tecnología', etc.\n",
    "\n",
    "El trabajo práctico consiste en:\n",
    "- seleccionar 1 dataset. Por ejemplo: 'tecnología'\n",
    "- aplicar una metodología basada en LDA para conocer cuáles son los tópicos conversados por los medios cuándo se habla de 'tecnología'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}