Movie Review Sentiment Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#All the necessary imports\n",
    "import numpy as np\n",
    "import nltk\n",
    "import pandas as pd\n",
    "import os\n",
    "import re\n",
    "import csv\n",
    "import gzip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This is an example of a movie review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Given the title, this first follow-up to QUARTET (1948) obviously reduces the number of W. Somerset Maugham stories which comprise the film. The author still turns up to introduce the episodes, but there’s no epilogue this time around; by the way, while the script of the original compendium gave sole credit to R.C. Sheriff, here Maugham himself also lent a hand in the adaptation, as well as Noel Langley (though it’s unclear whether they contributed one segment each or else worked in unison). As can be expected, much of the crew of QUARTET has been retained for the second installment – though this also extends to at least three cast members, namely Naunton Wayne, Wilfrid Hyde-White and Felix Aylmer (the last two had bit parts in the episode from QUARTET entitled “The Colonel’s Lady”). While TRIO ultimately emerges to be a lesser achievement than its predecessor (slightly unbalanced by the third story which takes up more than half the running-time), it’s still done with the utmost care, acted with verve by a stellar cast and is solidly enjoyable into the bargain.<br /><br />“The Verger” tells of a church sexton (James Hayter) – for which the story’s title is another word – who’s dismissed after 17 years of service by the new parish priest (Michael Hordern) simply because he’s illiterate. Rather than rest on his laurels, despite his age, he not only takes a wife (his landlady, played by Kathleen Harrison) but opens a tobacconist shop strategically placed in a lengthy stretch of road where no such service is offered – and, with business flourishing, this is developed into a whole chain. The last scene, then, sees him pay a visit to bank manager Felix Aylmer who, not only is surprised to learn of Hayter’s lack of education, but is prompted to ask him what his other interests were – to which the wealthy (and respected) tobacconist replies, with some measure of irony, that he had the calling to be a verger! <br /><br />The second episode, “Mr. Know-All”, is the shortest but also perhaps the most engaging: a voyage at sea is utterly beleaguered by the insufferable presence of a pompous young man (Nigel Patrick), British despite his foreign-sounding name of Kelada, who professes to be an authority on virtually every subject under the sun. Naunton Wayne and Wilfrid Hyde-White are the two passengers who have to put up with him the most – the latter because he shares a cabin with the man and the former in view of Patrick’s attentions to his pretty wife (Anne Crawford). During a fancy-dress party, however, the passengers decide to enact their ‘revenge’ on Kelada by having one of them impersonate him (a jest which he naturally doesn’t appreciate)!; still, it’s here that he contrives to show a decent side to his character – told by Crawford that the necklace she’s wearing is an imitation, Wayne challenges Patrick to name its price…but the latter realizes immediately that it’s the genuine article and that this would compromise Crawford’s position if he were to tell, so Kelada allows himself to be publicly ridiculed rather than expose the fact that the woman probably has a secret admirer! <br /><br />As can also be deduced from the title, “Sanatorium” deals with the myriad patients at such a place – run by Andre' Morell; the protagonist is a new intern, Roland Culver, who wistfully observes the various goings-on. The narrative, in fact, highlights in particular three separate strands of plot – one humorous (the ‘feud’ between two aged Scots long resident at the sanatorium, played by Finlay Currie and John Laurie), one melodramatic (the erratic relationship between disgruntled patient Raymond Huntley and long-suffering but devoted wife Betty Ann Davies) and one bittersweet (the romance between naïve but charming Jean Simmons and dashing cad Michael Rennie which, in spite of having pretty much everything against it including the fact that Morell has diagnosed Simmons as a ‘lifer’ while Rennie only has a few years left to him, leads the couple to the altar).\n"
     ]
    }
   ],
   "source": [
    "f = open('./aclImdb/train/pos/10327_7.txt', encoding=\"utf8\")\n",
    "print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Putting the train and test data into a pandas dataframe. It shuffles the data as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(                                                    text  sentiment\n",
       " 0      I tend to like character-driven films. I also ...          0\n",
       " 1      Let's cut a long story short. I loved every mi...          1\n",
       " 2      I loved the first Little Mermaid. I know the s...          0\n",
       " 3      This is a tongue in cheek movie from the very ...          1\n",
       " 4      While i was the video store i was browsing thr...          1\n",
       " ...                                                  ...        ...\n",
       " 24995  I used to always love the bill because of its ...          0\n",
       " 24996  This is an excellent film about a traditional ...          1\n",
       " 24997  Sweet romantic drama/comedy about Stewart and ...          1\n",
       " 24998  In the early 1970s, many of us who had embrace...          1\n",
       " 24999  I don't know where to start; the acting, the s...          0\n",
       " \n",
       " [25000 rows x 2 columns],\n",
       "                                                     text  sentiment\n",
       " 0      This could have been a good movie if more thin...          0\n",
       " 1      It starts really interesting - the story devel...          0\n",
       " 2      Life was going great for New York City adverti...          1\n",
       " 3      This may just be the worst movie of all time. ...          0\n",
       " 4      As I stated earlier this year, in my review of...          1\n",
       " ...                                                  ...        ...\n",
       " 24995  The film someone had to make.<br /><br />Waco:...          1\n",
       " 24996  After hearing that some of the people behind t...          0\n",
       " 24997  Who can ask for more? Taking my 2 and 4 year o...          1\n",
       " 24998  I went to see this movie not expecting much, b...          1\n",
       " 24999  An example of genius filmaking. The epic story...          1\n",
       " \n",
       " [25000 rows x 2 columns])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "directory = \"./aclImdb\" #Make sure you put the data folder in the same directory as this jupyter notebook file\n",
    "labeledData = {}\n",
    "for i in [\"train\", \"test\"]:\n",
    "    labeledData[i] = []\n",
    "    for sentiment in [\"pos\", \"neg\"]:\n",
    "        score = 1 if sentiment == \"pos\" else 0\n",
    "        path = os.path.join(directory, i, sentiment)\n",
    "        for filename in os.listdir(path):\n",
    "            with open(os.path.join(path, filename), encoding=\"utf8\") as f:\n",
    "                labeledData[i].append([f.read(), score])  #Initially adds them to separate lists\n",
    "\n",
    "np.random.shuffle(labeledData[\"train\"]) #Shuffling\n",
    "labeledData[\"train\"] = pd.DataFrame(labeledData[\"train\"], columns = ['text', 'sentiment']) #Putting them in a dataframe\n",
    "np.random.shuffle(labeledData[\"test\"])\n",
    "labeledData[\"test\"] = pd.DataFrame(labeledData[\"test\"], columns = ['text', 'sentiment'])\n",
    "labeledData[\"train\"], labeledData[\"test\"] #Prints out both pandas dataframes\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = labeledData[\"train\"].to_csv(r'C:\\Users\\mdodda3-gtri\\Untitled Folder\\train.csv.gz', index = None, header=False, compression='gzip')\n",
    "test = labeledData[\"test\"].to_csv(r'C:\\Users\\mdodda3-gtri\\Untitled Folder\\test.csv.gz', index = None, header=False, compression='gzip')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# train1 = labeledData[\"train\"].iloc[:6250].to_csv (r'C:\\Users\\mdodda3-gtri\\Untitled Folder\\train1.csv.gz', index = None, header=False, compression='gzip')\n",
    "# train2 = labeledData[\"train\"].iloc[6250:12500].to_csv (r'C:\\Users\\mdodda3-gtri\\Untitled Folder\\train2.csv.gz', index = None, header=False, compression='gzip')\n",
    "# train3 = labeledData[\"train\"].iloc[12500:18750].to_csv (r'C:\\Users\\mdodda3-gtri\\Untitled Folder\\train3.csv.gz', index = None, header=False, compression='gzip')\n",
    "# train4 = labeledData[\"train\"].iloc[18750:].to_csv (r'C:\\Users\\mdodda3-gtri\\Untitled Folder\\train4.csv.gz', index = None, header=False, compression='gzip')\n",
    "\n",
    "# test1 = labeledData[\"test\"].iloc[:6250].to_csv (r'C:\\Users\\mdodda3-gtri\\Untitled Folder\\test1.csv.gz', index = None, header=False, compression='gzip')\n",
    "# test2 = labeledData[\"test\"].iloc[6250:12500].to_csv (r'C:\\Users\\mdodda3-gtri\\Untitled Folder\\test2.csv.gz', index = None, header=False, compression='gzip')\n",
    "# test3 = labeledData[\"test\"].iloc[12500:18750].to_csv (r'C:\\Users\\mdodda3-gtri\\Untitled Folder\\test3.csv.gz', index = None, header=False, compression='gzip')\n",
    "# test4 = labeledData[\"test\"].iloc[18750:].to_csv (r'C:\\Users\\mdodda3-gtri\\Untitled Folder\\test4.csv.gz', index = None, header=False, compression='gzip')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first column contains the movie reviews in separated rows.\n",
    "The second column indicates whether the review is a positive or negative review. \n",
    "A positive reivew has 7-10 stars, A negative review has 1-4 stars. 5-6 stars are disregarded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        0\n",
       "1        1\n",
       "2        0\n",
       "3        1\n",
       "4        1\n",
       "        ..\n",
       "24995    0\n",
       "24996    1\n",
       "24997    1\n",
       "24998    1\n",
       "24999    0\n",
       "Name: sentiment, Length: 25000, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000,)\n",
      "(25000,)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(1, 50000)"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "total = np.vstack((labeledData[\"train\"][\"text\"], labeledData[\"test\"][\"text\"]))\n",
    "#total = np.array([labeledData[\"test\"][\"text\"]])\n",
    "for col in total:\n",
    "    print(col.shape)\n",
    "total.reshape(1,total.shape[1] * total.shape[0]).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(pd.DataFrame(labeledData[\"train\"][\"text\"].values))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk import  word_tokenize\n",
    "from nltk.stem import WordNetLemmatizer, PorterStemmer, SnowballStemmer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "stemmer =SnowballStemmer(\"english\")\n",
    "analyzer = CountVectorizer().build_analyzer()\n",
    "\n",
    "def stemmed_words(doc):\n",
    "    return (stemmer.stem(w) for w in analyzer(doc))\n",
    "class LemmaTokenizer(object):\n",
    "    def __init__(self):\n",
    "        self.wnl = WordNetLemmatizer()\n",
    "    def __call__(self, articles):\n",
    "        return [self.wnl.lemmatize(t) for t in word_tokenize(articles)]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting the movie reviews into Specific Bag of Words Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'CountVectorizer' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-23-119291a539e1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m \u001b[0;31m#how each review will be vectorized\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 18\u001b[0;31m vectorizer = CountVectorizer(stop_words=stop_words, #These stop words are removed\n\u001b[0m\u001b[1;32m     19\u001b[0m                              \u001b[0mbinary\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;31m# if it contains in list it's 1, else it is 0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m                              \u001b[0mngram_range\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;31m#contains pairs of words as well\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'CountVectorizer' is not defined"
     ]
    }
   ],
   "source": [
    "#sklearn imports\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.svm import LinearSVC, NuSVC, SVC\n",
    "from sklearn.linear_model import OrthogonalMatchingPursuit, LogisticRegression, SGDClassifier, PassiveAggressiveClassifier, Perceptron, RidgeClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.naive_bayes import MultinomialNB, BernoulliNB, ComplementNB\n",
    "from sklearn.neighbors import KNeighborsClassifier, NearestCentroid\n",
    "from sklearn.feature_extraction import stop_words as stopwords\n",
    "\n",
    "\n",
    "\n",
    "#Stop word list created to be used\n",
    "stop_words = ['in', 'of', 'at', 'a', 'the']\n",
    "\n",
    "#how each review will be vectorized\n",
    "vectorizer = CountVectorizer(stop_words=stop_words, #These stop words are removed\n",
    "                             binary=True, # if it contains in list it's 1, else it is 0\n",
    "                             ngram_range=(1,2), #contains pairs of words as well\n",
    "                            \n",
    "                            )\n",
    "newVect = CountVectorizer (tokenizer = LemmaTokenizer(),\n",
    "                           stop_words=stop_words,\n",
    "                           binary=True,\n",
    "                           ngram_range=(1,2))\n",
    "\n",
    "\n",
    "\n",
    "x_train = vectorizer.fit_transform(labeledData[\"train\"][\"text\"]) #reviews from train dataframe are vectorized\n",
    "x_test = vectorizer.transform(labeledData[\"test\"][\"text\"])  #reviews from test dataframe are vectorized\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.preprocessing.text import Tokenizer, one_hot, text_to_word_sequence\n",
    "import keras\n",
    "import gensim\n",
    "train_data = labeledData[\"train\"][\"text\"]\n",
    "test_data = labeledData[\"test\"][\"text\"]\n",
    "\n",
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(train_data)\n",
    "x_train = tokenizer.texts_to_sequences(train_data)\n",
    "x_test = tokenizer.texts_to_sequences(test_data)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "88582"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#x_train\n",
    "#x_train = np.array([np.array(xi) for xi in x_train])\n",
    "#x_test = np.array([np.array(xi) for xi in x_test])\n",
    "len(tokenizer.word_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (None, 200, 300)          26574900  \n",
      "_________________________________________________________________\n",
      "lstm_3 (LSTM)                (None, 150)               270600    \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 151       \n",
      "=================================================================\n",
      "Total params: 26,845,651\n",
      "Trainable params: 26,845,651\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "Xencoded = keras.preprocessing.sequence.pad_sequences(x_train, maxlen=200, padding='post')\n",
    "XencodedTest = keras.preprocessing.sequence.pad_sequences(x_test, maxlen=200, padding = 'post')\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "embedding = keras.layers.embeddings.Embedding(input_dim=len(tokenizer.word_index)+1, output_dim=300, input_length=200, trainable=True, mask_zero=True)\n",
    "model.add(embedding)\n",
    "model.add(keras.layers.LSTM(units=150, dropout=0.2, recurrent_dropout=0.2, return_sequences=False))\n",
    "model.add(keras.layers.Dense(1, activation='softmax'))\n",
    "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])\n",
    "print(model.summary())\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Epoch 1/2\n",
      " 1700/25000 [=>............................] - ETA: 37:24 - loss: 7.7555 - acc: 0.5135"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-22-aa56845a2b39>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mXencoded\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabeledData\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'train'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'sentiment'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/anaconda3/lib/python3.7/site-packages/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)\u001b[0m\n\u001b[1;32m   1037\u001b[0m                                         \u001b[0minitial_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minitial_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1038\u001b[0m                                         \u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msteps_per_epoch\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1039\u001b[0;31m                                         validation_steps=validation_steps)\n\u001b[0m\u001b[1;32m   1040\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1041\u001b[0m     def evaluate(self, x=None, y=None,\n",
      "\u001b[0;32m/usr/local/anaconda3/lib/python3.7/site-packages/keras/engine/training_arrays.py\u001b[0m in \u001b[0;36mfit_loop\u001b[0;34m(model, f, ins, out_labels, batch_size, epochs, verbose, callbacks, val_f, val_ins, shuffle, callback_metrics, initial_epoch, steps_per_epoch, validation_steps)\u001b[0m\n\u001b[1;32m    197\u001b[0m                     \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mins_batch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtoarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    198\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 199\u001b[0;31m                 \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mins_batch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    200\u001b[0m                 \u001b[0mouts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mto_list\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mouts\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    201\u001b[0m                 \u001b[0;32mfor\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mo\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout_labels\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mouts\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2713\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_legacy_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2714\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2715\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2716\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2717\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mpy_any\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mis_tensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, inputs)\u001b[0m\n\u001b[1;32m   2673\u001b[0m             \u001b[0mfetched\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_callable_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0marray_vals\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_metadata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2674\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2675\u001b[0;31m             \u001b[0mfetched\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_callable_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0marray_vals\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2676\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mfetched\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2677\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1437\u001b[0m           ret = tf_session.TF_SessionRunCallable(\n\u001b[1;32m   1438\u001b[0m               \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_session\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_handle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstatus\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1439\u001b[0;31m               run_metadata_ptr)\n\u001b[0m\u001b[1;32m   1440\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1441\u001b[0m           \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "model.fit(Xencoded, labeledData['train']['sentiment'], batch_size=100, epochs=2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.evaluate(XencodedTest, )\n",
    "# from sklearn.model_selection import StratifiedShuffleSplit\n",
    "# ss = StratifiedShuffleSplit(n_splits = 1, test_size= 0.2, random_state=1).split(Xencoded, labeledData[\"\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['d'],\n",
       "       ['ldkjsf']], dtype=object)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train_labels = keras.utils.to_categorical(labeledData[\"train\"][\"sentiment\"], 2)\n",
    "# test_labels = keras.utils.to_categorical(labeledData[\"test\"][\"sentiment\"], 2)\n",
    "# early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=2, mode='auto', restore_best_weights=False)\n",
    "# history = model.fit(x=x_train, y=train_labels, epochs=50, batch_size=32, shuffle=True, validation_data = (XencodedTest, test_labels), verbose=2, callbacks=[early_stop])\n",
    "# predicted = model.predict(test_x, verbose=2)\n",
    "# predicted_labels = predicted.argmax(axis=1)\n",
    "\n",
    "l = [\"34\", \"34\"]\n",
    "l=[\"d\"]\n",
    "l = ''.join(l)\n",
    "\n",
    "a = np.array([l], dtype = object)\n",
    "b = np.array([\"ldkjsf\"], dtype = object)\n",
    "np.vstack([a,b])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a model, training, and applying the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'LinearSVC' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-804182f316c4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m#Any linear model can be used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mLinearSVC\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mC\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;31m#This is another model that can be used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mmodel2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mLogisticRegression\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mmodel4\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mSGDClassifier\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'LinearSVC' is not defined"
     ]
    }
   ],
   "source": [
    "#Any linear model can be used\n",
    "model = LinearSVC(C=1)  \n",
    "#This is another model that can be used\n",
    "model2 = LogisticRegression()   \n",
    "model4 = SGDClassifier()\n",
    "model5 = PassiveAggressiveClassifier()\n",
    "model6 = NuSVC()\n",
    "model6 = NearestCentroid()\n",
    "#model trained associating review vectors to it sentiment scores \n",
    "model.fit(x_train, labeledData[\"train\"][\"sentiment\"]) \n",
    "#applying model on test data creates what model thinks is sentiment scores associated with each movie review\n",
    "y_pred = model.predict(x_test) \n",
    "#accuracy score created by comparing to actual sentiment score to model's predicted sentiment score\n",
    "acc = accuracy_score(labeledData[\"test\"][\"sentiment\"], y_pred)\n",
    "print(\"Accuracy score of model: \"+str(acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[11154,  1346],\n",
       "       [ 1275, 11225]], dtype=int64)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix as cm\n",
    "cm(labeledData[\"test\"][\"sentiment\"], y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#This is a the representation of the matrix of all the vectorized movie reviews. \n",
    "#The data is very sparse, so this data only shows the position on if a word is contained in a movie review\n",
    "#The first number in the ordered pair represents the movie review. \n",
    "#The second number in the ordered pair represents a particular word.\n",
    "#The row of 1 shows that it is the word is there in the movie review.\n",
    "#Everthing else are 0s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(np.hstack((x_train[0], np.array(1))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(5):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#This prints out the first 10 n-grams associated with the first review above and the correspoding indeces for each n-gram\n",
    "count = 10;\n",
    "for i in x_train[0].nonzero()[1]:\n",
    "    print(str(i)+\": \" + vectorizer.get_feature_names()[i])\n",
    "    count-=1\n",
    "    if (count == 0):\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gensim.models import Word2Vec\n",
    "#gzip.open(\"C:\\Users\\mdodda3-gtri\\emade\\datasets\\movie_reviews\\GoogleNews-vectors-negative300.bin.gz\")\n",
    "wv = gensim.models.KeyedVectors.load_word2vec_format(\"GoogleNews-vectors-negative300.bin.gz\", binary=True)\n",
    "wv.init_sims(replace=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using pipeline to make it easier (puts, vectorizer and model in one line)\n",
    "Using Tfidf by transforming original countvectorizer using Tfidf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tokenizer = Tokenizer()\n",
    "# tokenizer.fit_on_texts(labeledData[\"train\"][\"text\"])\n",
    "# x_train = tokenizer.texts_to_sequences(labeledData[\"train\"][\"text\"])\n",
    "# x_test =  tokenizer.texts_to_sequences(labeledData[\"train\"][\"text\"])\n",
    "\n",
    "import time\n",
    "\n",
    "start = time.time()\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "stop_words = ['ourselves', 'hers', 'between', 'yourself', 'but', 'again', 'there', 'about', 'once', 'during', 'out', 'very', 'having', 'with', 'they', 'own', 'an', 'be', 'some', 'for', 'do', 'its', 'yours', 'such', 'into', 'of', 'most', 'itself', 'other', 'off', 'is', 's', 'am', 'or', 'who', 'as', 'from', 'him', 'each', 'the', 'themselves', 'until', 'below', 'are', 'we', 'these', 'your', 'his', 'through', 'don', 'nor', 'me', 'were', 'her', 'more', 'himself', 'this', 'down', 'should', 'our', 'their', 'while', 'above', 'both', 'up', 'to', 'ours', 'had', 'she', 'all', 'no', 'when', 'at', 'any', 'before', 'them', 'same', 'and', 'been', 'have', 'in', 'will', 'on', 'does', 'yourselves', 'then', 'that', 'because', 'what', 'over', 'why', 'so', 'can', 'did', 'not', 'now', 'under', 'he', 'you', 'herself', 'has', 'just', 'where', 'too', 'only', 'myself', 'which', 'those', 'i', 'after', 'few', 'whom', 't', 'being', 'if', 'theirs', 'my', 'against', 'a', 'by', 'doing', 'it', 'how', 'further', 'was', 'here', 'than']\n",
    "stop_words = ['in', 'of', 'at', 'a', 'the']\n",
    "def tokenize(val):\n",
    "    ans = text_to_word_sequence(val)\n",
    "    ans = [a for a in ans if not a in stop_words]\n",
    "    return ans\n",
    "reviews = labeledData[\"train\"][\"text\"].append(labeledData[\"test\"][\"text\"], ignore_index=True).values\n",
    "\n",
    "words = [tokenize(val) for val in reviews.tolist()]\n",
    "\n",
    "model = gensim.models.Word2Vec(sentences=words, size=100, window = 1000, workers = 4, min_count=1, sg = 0)\n",
    "\n",
    "def method(list, wv):\n",
    "    mean = []\n",
    "    for word in list:\n",
    "        if word in wv.vocab:\n",
    "            mean.append(wv[word])\n",
    "        else:\n",
    "            a.append(word)\n",
    "    mean = gensim.matutils.unitvec(np.array(mean).mean(axis=0)).astype(np.float32)\n",
    "    return mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [],
   "source": [
    "#x_train = np.array([method(review, wv) for review in reviews.tolist()])\n",
    "#x_train.shape\n",
    "x_train = words[:25000]\n",
    "x_train = np.array([method(review, model.wv) for review in x_train])\n",
    "x_test = words[25000:]\n",
    "x_test = np.array([method(review, model.wv) for review in x_test])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [],
   "source": [
    "#.85428\n",
    "#159.6575005054474 seconds\n",
    "## CBOW, size changing\n",
    "#10: .75536\n",
    "#25: \n",
    "#50: .84576\n",
    "#100:.8596 ~40\n",
    "#500: .87052 ~76\n",
    "#750: .87104\n",
    "#1000: .87128 ~116.53\n",
    "\n",
    "##CBOW window\n",
    "#1 .833\n",
    "#5 .8501\n",
    "#10 .86 ~45\n",
    "#25 .87 ~ 57\n",
    "#50 same as 25\n",
    "#100 .89 105\n",
    "#200 .887 140\n",
    "#500 .892\n",
    "#1000 .894\n",
    "\n",
    "##Skip-Gram\n",
    "#25: .85\n",
    "#50: .87464\n",
    "#100: .88 ~ 154\n",
    "#200: .89 ~ 190\n",
    "#500:  .89 322   ~ (300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy score of model: 0.89452\n",
      "224.66473388671875\n"
     ]
    }
   ],
   "source": [
    "model = LinearSVC()\n",
    "\n",
    "model.fit(x_train, labeledData[\"train\"][\"sentiment\"]) \n",
    "#applying model on test data creates what model thinks is sentiment scores associated with each movie review\n",
    "y_pred = model.predict(x_test) \n",
    "#accuracy score created by comparing to actual sentiment score to model's predicted sentiment score\n",
    "acc = accuracy_score(labeledData[\"test\"][\"sentiment\"], y_pred)\n",
    "print(\"Accuracy score of model: \"+str(acc))\n",
    "end = time.time()\n",
    "print(end - start)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.word_index\n",
    "#wv.syn0norm\n",
    "#wv.wv.most_similar(\"little\")\n",
    "# for word, i in tokenizer.items():\n",
    "wv.vocab[\"hello\"].count\n",
    "wv[\"hello\"]\n",
    "tokenizer.word_index\n",
    "\n",
    "def tokenize(val):\n",
    "    ans = text_to_word_sequence(val)\n",
    "    ans = [a for a in ans if not a in stop_words]\n",
    "    return ans\n",
    "\n",
    "x_train_list = [tokenize(val) for val in labeledData[\"train\"][\"text\"].values.tolist()]\n",
    "x_test_list = [tokenize(val) for val in labeledData[\"test\"][\"text\"].values.tolist()]\n",
    "a =[]\n",
    "def method(list, wv):\n",
    "    mean = []\n",
    "    for word in list:\n",
    "        if word in wv.vocab:\n",
    "            mean.append(wv[word])\n",
    "        else:\n",
    "            a.append(word)\n",
    "    mean = gensim.matutils.unitvec(np.array(mean).mean(axis=0)).astype(np.float32)\n",
    "    return mean\n",
    "x_train = np.array([method(review, wv) for review in x_train_list])\n",
    "x_test = np.array([method(review, wv) for review in x_test_list])\n",
    "\n",
    "\n",
    "model = LinearSVC()\n",
    "\n",
    "model.fit(x_train, labeledData[\"train\"][\"sentiment\"]) \n",
    "#applying model on test data creates what model thinks is sentiment scores associated with each movie review\n",
    "y_pred = model.predict(x_test) \n",
    "#accuracy score created by comparing to actual sentiment score to model's predicted sentiment score\n",
    "acc = accuracy_score(labeledData[\"test\"][\"sentiment\"], y_pred)\n",
    "print(\"Accuracy score of model: \"+str(acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_list[0]\n",
    "print(a[89])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # nltk.download('punkt')\n",
    "# # [nltk.word_tokenize(sentences) for sentences in train]\n",
    "# # model = gensim.models.Word2Vec(sentences = )\n",
    "# from keras.preprocessing.sequence import pad_sequences\n",
    "# wv.wv.most_similar('horrible')\n",
    "\n",
    "\n",
    "\n",
    "# tokenizer_obj = Tokenizer()\n",
    "# tokenizer_obj.fit_on_texts(labeledData[\"train\"][\"text\"])\n",
    "# sequences = tokenizer_obj.texts_to_sequences(labeledData[\"train\"][\"text\"])\n",
    "# word_index = tokenizer_obj.word_index\n",
    "# max_length = max([len(s.split()) for s in labeledData[\"train\"][\"text\"] + labeledData[\"test\"][\"text\"]])\n",
    "# review_pad = pad_sequences(sequences, maxlen = max_length)\n",
    "\n",
    "# num_words = len(word_index) + 1\n",
    "# embedding_matrix = np.zeros((num_words, 100))\n",
    "\n",
    "# for word, i in word_index.items():\n",
    "#     if i > num_words:\n",
    "#         continue\n",
    "#     embedding_vector = embeddings_index.get(word)\n",
    "#     if embedding_vector is not None:\n",
    "#         embedding_matrix[i] = embedding_vector\n",
    "# type(embedding_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "from sklearn.ensemble import RandomForestClassifier, BaggingClassifier, GradientBoostingClassifier, AdaBoostClassifier,ExtraTreesClassifier\n",
    "\n",
    "\n",
    "#pipe = Pipeline([('vect', CountVectorizer(binary=True, ngram_range=(1,2))),('clf',LogisticRegression(C=.05))])fg\n",
    "stop_words = ['in', 'of', 'at', 'a', 'the']\n",
    "\n",
    "##pipe = Pipeline([('vect', CountVectorizer(binary=True, ngram_range=(1,2), stop_words=stop_words)), ('clf',LinearSVC(C=.01))])##\n",
    "\n",
    "#Making pipeline. has vectorizer, uses tfidf, and uses multinomialnb for the model\n",
    "pipe1 = Pipeline([('vect', CountVectorizer(tokenizer = LemmaTokenizer(), binary=True, ngram_range=(1,2), stop_words=stop_words)), \n",
    "                 ('clf',SGDClassifier())])\n",
    "pipe2 = Pipeline([('vect', CountVectorizer(stop_words=stop_words, binary=True, ngram_range=(1,2) )), ('clf', SGDClassifier())])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#fits whole pipeline using the train text as x value and train sentiment as y value and makes model\n",
    "for pipe in [pipe1, pipe2]:\n",
    "    pipe.fit(labeledData[\"train\"][\"text\"], labeledData[\"train\"][\"sentiment\"]) \n",
    "    #model predicts sentiment scores using test dataframe\n",
    "    y_pred = pipe.predict(labeledData[\"test\"][\"text\"])\n",
    "    #compares model sentiment scores to actual sentiment scores\n",
    "    acc = accuracy_score(labeledData[\"test\"][\"sentiment\"], y_pred)\n",
    "    print(\"Accuracy score of the model: \"+ str(acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from gensim.models import Word2Vec\n",
    "# wv = gensim.models.KeyedVectors.load_word2vec_format(\"GoogleNews-vectors-negative300.bin.gz\", binary=True)\n",
    "# wv.init_sims(replace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [],
   "source": [
    "n =np.array([1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[1]]])"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([n])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}