COMP30027_Assignment2_text_features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COMP30027 Machine Learning Assignment 2\n",
    "\n",
    "## Description of text features\n",
    "\n",
    "This notebook describes the pre-computed text features provided for assignment 2. **You do not need to recompute the features yourself for this assignment** -- this information is just for your reference. However, feel free to experiment with different text features if you are interested. If you do want to try generating your own text features, some things to keep in mind:\n",
    "- There are many different decisions you can make throughout the feature design process, from the text preprocessing to the size of the output vectors. There's no guarantee that the defaults we chose will produce the best possible text features for this classification task, so feel free to experiment with different settings.\n",
    "- These features must be trained on a training corpus. Generally, the training corpus should not include validation samples, but for the purposes of this assignment we have used the entire non-test set (training+validation) as the training corpus, to allow you to experiment with different validation sets. If you recompute the text features as part of your own model, you should exclude validation samples and retrain on training samples only. Note that if you do N-fold cross-validation, this means generating N sets of features for N different training-validation splits.\n",
    "- This code may take a long time to run and require a good bit of memory, which is why we are not requiring you to recompute these features yourself. doc2vec in particular is very slow unless you can implement some speed-ups in C."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# read text\n",
    "# for DEMONSTRATION PURPOSES, the entire training set will be used to train the models and also as a test set\n",
    "x_train_original = pd.read_csv(r\"recipe_train.csv\", index_col = False, delimiter = ',', header=0)\n",
    "# use recipe name as an example\n",
    "train_corpus_name = x_train_original['name']\n",
    "test_name = x_train_original['name']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Count vectorizer\n",
    "\n",
    "A count vectorizer converts documents to vectors which represent word counts. Each column in the output represents a different word and the values indicate the number of times that word appeared in the document. The overall size of a count vector matrix can be quite large (the number of columns is the total number of different words used across all documents in a corpus), but most entries in the matrix are zero (each document contains only a few of all the possible words). Therefore, it is most efficient to represent the count vectors as a sparse matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10892\n",
      "(40000, 10892)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# preprocess text and compute counts\n",
    "vocab_name = CountVectorizer(stop_words='english').fit(train_corpus_name)\n",
    "\n",
    "# generate counts for a new set of documents\n",
    "x_train_name = vocab_name.transform(train_corpus_name)\n",
    "x_test_name = vocab_name.transform(test_name)\n",
    "\n",
    "# check the number of words in vocabulary\n",
    "print(len(vocab_name.vocabulary_))\n",
    "# check the shape of sparse matrix\n",
    "print(x_train_name.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## doc2vec\n",
    "\n",
    "doc2vec methods are an extension of word2vec. word2vec maps words to a high-dimensional vector space in such a way that words which appear in similar contexts will be close together in the space. doc2vec does a similar embedding for multi-word passages. The doc2vec (or Paragraph Vector) method was introduced by:\n",
    "\n",
    "**Le & Mikolov (2014)** Distributed Representations of Sentences and Documents<br>\n",
    "https://arxiv.org/pdf/1405.4053v2.pdf\n",
    "\n",
    "The implementation of doc2vec used for this project is from gensim and documented here:<br>\n",
    "https://radimrehurek.com/gensim/models/doc2vec.html\n",
    "\n",
    "The size of the output vector is a free parameter. Most implemementations use around 100-300 dimensions, but the best size depends on the problem you're trying to solve with the embeddings and the number of training samples, so you may wish to try different vector sizes. We provided three sets of doc2vec features, and their dimensions are 50 and 100. The vectors themselves represent directions in a high-dimensional concept space; the columns do not represent specific words or phrases. Values in the vector are continuous real numbers and can be negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(40000, 20)\n"
     ]
    }
   ],
   "source": [
    "import gensim\n",
    "\n",
    "# size of the output vector\n",
    "vec_size = 20\n",
    "\n",
    "# function to preprocess and tokenize text\n",
    "def tokenize_corpus(txt, tokens_only=False):\n",
    "    for i, line in enumerate(txt):\n",
    "        tokens = gensim.utils.simple_preprocess(line)\n",
    "        if tokens_only:\n",
    "            yield tokens\n",
    "        else:\n",
    "            yield gensim.models.doc2vec.TaggedDocument(tokens, [i])\n",
    "\n",
    "# tokenize a training corpus\n",
    "corpus_name = list(tokenize_corpus(train_corpus_name))\n",
    "\n",
    "# train doc2vec on the training corpus\n",
    "model = gensim.models.doc2vec.Doc2Vec(vector_size=vec_size, min_count=2, epochs=40)\n",
    "model.build_vocab(corpus_name)\n",
    "model.train(corpus_name, total_examples=model.corpus_count, epochs=model.epochs)\n",
    "\n",
    "# tokenize new documents\n",
    "doc = list(tokenize_corpus(test_name, tokens_only=True))\n",
    "\n",
    "# generate embeddings for the new documents\n",
    "x_test_name = np.zeros((len(doc),vec_size))\n",
    "for i in range(len(doc)):\n",
    "    x_test_name[i,:] = model.infer_vector(doc[i])\n",
    "    \n",
    "# check the shape of doc_emb\n",
    "print(x_test_name.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}