CNN+BiLSTM+CRF-WiNER-EN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s6Oi4OFtlMFM"
   },
   "source": [
    "# CNN + Glove + BiLSTM + CRF model for Entity Extraction on WikiNER (EN)\n",
    "\n",
    "In this notebook, we implement the neural model described in [this paper](https://www.aclweb.org/anthology/P16-1101.pdf). This model uses:\n",
    "* Character-level informations extracted with a CNN;\n",
    "* Word-level informations starting from Glove 100-dimensional 6B embedding;\n",
    "* A BiLSTM and a CRF layer for making predictions.\n",
    "\n",
    "Data preprocessing is composed of padding sentences plus token encoding and character-sequences padding to fixed length. Then, we implement this model using `tensorflow.keras` and the `tf2crf` package for a CRF layer compatible with tensorflow. We test it on the the WikiNER english dataset, using the `seqeval` package for f1-score evaluation.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-nmzyeF3u_Ni"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import string\n",
    "import numpy as np\n",
    "from utils import dataio, kerasutils, modelutils\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.utils import to_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RzksTlOXmRrW"
   },
   "source": [
    "## Load dataset\n",
    "\n",
    "Thanks to the author of [this repo](https://github.com/dice-group/FOX/blob/master/input/Wikiner/aij-wikiner-en-wp3.bz2) that makes WikiNER data easily available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 118
    },
    "colab_type": "code",
    "id": "lyW9pOkIyCE-",
    "outputId": "86507bba-e635-4cd1-fcb0-bffb1fe0a8e1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read 142153 sentences.\n"
     ]
    }
   ],
   "source": [
    "file_path = os.path.join('data', 'wikiner-en-wp3-raw.txt')\n",
    "raw, ner, output_labels = dataio.load_wikiner(file_path, token_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Labels: {'I-MISC', 'I-ORG', 'O', 'I-PER', 'B-LOC', 'I-LOC', 'B-ORG', 'B-MISC', 'B-PER'}\n"
     ]
    }
   ],
   "source": [
    "print(\"Labels:\", output_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence Example:\n",
      "The             | I-MISC\n",
      "Oxford          | I-MISC\n",
      "Companion       | I-MISC\n",
      "to              | I-MISC\n",
      "Philosophy      | I-MISC\n",
      "says            | O\n",
      ",               | O\n",
      "\"               | O\n",
      "there           | O\n",
      "is              | O\n",
      "no              | O\n",
      "single          | O\n",
      "defining        | O\n",
      "position        | O\n",
      "that            | O\n",
      "all             | O\n",
      "anarchists      | O\n",
      "hold            | O\n",
      ",               | O\n",
      "and             | O\n",
      "those           | O\n",
      "considered      | O\n",
      "anarchists      | O\n",
      "at              | O\n",
      "best            | O\n",
      "share           | O\n",
      "a               | O\n",
      "certain         | O\n",
      "family          | O\n",
      "resemblance     | O\n",
      ".               | O\n",
      "\"               | O\n"
     ]
    }
   ],
   "source": [
    "print(\"Sentence Example:\")\n",
    "for i in range(len(raw[0])):\n",
    "    print(f'{raw[0][i]:15} | {ner[0][i]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "opnm9IMny5ID"
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kjmchvML7JWw"
   },
   "source": [
    "# Data Preparation\n",
    "Prepare character- and word-level input for the model.\n",
    "\n",
    "## Sentence encoding and padding\n",
    "We use a Keras `Tokenizer` to extract the vocabulary and encode words. We pad sentences to a fixed length because it is required from LSTM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mgkojyV47DDQ"
   },
   "outputs": [],
   "source": [
    "# integer encode sequences of words\n",
    "token_tokenizer = Tokenizer()    # Automatically lowers tokens\n",
    "token_tokenizer.fit_on_texts(raw)\n",
    "sequences = token_tokenizer.texts_to_sequences(raw)\n",
    "\n",
    "# Label encoding\n",
    "tag2idx = { tag: idx for idx, tag in enumerate(output_labels) }\n",
    "idx2tag = { idx: tag for tag, idx in tag2idx.items() }\n",
    "ner_sequences = [[tag2idx[tag] for tag in sentence] for sentence in ner]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "HVOgrGyg5D06",
    "outputId": "b764bd2b-3e1c-4fa6-c455-c5087e8b35c7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "108276\n"
     ]
    }
   ],
   "source": [
    "vocabulary_size = len(token_tokenizer.word_counts)\n",
    "print(vocabulary_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XhUgVKH85GU_"
   },
   "outputs": [],
   "source": [
    "max_sentence_len = 50\n",
    "X_sent = pad_sequences(sequences, maxlen=max_sentence_len, padding='post', truncating='post')\n",
    "Y = pad_sequences(ner_sequences, maxlen=max_sentence_len, value=tag2idx['O'], padding='post', truncating='post')\n",
    "\n",
    "X_sent = np.array(X_sent)\n",
    "Y = np.array(Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 121
    },
    "colab_type": "code",
    "id": "TYX5wdr75Zp-",
    "outputId": "5a204f82-2574-4aa6-c07a-14548b4f93ab"
   },
   "outputs": [],
   "source": [
    "token_tokenizer.index_word[0] = '_PAD_'\n",
    "token_tokenizer.word_index['_PAD_'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 121
    },
    "colab_type": "code",
    "id": "TYX5wdr75Zp-",
    "outputId": "5a204f82-2574-4aa6-c07a-14548b4f93ab"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoded and padded sentence:\n",
      "     1 | the\n",
      "  2653 | oxford\n",
      "  4672 | companion\n",
      "     7 | to\n",
      "   934 | philosophy\n",
      "  1437 | says\n",
      "     2 | ,\n",
      "    10 | \"\n",
      "    68 | there\n",
      "    12 | is\n",
      "    92 | no\n",
      "   369 | single\n",
      "  6229 | defining\n",
      "   456 | position\n",
      "    16 | that\n",
      "    62 | all\n",
      "  7102 | anarchists\n",
      "  1284 | hold\n",
      "     2 | ,\n",
      "     6 | and\n"
     ]
    }
   ],
   "source": [
    "print('Encoded and padded sentence:')\n",
    "for i in range(len(X_sent[0][:20])):\n",
    "    print(f'{X_sent[0][i]:6} | {token_tokenizer.index_word[X_sent[0][i]]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "t3FaEIl15nvp"
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RwKih7le7Oic"
   },
   "source": [
    "## Character encoding and padding\n",
    "In order to extract character-level informations, we have to:\n",
    "* Encode characters with integers;\n",
    "* Pad words to a fixed lengths;\n",
    "* Use the 0 as padding integer both for sentence padding and for word padding.\n",
    "\n",
    "We don't want to truncate words because prefix and suffix contains precious informations, so we take the longest words and we pad words to its length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yENsgixv-kmm"
   },
   "outputs": [],
   "source": [
    "def to_char_list(data):\n",
    "    '''Transform all the words of a dataset into lists of characters'''\n",
    "    \n",
    "    char_data = []\n",
    "    for sentence in data:\n",
    "        char_sent = []\n",
    "        for word in sentence:\n",
    "            char_sent.append(list(word))\n",
    "        char_data.append(char_sent)\n",
    "    return char_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "colab_type": "code",
    "id": "ISB3Hepk5Yro",
    "outputId": "e3d2cc7b-7c9d-4de1-e263-91c51345ebc2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['T', 'h', 'e']\n",
      "['O', 'x', 'f', 'o', 'r', 'd']\n",
      "['C', 'o', 'm', 'p', 'a', 'n', 'i', 'o', 'n']\n",
      "['t', 'o']\n",
      "['P', 'h', 'i', 'l', 'o', 's', 'o', 'p', 'h', 'y']\n",
      "['s', 'a', 'y', 's']\n",
      "[',']\n",
      "['\"']\n",
      "['t', 'h', 'e', 'r', 'e']\n",
      "['i', 's']\n",
      "['n', 'o']\n",
      "['s', 'i', 'n', 'g', 'l', 'e']\n",
      "['d', 'e', 'f', 'i', 'n', 'i', 'n', 'g']\n",
      "['p', 'o', 's', 'i', 't', 'i', 'o', 'n']\n",
      "['t', 'h', 'a', 't']\n",
      "['a', 'l', 'l']\n",
      "['a', 'n', 'a', 'r', 'c', 'h', 'i', 's', 't', 's']\n",
      "['h', 'o', 'l', 'd']\n",
      "[',']\n",
      "['a', 'n', 'd']\n",
      "['t', 'h', 'o', 's', 'e']\n",
      "['c', 'o', 'n', 's', 'i', 'd', 'e', 'r', 'e', 'd']\n",
      "['a', 'n', 'a', 'r', 'c', 'h', 'i', 's', 't', 's']\n",
      "['a', 't']\n",
      "['b', 'e', 's', 't']\n",
      "['s', 'h', 'a', 'r', 'e']\n",
      "['a']\n",
      "['c', 'e', 'r', 't', 'a', 'i', 'n']\n",
      "['f', 'a', 'm', 'i', 'l', 'y']\n",
      "['r', 'e', 's', 'e', 'm', 'b', 'l', 'a', 'n', 'c', 'e']\n",
      "['.']\n",
      "['\"']\n",
      "==============================\n",
      "142153\n"
     ]
    }
   ],
   "source": [
    "raw_char = to_char_list(raw)\n",
    "\n",
    "for token in raw_char[0]:\n",
    "    print(token)\n",
    "print('='*30)\n",
    "print(len(raw_char))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LRBCQI7DwudX"
   },
   "outputs": [],
   "source": [
    "# Sanity check of preprocessed data dimensions. If it does not output anything,\n",
    "# everything is fine.\n",
    "for sent_idx in range(len(raw)):\n",
    "    if len(raw_char[sent_idx]) != len(sequences[sent_idx]):\n",
    "        print('sequence len error')\n",
    "        print(raw_char[sent_idx])\n",
    "        print(sequences[sent_idx])\n",
    "    for word_idx in range(len(raw[sent_idx])):\n",
    "        if len(raw_char[sent_idx][word_idx]) != len(raw[sent_idx][word_idx]):\n",
    "            print('word len error')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Character vocabulary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "OL9BX1ZEK2RN",
    "outputId": "e714327e-05b6-4679-ca82-07942de92ed2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Charset dimension: 94\n",
      "Charset: abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n"
     ]
    }
   ],
   "source": [
    "# NOTE: Tokenizer may take an argument char_level=True. We should try it in \n",
    "# order to get a cleaner code, but in this way we do not have a fixed length\n",
    "# for words.\n",
    "char_tokenizer = Tokenizer(lower=False, filters='')\n",
    "# Build a list with all the characters\n",
    "charset = string.ascii_letters + string.digits + string.punctuation\n",
    "print(f'Charset dimension: {len(charset)}')\n",
    "print(f'Charset: {charset}')\n",
    "char_tokenizer.fit_on_texts(list(charset))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "u-i2B1_xMp1y"
   },
   "outputs": [],
   "source": [
    "# Add padding to the tokenizer with the 0 integer encoding\n",
    "char_tokenizer.index_word[0] = '_PAD_'\n",
    "char_tokenizer.word_index['_PAD_'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y68BSHYmM9-B"
   },
   "source": [
    "#### Pad sentences\n",
    "Set the lengths to `max_sentence_len` (50) with padding and truncate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "vE54ewxXM3XD",
    "outputId": "4f11bf5e-3f98-460c-bbfd-e8ff7c80f168"
   },
   "outputs": [],
   "source": [
    "for sent_idx in range(len(raw_char)):\n",
    "    if len(raw_char[sent_idx]) > max_sentence_len:\n",
    "        # Truncate long sentences\n",
    "        raw_char[sent_idx] = raw_char[sent_idx][:max_sentence_len]\n",
    "    while len(raw_char[sent_idx]) < max_sentence_len:\n",
    "        # Pad sentences with '_PAD_' characters\n",
    "        pad_word = []\n",
    "        pad_word.append(char_tokenizer.index_word[0])\n",
    "        raw_char[sent_idx].append(pad_word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "vE54ewxXM3XD",
    "outputId": "4f11bf5e-3f98-460c-bbfd-e8ff7c80f168"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Padded sentence:\n",
      "['T', 'h', 'e']\n",
      "['O', 'x', 'f', 'o', 'r', 'd']\n",
      "['C', 'o', 'm', 'p', 'a', 'n', 'i', 'o', 'n']\n",
      "['t', 'o']\n",
      "['P', 'h', 'i', 'l', 'o', 's', 'o', 'p', 'h', 'y']\n",
      "['s', 'a', 'y', 's']\n",
      "[',']\n",
      "['\"']\n",
      "['t', 'h', 'e', 'r', 'e']\n",
      "['i', 's']\n",
      "['n', 'o']\n",
      "['s', 'i', 'n', 'g', 'l', 'e']\n",
      "['d', 'e', 'f', 'i', 'n', 'i', 'n', 'g']\n",
      "['p', 'o', 's', 'i', 't', 'i', 'o', 'n']\n",
      "['t', 'h', 'a', 't']\n",
      "['a', 'l', 'l']\n",
      "['a', 'n', 'a', 'r', 'c', 'h', 'i', 's', 't', 's']\n",
      "['h', 'o', 'l', 'd']\n",
      "[',']\n",
      "['a', 'n', 'd']\n",
      "['t', 'h', 'o', 's', 'e']\n",
      "['c', 'o', 'n', 's', 'i', 'd', 'e', 'r', 'e', 'd']\n",
      "['a', 'n', 'a', 'r', 'c', 'h', 'i', 's', 't', 's']\n",
      "['a', 't']\n",
      "['b', 'e', 's', 't']\n",
      "['s', 'h', 'a', 'r', 'e']\n",
      "['a']\n",
      "['c', 'e', 'r', 't', 'a', 'i', 'n']\n",
      "['f', 'a', 'm', 'i', 'l', 'y']\n",
      "['r', 'e', 's', 'e', 'm', 'b', 'l', 'a', 'n', 'c', 'e']\n",
      "['.']\n",
      "['\"']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n",
      "['_PAD_']\n"
     ]
    }
   ],
   "source": [
    "print('Padded sentence:')\n",
    "for token in raw_char[0]:\n",
    "    print(token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "hA6-2tB9NsuI",
    "outputId": "d5c44943-d589-4132-ae3f-83116642b1ef"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "95"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(char_tokenizer.word_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZmFx1qRLoP5G"
   },
   "source": [
    "#### Encode characters with integers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wF5ykXXaoceE"
   },
   "outputs": [],
   "source": [
    "char_seq = []\n",
    "for sentence in raw_char:\n",
    "    char_seq.append(char_tokenizer.texts_to_sequences(sentence))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 894
    },
    "colab_type": "code",
    "id": "epsNNx8VNxO3",
    "outputId": "efbfdcc9-9ad8-4ca6-acb7-33d4b2223540"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[46, 8, 5] => ['T', 'h', 'e']\n",
      "[41, 24, 6, 15, 18, 4] => ['O', 'x', 'f', 'o', 'r', 'd']\n",
      "[29, 15, 13, 16, 1, 14, 9, 15, 14] => ['C', 'o', 'm', 'p', 'a', 'n', 'i', 'o', 'n']\n",
      "[20, 15] => ['t', 'o']\n",
      "[42, 8, 9, 12, 15, 19, 15, 16, 8, 25] => ['P', 'h', 'i', 'l', 'o', 's', 'o', 'p', 'h', 'y']\n",
      "[19, 1, 25, 19] => ['s', 'a', 'y', 's']\n",
      "[74] => [',']\n",
      "[64] => ['\"']\n",
      "[20, 8, 5, 18, 5] => ['t', 'h', 'e', 'r', 'e']\n",
      "[9, 19] => ['i', 's']\n",
      "[14, 15] => ['n', 'o']\n",
      "[19, 9, 14, 7, 12, 5] => ['s', 'i', 'n', 'g', 'l', 'e']\n",
      "[4, 5, 6, 9, 14, 9, 14, 7] => ['d', 'e', 'f', 'i', 'n', 'i', 'n', 'g']\n",
      "[16, 15, 19, 9, 20, 9, 15, 14] => ['p', 'o', 's', 'i', 't', 'i', 'o', 'n']\n",
      "[20, 8, 1, 20] => ['t', 'h', 'a', 't']\n",
      "[1, 12, 12] => ['a', 'l', 'l']\n",
      "[1, 14, 1, 18, 3, 8, 9, 19, 20, 19] => ['a', 'n', 'a', 'r', 'c', 'h', 'i', 's', 't', 's']\n",
      "[8, 15, 12, 4] => ['h', 'o', 'l', 'd']\n",
      "[74] => [',']\n",
      "[1, 14, 4] => ['a', 'n', 'd']\n",
      "[20, 8, 15, 19, 5] => ['t', 'h', 'o', 's', 'e']\n",
      "[3, 15, 14, 19, 9, 4, 5, 18, 5, 4] => ['c', 'o', 'n', 's', 'i', 'd', 'e', 'r', 'e', 'd']\n",
      "[1, 14, 1, 18, 3, 8, 9, 19, 20, 19] => ['a', 'n', 'a', 'r', 'c', 'h', 'i', 's', 't', 's']\n",
      "[1, 20] => ['a', 't']\n",
      "[2, 5, 19, 20] => ['b', 'e', 's', 't']\n",
      "[19, 8, 1, 18, 5] => ['s', 'h', 'a', 'r', 'e']\n",
      "[1] => ['a']\n",
      "[3, 5, 18, 20, 1, 9, 14] => ['c', 'e', 'r', 't', 'a', 'i', 'n']\n",
      "[6, 1, 13, 9, 12, 25] => ['f', 'a', 'm', 'i', 'l', 'y']\n",
      "[18, 5, 19, 5, 13, 2, 12, 1, 14, 3, 5] => ['r', 'e', 's', 'e', 'm', 'b', 'l', 'a', 'n', 'c', 'e']\n",
      "[76] => ['.']\n",
      "[64] => ['\"']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n",
      "[0] => ['_PAD_']\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(char_seq[0])):\n",
    "    w = [char_tokenizer.index_word[letter] for letter in char_seq[0][i]]\n",
    "    print(char_seq[0][i], '=>', w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UkFWAk6IOV-v"
   },
   "source": [
    "#### Pad words \n",
    "Set all the words to `maxlen` with padding and (possibly without) truncate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ONfUhZcHOVem"
   },
   "outputs": [],
   "source": [
    "def pad_words(sentence, maxlen, pad=0):\n",
    "    padded_sentence = []\n",
    "    for word in sentence:\n",
    "        new_word = word.copy()\n",
    "        if len(word) > maxlen:\n",
    "            new_word = word[:maxlen]\n",
    "        else:\n",
    "            while maxlen - len(new_word) > 1:\n",
    "                new_word.append(pad)\n",
    "                new_word.insert(0, pad)\n",
    "            if maxlen - len(new_word) == 1:\n",
    "                new_word.insert(0, pad)\n",
    "        padded_sentence.append(new_word)\n",
    "    \n",
    "    return padded_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "zu6KDDBuO-Yf",
    "outputId": "96a1e8d9-75d0-4500-ed37-51668e1d530d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word length: 93\n"
     ]
    }
   ],
   "source": [
    "max_word_len = max([len(word) for word in token_tokenizer.word_index.keys()])\n",
    "print('Word length:', max_word_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What? A word of 93 characters? Let's get deeper:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'news://alt.games.video.tiger.game-com'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words = list(token_tokenizer.word_index.keys())\n",
    "sorted(words, key=lambda w:len(w))[-5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it makes more sense: the datatset contains URLs and an URL is a single token!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PqbkIqVCvlKM"
   },
   "outputs": [],
   "source": [
    "X_char = np.array([pad_words(sentence, maxlen=max_word_len) for sentence in char_seq])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "F_TnLzE3Pp-W"
   },
   "outputs": [],
   "source": [
    "# Sanity check of preprocessed data dimensions. If it does not output anything,\n",
    "# everything is fine.\n",
    "for sentence in X_char:\n",
    "    if len(sentence) != max_sentence_len:\n",
    "        print('sentence error')\n",
    "    for word in sentence:\n",
    "        if len(word) != max_word_len:\n",
    "            print(f'word error: {len(word)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "V3fqZ0_opkhx"
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bXVl5Praplz4"
   },
   "source": [
    "# Model implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bWXt5ji7xmRD"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential, Model\n",
    "from tensorflow.keras.layers import Embedding, Conv1D, TimeDistributed, Dropout, Input, \\\n",
    "    MaxPooling1D, Flatten, concatenate, Bidirectional, LSTM, Dense\n",
    "from tensorflow.keras.utils import plot_model\n",
    "from tensorflow.keras.metrics import Precision, Recall\n",
    "from tensorflow.keras.optimizers import SGD\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "from tf2crf import CRF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "73W5p2gcJ5kb"
   },
   "source": [
    "#### Hyperparameters of the model\n",
    "You can choose between the parametrization of two proposed models:\n",
    "* Ma, Xuezhe, and Eduard Hovy. \"End-to-end sequence labeling via bi-directional lstm-cnns-crf.\" *arXiv preprint arXiv:1603.01354* (2016).\n",
    "* Chiu, Jason PC, and Eric Nichols. \"Named entity recognition with bidirectional LSTM-CNNs.\" *Transactions of the Association for Computational Linguistics 4* (2016): 357-370.\n",
    "The first works better, but it may be because second originally included the use of additional word features that we don't consider."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "USE_CHIU_CONFIG = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dywNEZQjpjxo"
   },
   "outputs": [],
   "source": [
    "if USE_CHIU_CONFIG:\n",
    "    char_embedding_dim = 25\n",
    "    cnn_window_size = 3\n",
    "    cnn_filters_number = 53\n",
    "\n",
    "    word_embedding_dim = 100\n",
    "    hidden_cells = 275\n",
    "    drop=0.68\n",
    "\n",
    "    batch_size = 9\n",
    "    epochs = 80\n",
    "else:\n",
    "    char_embedding_dim = 30\n",
    "    cnn_window_size = 3\n",
    "    cnn_filters_number = 30\n",
    "\n",
    "    word_embedding_dim = 100\n",
    "    hidden_cells = 200\n",
    "    drop=0.5\n",
    "\n",
    "    batch_size = 10\n",
    "    epochs = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 50
    },
    "colab_type": "code",
    "id": "iop9KKVsBdkN",
    "outputId": "9701b9e3-8859-4412-c616-4564b842cfc3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence token length: 50\n",
      "Word character length: 93\n"
     ]
    }
   ],
   "source": [
    "print('Sentence token length:', max_sentence_len)\n",
    "print('Word character length:', max_word_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zmnpU7dNtTj-"
   },
   "source": [
    "## CNN\n",
    "We use a Convolutive Neural Network in order to extract pattern informations from the letters of the word. The CNN embedding is formed by:\n",
    "* A `keras.layers.Embedding` layer, which is a lookup table that associate a vector to each character;\n",
    "* A 1-dimensional convolution on the embedding vectors in order to capture patterns in letters;\n",
    "* A MaxPool1d that transforms a series of vectors in a unique vectors which contains informations from the characters of the word. \n",
    "\n",
    "Credits to the author of [this repo](https://github.com/kamalkraj/Named-Entity-Recognition-with-Bidirectional-LSTM-CNNs/blob/master/nn.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KMJjHLHXw3KK"
   },
   "outputs": [],
   "source": [
    "cnn_input = Input(shape=(max_sentence_len, max_word_len,), name='char_encoding')\n",
    "# We use TimeDistributed layer because we have two level of sequences:\n",
    "# * The sentence is a sequence of words;\n",
    "# * The word is a sequence of characters;\n",
    "# We want to work on the lowest sequence. the sequence of characters, so the\n",
    "# TimeDistributed layer allow us to apply this model to each word. \n",
    "cnn = TimeDistributed(Embedding(len(char_tokenizer.word_index), char_embedding_dim), name='cnn_Embedding')(cnn_input)\n",
    "cnn = Dropout(drop)(cnn)\n",
    "cnn = TimeDistributed(Conv1D(filters=cnn_filters_number, kernel_size=cnn_window_size, padding='same'), name='cnn_Convolution1d')(cnn)\n",
    "cnn = TimeDistributed(MaxPooling1D(max_word_len), name='cnn_MaxPooling1d')(cnn)\n",
    "# We finally obtain a 30-dimensional vector for each word which contains \n",
    "# char-level informations!\n",
    "cnn_out = TimeDistributed(Flatten(), name='cnn_Flatten')(cnn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uI79cLjoHK4i"
   },
   "source": [
    "## Glove\n",
    "We load Glove embedding in order to embed tokens and capture word-level informations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "lc1MVNBEptQ7",
    "outputId": "3b4323da-1aac-4138-bf7c-c43fd8919863"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 400001 word vectors.\n"
     ]
    }
   ],
   "source": [
    "glove_embedding_path = os.path.join('embeddings', 'glove.6B.100d.txt')\n",
    "embedding_dim = 100\n",
    "embedding_matrix = kerasutils.load_glove_embedding_matrix(glove_embedding_path, token_tokenizer.word_index, embedding_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uTGqWRvULQOZ"
   },
   "outputs": [],
   "source": [
    "word_input = Input(shape=(max_sentence_len,), name='word_encoding')\n",
    "word_embed = Embedding(len(token_tokenizer.word_index)+1, word_embedding_dim, \n",
    "                       weights=[embedding_matrix], input_length=max_sentence_len,\n",
    "                       trainable=True, mask_zero=True, \n",
    "                       name='Glove_Embedding')(word_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sJL8K7CkK9XZ"
   },
   "source": [
    "# BiLSTM + CRF\n",
    "We concatenate character- and word-level informations and pass it to a bidirectional LSTM:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "S-gjEgMfK8oL"
   },
   "outputs": [],
   "source": [
    "x = concatenate([word_embed, cnn_out], axis=-1)\n",
    "x = Dropout(drop)(x)\n",
    "x = Bidirectional(LSTM(hidden_cells, return_sequences=True, dropout=drop))(x)\n",
    "x = Dense(len(output_labels), activation='relu', name='Dense_Layer')(x)\n",
    "crf = CRF(len(output_labels), dtype='float32', name='CRF_Layer')\n",
    "out = crf(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Bvj86RdcyAFv"
   },
   "outputs": [],
   "source": [
    "model = Model(\n",
    "    inputs=[cnn_input, word_input],\n",
    "    outputs=out\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 605
    },
    "colab_type": "code",
    "id": "hR-_OjLb1Xk0",
    "outputId": "1d3b5c2f-3a9c-4511-ea32-385590ece220"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"functional_1\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "char_encoding (InputLayer)      [(None, 50, 93)]     0                                            \n",
      "__________________________________________________________________________________________________\n",
      "cnn_Embedding (TimeDistributed) (None, 50, 93, 30)   2850        char_encoding[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "dropout (Dropout)               (None, 50, 93, 30)   0           cnn_Embedding[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "cnn_Convolution1d (TimeDistribu (None, 50, 93, 30)   2730        dropout[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "word_encoding (InputLayer)      [(None, 50)]         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "cnn_MaxPooling1d (TimeDistribut (None, 50, 1, 30)    0           cnn_Convolution1d[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "Glove_Embedding (Embedding)     (None, 50, 100)      10827800    word_encoding[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "cnn_Flatten (TimeDistributed)   (None, 50, 30)       0           cnn_MaxPooling1d[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "concatenate (Concatenate)       (None, 50, 130)      0           Glove_Embedding[0][0]            \n",
      "                                                                 cnn_Flatten[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 50, 130)      0           concatenate[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional (Bidirectional)   (None, 50, 400)      529600      dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "Dense_Layer (Dense)             (None, 50, 9)        3609        bidirectional[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "CRF_Layer (CRF)                 (None, 50)           81          Dense_Layer[0][0]                \n",
      "==================================================================================================\n",
      "Total params: 11,366,670\n",
      "Trainable params: 11,366,670\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.compile(\n",
    "    loss=crf.loss, \n",
    "    optimizer='adam',\n",
    "    metrics=[crf.accuracy]\n",
    ")\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BliGFagr3tNE"
   },
   "outputs": [],
   "source": [
    "early_stopping_callback = EarlyStopping(monitor=\"val_loss\",\n",
    "                                        patience=3, min_delta=0.001, verbose=1, \n",
    "                                        restore_best_weights=True)\n",
    "# early_stopping_callback = EarlyStopping(monitor=\"val_accuracy\",\n",
    "#                                         patience=3, min_delta=0.005, verbose=1, \n",
    "#                                         restore_best_weights=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "# Train-test split\n",
    "X_sent_train, X_sent_test, Y_train, Y_test = train_test_split(X_sent, Y, test_size=0.2, random_state=3791)\n",
    "X_char_train, X_char_test, _, _ = train_test_split(X_char, Y, test_size=0.2, random_state=3791)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 689
    },
    "colab_type": "code",
    "id": "4m1DOCgVHgom",
    "outputId": "e9d09231-c5f1-4cf7-fbca-6b62a130681a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "9098/9098 [==============================] - 1627s 179ms/step - loss: 1.4325 - accuracy: 0.9534 - val_loss: 5.2488 - val_accuracy: 0.9734\n",
      "Epoch 2/20\n",
      "9098/9098 [==============================] - 1606s 176ms/step - loss: 0.8238 - accuracy: 0.9695 - val_loss: 4.6694 - val_accuracy: 0.9772\n",
      "Epoch 3/20\n",
      "9098/9098 [==============================] - 1609s 177ms/step - loss: 0.7119 - accuracy: 0.9733 - val_loss: 4.9425 - val_accuracy: 0.9790\n",
      "Epoch 4/20\n",
      "9098/9098 [==============================] - 1611s 177ms/step - loss: 0.6466 - accuracy: 0.9755 - val_loss: 5.3370 - val_accuracy: 0.9792\n",
      "Epoch 5/20\n",
      "9098/9098 [==============================] - ETA: 0s - loss: 0.5991 - accuracy: 0.9771 ETA: 1s - loss: 0.5992 - accuRestoring model weights from the end of the best epoch.\n",
      "9098/9098 [==============================] - 1611s 177ms/step - loss: 0.5991 - accuracy: 0.9771 - val_loss: 5.8173 - val_accuracy: 0.9797\n",
      "Epoch 00005: early stopping\n"
     ]
    }
   ],
   "source": [
    "history = model.fit([X_char_train, X_sent_train],\n",
    "    Y_train, \n",
    "    batch_size=batch_size, \n",
    "    epochs=epochs,\n",
    "    verbose=1,\n",
    "    callbacks=[early_stopping_callback],\n",
    "    validation_split=0.2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Gg3qkwMFhaEB"
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yYvfaGaVhb5-"
   },
   "source": [
    "## Evaluation\n",
    "We evaluate three aspects of the model:\n",
    "* **Memory consumption** using the `kerasutils.print_model_memory_usage()` function (found [here](https://stackoverflow.com/questions/43137288/how-to-determine-needed-memory-of-keras-model));\n",
    "* **Latency in prediction** using the function `time.process_time()`;\n",
    "* **F1-score** _on entities_ on the test set using `seqeval`;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "qH5-b5wRXRpx",
    "outputId": "7686d741-d915-423f-88f7-5a3963bef76e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model size: 61.087 MB\n"
     ]
    }
   ],
   "source": [
    "kerasutils.print_model_memory_usage(batch_size, model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "XjCR4W95Xa2l",
    "outputId": "0343233e-f9b7-47f4-e679-d41c209d0cd3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model latency in predictions: 0.00702 s\n"
     ]
    }
   ],
   "source": [
    "print(f'Model latency in predictions: {modelutils.compute_prediction_latency([X_char_test, X_sent_test], model, n_instances=len(X_sent_test)):.3} s')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 924
    },
    "colab_type": "code",
    "id": "GjTgiaPWX96F",
    "outputId": "98ce18a5-5d8c-495e-d4f0-e4ecc4eb394d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Set\n",
      "           precision    recall  f1-score   support\n",
      "\n",
      "      LOC      0.837     0.897     0.866     68020\n",
      "      ORG      0.847     0.739     0.790     39297\n",
      "     MISC      0.791     0.750     0.770     58442\n",
      "      PER      0.921     0.952     0.936     76219\n",
      "\n",
      "micro avg      0.855     0.853     0.854    241978\n",
      "macro avg      0.854     0.853     0.852    241978\n",
      "\n",
      "\n",
      "\n",
      "Test Set\n",
      "           precision    recall  f1-score   support\n",
      "\n",
      "     MISC      0.776     0.732     0.753     14427\n",
      "      PER      0.908     0.946     0.927     19192\n",
      "      LOC      0.826     0.886     0.855     17119\n",
      "      ORG      0.830     0.721     0.772      9760\n",
      "\n",
      "micro avg      0.842     0.841     0.842     60498\n",
      "macro avg      0.841     0.841     0.840     60498\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from seqeval.metrics import classification_report\n",
    "\n",
    "\n",
    "datasets = [('Training Set', X_char_train, X_sent_train, Y_train), \n",
    "            ('Test Set', X_char_test, X_sent_test, Y_test)]\n",
    "\n",
    "for title, X_char, X_sent, Y in datasets:\n",
    "    Y_pred = model.predict({'char_encoding': X_char, 'word_encoding': X_sent}, batch_size=batch_size)\n",
    "    Y, Y_pred = kerasutils.remove_seq_padding(X_sent, Y, Y_pred)\n",
    "    Y, Y_pred = modelutils.from_encode_to_literal_labels(Y, Y_pred, idx2tag)\n",
    "    print(title)\n",
    "    print(classification_report(Y, Y_pred, digits=3))\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Glove+CNN+BiLSTM+CRF_CoNLL.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}