LSTM-WiNER-IT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM for Entity Extraction on WikiNER (Italian)\n",
    "\n",
    "WikiNER is a dataset of annotated sentences for Entity Extraction taken from Wikipedia. In this notebook, we train and evaluate a Bidirectional LSTM neural network model on the italian WikiNER dataset to recognize Person, Locations and Organizations.\n",
    "\n",
    "We use `tf.keras.preprocessing.text.Tokenizer` for text preprocessing, we pad all the santences to the same length and load [this word2vec embedding](http://www.italianlp.it/resources/italian-word-embeddings/) for token encoding, then we use `tensorflow.keras` to build the model. Evaluation is made with the `seqeval` package.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "from utils import dataio, kerasutils, modelutils\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.utils import to_categorical, plot_model\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "from seqeval.metrics import classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Dataset\n",
    "\n",
    "Thanks to [this repo](https://github.com/dice-group/FOX/blob/master/input/Wikiner/aij-wikiner-it-wp3.bz2) that makes WikiNER data easily available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Read 127940 sentences.\n"
     ]
    }
   ],
   "source": [
    "file_path = os.path.join('data', 'wikiner-it-wp3-raw.txt')\n",
    "sentences, tags, output_labels = dataio.load_wikiner(file_path, token_only=True)\n",
    "\n",
    "# A specific text preprocessing is required to effectively use itWac italian\n",
    "# word embedding\n",
    "sentences = dataio.itwac_preprocess_data(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'B-LOC', 'I-MISC', 'I-LOC', 'I-ORG', 'B-MISC', 'B-ORG', 'O', 'I-PER', 'B-PER'}\n",
      "['Seguirono', 'Lamarck', '(', '1744', '--', '1829', ')', ',', 'Blumenbach', '(', '1752', '--', '1840', ')', ',', 'con', 'le', 'sue', 'norme', 'descrittive', 'del', 'cranio', ',', 'Paul', 'Broca', 'con', 'la', 'focalizzazione', 'dei', 'rapporti', 'tra', 'morfologia', 'e', 'funzionalità', '.']\n",
      "['O', 'I-PER', 'O', 'O', 'O', 'O', 'O', 'O', 'I-PER', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'I-PER', 'I-PER', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']\n"
     ]
    }
   ],
   "source": [
    "print(output_labels)\n",
    "print(sentences[1])\n",
    "print(tags[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Text preprocessing and token encoding\n",
    "\n",
    "#### Token Ordinal Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'B-LOC': 0, 'I-MISC': 1, 'I-LOC': 2, 'I-ORG': 3, 'B-MISC': 4, 'B-ORG': 5, 'O': 6, 'I-PER': 7, 'B-PER': 8}\n"
     ]
    }
   ],
   "source": [
    "tokenizer = Tokenizer(lower=False)\n",
    "tokenizer.fit_on_texts(sentences)\n",
    "X = tokenizer.texts_to_sequences(sentences)\n",
    "\n",
    "tag2idx = { tag: idx for idx, tag in enumerate(output_labels) }\n",
    "print(tag2idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx2tag = { idx: tag for tag, idx in tag2idx.items() }\n",
    "tags = [[tag2idx[tag] for tag in sentence] for sentence in tags]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Seguirono', 'Lamarck', '(', '1744', '--', '1829', ')', ',', 'Blumenbach', '(', '1752', '--', '1840', ')', ',', 'con', 'le', 'sue', 'norme', 'descrittive', 'del', 'cranio', ',', 'Paul', 'Broca', 'con', 'la', 'focalizzazione', 'dei', 'rapporti', 'tra', 'morfologia', 'e', 'funzionalità', '.']\n",
      "[9039, 26311, 21, 22394, 73, 9233, 20, 1, 68504, 21, 16395, 73, 9040, 20, 1, 16, 24, 139, 4538, 42627, 10, 19505, 1, 1837, 42628, 16, 6, 51767, 28, 974, 56, 10484, 4, 4599, 3]\n",
      "  9039 | Seguirono\n",
      " 26311 | Lamarck\n",
      "    21 | (\n",
      " 22394 | 1744\n",
      "    73 | --\n",
      "  9233 | 1829\n",
      "    20 | )\n",
      "     1 | ,\n",
      " 68504 | Blumenbach\n",
      "    21 | (\n",
      " 16395 | 1752\n",
      "    73 | --\n",
      "  9040 | 1840\n",
      "    20 | )\n",
      "     1 | ,\n",
      "    16 | con\n",
      "    24 | le\n",
      "   139 | sue\n",
      "  4538 | norme\n",
      " 42627 | descrittive\n",
      "    10 | del\n",
      " 19505 | cranio\n",
      "     1 | ,\n",
      "  1837 | Paul\n",
      " 42628 | Broca\n",
      "    16 | con\n",
      "     6 | la\n",
      " 51767 | focalizzazione\n",
      "    28 | dei\n",
      "   974 | rapporti\n",
      "    56 | tra\n",
      " 10484 | morfologia\n",
      "     4 | e\n",
      "  4599 | funzionalità\n",
      "     3 | .\n"
     ]
    }
   ],
   "source": [
    "print(sentences[1])\n",
    "print(X[1])\n",
    "for i in X[1]:\n",
    "    print(f'{i:6} | {tokenizer.index_word[i]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "117893\n"
     ]
    }
   ],
   "source": [
    "vocabulary_size = len(tokenizer.word_counts)\n",
    "print(vocabulary_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Seguirono', 'Lamarck', '(', '1744', '--', '1829', ')', ',', 'Blumenbach', '(', '1752', '--', '1840', ')', ',', 'con', 'le', 'sue', 'norme', 'descrittive', 'del', 'cranio', ',', 'Paul', 'Broca', 'con', 'la', 'focalizzazione', 'dei', 'rapporti', 'tra', 'morfologia', 'e', 'funzionalità', '.']\n",
      "[6, 7, 6, 6, 6, 6, 6, 6, 7, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6]\n",
      "6 : O\n",
      "7 : I-PER\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "7 : I-PER\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "7 : I-PER\n",
      "7 : I-PER\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n",
      "6 : O\n"
     ]
    }
   ],
   "source": [
    "print(sentences[1])\n",
    "print(tags[1])\n",
    "for i in tags[1]:\n",
    "    print(f'{i} : {idx2tag[i]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sequence Padding\n",
    "\n",
    "The input sequence of an LSTM model must have a fixed length. We choose the most appropriate seqence length given the length of the sentences of the dataset, than we pad shorter sentences and truncate the longer ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Longest sequence: 206\n",
      "[('75%', 35.0), ('80%', 38.0), ('85%', 42.0), ('90%', 47.0), ('95%', 56.0), ('100%', 206.0)]\n"
     ]
    }
   ],
   "source": [
    "sequence_len = np.array([len(s) for s in sentences])\n",
    "longest_sequence = sequence_len.max()\n",
    "print(f'Longest sequence: {longest_sequence}')\n",
    "\n",
    "print([(str(p) + '%', np.percentile(sequence_len, p)) for p in range(75,101, 5)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_sequence_len = 60\n",
    "X = pad_sequences(X, maxlen=max_sequence_len, padding='post', truncating='post')\n",
    "\n",
    "y = pad_sequences(tags, maxlen=max_sequence_len, value=tag2idx['O'], padding='post', truncating='post')\n",
    "y = to_categorical(y, num_classes=len(output_labels), dtype='int32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.index_word[0] = '_PAD_'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array(X)\n",
    "y = np.array(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(127940, 60)\n",
      "(127940, 60, 9)\n"
     ]
    }
   ],
   "source": [
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build, train and evaluate an LSTM model\n",
    "\n",
    "itWac embedding format is equal to the Glove embedding one, so we use the `load_glove_embeddings_matrix()` function. This and the function which creates the LSTM model can be found in the `utils/kerasutils.py` module. Training stopping criterion is Early Stopping with patience on the loss value on validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "USE_W2V=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 1247492 word vectors.\n"
     ]
    }
   ],
   "source": [
    "w2v_matrix=None\n",
    "if USE_W2V:\n",
    "    w2v_embedding_path = os.path.join('embeddings', 'w2v.itWac.128d.txt')\n",
    "    embedding_dim = 128\n",
    "    w2v_matrix = kerasutils.load_glove_embedding_matrix(w2v_embedding_path, tokenizer.word_index, embedding_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, 60, 128)           15090432  \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 60, 128)           0         \n",
      "_________________________________________________________________\n",
      "bidirectional (Bidirectional (None, 60, 400)           526400    \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 60, 9)             3609      \n",
      "=================================================================\n",
      "Total params: 15,620,441\n",
      "Trainable params: 15,620,441\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = kerasutils.create_paper_BiLSTM(vocabulary_size+1, max_sequence_len, len(output_labels), \n",
    "                                 use_glove=USE_W2V, glove_matrix=w2v_matrix, embed_dim = 128)\n",
    "\n",
    "# Early stopping with patience on validation loss\n",
    "early_stopping_callback = EarlyStopping(monitor=\"val_loss\", min_delta=0.01, patience=3, verbose=1, mode=\"auto\", restore_best_weights=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create train-test-dev split. Fix random_state to improve repeatability.\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=3791)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.2, random_state=3791)\n",
    "batch_size = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "8189/8189 [==============================] - 1694s 207ms/step - loss: 0.0431 - accuracy: 0.9710 - precision: 0.9764 - recall: 0.9659 - val_loss: 0.0229 - val_accuracy: 0.9839 - val_precision: 0.9854 - val_recall: 0.9825\n",
      "Epoch 2/50\n",
      "8189/8189 [==============================] - 1656s 202ms/step - loss: 0.0195 - accuracy: 0.9864 - precision: 0.9876 - recall: 0.9854 - val_loss: 0.0200 - val_accuracy: 0.9860 - val_precision: 0.9870 - val_recall: 0.9851\n",
      "Epoch 3/50\n",
      "8189/8189 [==============================] - 1660s 203ms/step - loss: 0.0142 - accuracy: 0.9901 - precision: 0.9908 - recall: 0.9895 - val_loss: 0.0198 - val_accuracy: 0.9864 - val_precision: 0.9872 - val_recall: 0.9858\n",
      "Epoch 4/50\n",
      "8189/8189 [==============================] - ETA: 0s - loss: 0.0115 - accuracy: 0.9920 - precision: 0.9924 - recall: 0.9915Restoring model weights from the end of the best epoch.\n",
      "8189/8189 [==============================] - 1636s 200ms/step - loss: 0.0115 - accuracy: 0.9920 - precision: 0.9924 - recall: 0.9915 - val_loss: 0.0204 - val_accuracy: 0.9866 - val_precision: 0.9873 - val_recall: 0.9860\n",
      "Epoch 00004: early stopping\n",
      "Wall time: 1h 50min 55s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "history = model.fit(X_train, \n",
    "          y_train, \n",
    "          batch_size=batch_size, \n",
    "          epochs=50,\n",
    "          verbose=1,\n",
    "          callbacks=[early_stopping_callback],\n",
    "          validation_data=(X_valid, y_valid)\n",
    "         )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Evaluation\n",
    "We evaluate three aspects of the model:\n",
    "* **Memory consumption** using the `kerasutils.print_model_memory_usage()` function (found [here](https://stackoverflow.com/questions/43137288/how-to-determine-needed-memory-of-keras-model));\n",
    "* **Latency in prediction** using the function `time.process_time()`;\n",
    "* **F1-score** _on entities_ on the test set using `seqeval`;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model size: 61.109 MB\n"
     ]
    }
   ],
   "source": [
    "kerasutils.print_model_memory_usage(batch_size, model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model latency in prediction: 0.00499 s\n"
     ]
    }
   ],
   "source": [
    "print(f'Model latency in prediction: {modelutils.compute_prediction_latency(X_test, model):.3} s')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets = [('Training Set', X_train, y_train), ('Valid Set', X_valid, y_valid), ('Test Set', X_test, y_test)]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Set\n",
      "           precision    recall  f1-score   support\n",
      "\n",
      "      LOC      0.905     0.926     0.916     81845\n",
      "     MISC      0.781     0.768     0.774     24096\n",
      "      PER      0.929     0.952     0.940     45352\n",
      "      ORG      0.868     0.794     0.829     13541\n",
      "\n",
      "micro avg      0.891     0.899     0.895    164834\n",
      "macro avg      0.891     0.899     0.895    164834\n",
      "\n",
      "\n",
      "\n",
      "Valid Set\n",
      "           precision    recall  f1-score   support\n",
      "\n",
      "      LOC      0.877     0.897     0.887     20070\n",
      "      ORG      0.826     0.730     0.775      3412\n",
      "     MISC      0.719     0.720     0.720      6037\n",
      "      PER      0.899     0.925     0.911     11251\n",
      "\n",
      "micro avg      0.856     0.864     0.860     40770\n",
      "macro avg      0.855     0.864     0.859     40770\n",
      "\n",
      "\n",
      "\n",
      "Test Set\n",
      "           precision    recall  f1-score   support\n",
      "\n",
      "      ORG      0.828     0.746     0.785      4175\n",
      "     MISC      0.722     0.717     0.720      7317\n",
      "      LOC      0.883     0.901     0.892     25648\n",
      "      PER      0.905     0.930     0.917     14017\n",
      "\n",
      "micro avg      0.862     0.870     0.866     51157\n",
      "macro avg      0.861     0.870     0.866     51157\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for title, X, Y in datasets:\n",
    "    Y_pred = model.predict(X, batch_size=batch_size)\n",
    "    Y_pred = np.array(np.argmax(Y_pred, axis=-1))\n",
    "    Y = np.array(np.argmax(Y, axis=-1))\n",
    "    Y, Y_pred = kerasutils.remove_seq_padding(X, Y, Y_pred)\n",
    "    Y, Y_pred = modelutils.from_encode_to_literal_labels(Y, Y_pred, idx2tag)\n",
    "    print(title)\n",
    "    print(classification_report(Y, Y_pred, digits=3))\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}