l10c04_nlp_optimizing_the_text_generation_model.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "punL79CN7Ox6"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "cellView": "form",
        "id": "_ckMIh7O7s6D"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ph5eir3Pf-3z"
      },
      "source": [
        "# Optimizing the Text Generation Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S5Uhzt6vVIB2"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l10c04_nlp_optimizing_the_text_generation_model.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l10c04_nlp_optimizing_the_text_generation_model.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dCxhW3mtLmfb"
      },
      "source": [
        "You've already done some amazing work with generating new songs, but so far we've seen some issues with repetition and a fair amount of incoherence. By using more data and further tweaking the model, you'll be able to get improved results. We'll once again use the [Kaggle Song Lyrics Dataset](https://www.kaggle.com/mousehead/songlyrics) here."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4aHK2CYygXom"
      },
      "source": [
        "## Import TensorFlow and related functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "2LmLTREBf5ng"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "# Other imports for processing data\n",
        "import string\n",
        "import numpy as np\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GmLTO_dpgge9"
      },
      "source": [
        "## Get the Dataset\n",
        "\n",
        "As noted above, we'll utilize the [Song Lyrics dataset](https://www.kaggle.com/mousehead/songlyrics) on Kaggle again."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "4Bf5FVHfganK",
        "outputId": "79161ede-888d-4cf4-e6b8-5e29afcfdded",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-12-21 18:51:03--  https://drive.google.com/uc?id=1LiJFZd41ofrWoBtW-pMYsfz1w8Ny0Bj8\n",
            "Resolving drive.google.com (drive.google.com)... 173.194.197.102, 173.194.197.100, 173.194.197.139, ...\n",
            "Connecting to drive.google.com (drive.google.com)|173.194.197.102|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Moved Temporarily\n",
            "Location: https://doc-04-ak-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/4d4ll6g8kb4dtuoff28kdskn3a2i1asl/1640112600000/11118900490791463723/*/1LiJFZd41ofrWoBtW-pMYsfz1w8Ny0Bj8 [following]\n",
            "Warning: wildcards not supported in HTTP.\n",
            "--2021-12-21 18:51:05--  https://doc-04-ak-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/4d4ll6g8kb4dtuoff28kdskn3a2i1asl/1640112600000/11118900490791463723/*/1LiJFZd41ofrWoBtW-pMYsfz1w8Ny0Bj8\n",
            "Resolving doc-04-ak-docs.googleusercontent.com (doc-04-ak-docs.googleusercontent.com)... 74.125.69.132, 2607:f8b0:4001:c08::84\n",
            "Connecting to doc-04-ak-docs.googleusercontent.com (doc-04-ak-docs.googleusercontent.com)|74.125.69.132|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 72436445 (69M) [text/csv]\n",
            "Saving to: ‘/tmp/songdata.csv’\n",
            "\n",
            "/tmp/songdata.csv   100%[===================>]  69.08M   122MB/s    in 0.6s    \n",
            "\n",
            "2021-12-21 18:51:06 (122 MB/s) - ‘/tmp/songdata.csv’ saved [72436445/72436445]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget --no-check-certificate \\\n",
        "    https://drive.google.com/uc?id=1LiJFZd41ofrWoBtW-pMYsfz1w8Ny0Bj8 \\\n",
        "    -O /tmp/songdata.csv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jz9x-7dWihxx"
      },
      "source": [
        "## 250 Songs\n",
        "\n",
        "Now we've seen a model trained on just a small sample of songs, and how this often leads to repetition as you get further along in trying to generate new text. Let's switch to using the 250 songs instead, and see if our output improves. This will actually be nearly 10K lines of lyrics, which should be sufficient.\n",
        "\n",
        "Note that we won't use the full dataset here as it will take up quite a bit of RAM and processing time, but you're welcome to try doing so on your own later. If interested, you'll likely want to use only some of the more common words for the Tokenizer, which will help shrink processing time and memory needed (or else you'd have an output array hundreds of thousands of words long)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nWbMN_19jfRT"
      },
      "source": [
        "### Preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "LRmPPJegovBe"
      },
      "outputs": [],
      "source": [
        "def tokenize_corpus(corpus, num_words=-1):\n",
        "  # Fit a Tokenizer on the corpus\n",
        "  if num_words > -1:\n",
        "    tokenizer = Tokenizer(num_words=num_words)\n",
        "  else:\n",
        "    tokenizer = Tokenizer()\n",
        "  tokenizer.fit_on_texts(corpus)\n",
        "  return tokenizer\n",
        "\n",
        "def create_lyrics_corpus(dataset, field):\n",
        "  # Remove all other punctuation\n",
        "  dataset[field] = dataset[field].str.replace('[{}]'.format(string.punctuation), '')\n",
        "  # Make it lowercase\n",
        "  dataset[field] = dataset[field].str.lower()\n",
        "  # Make it one long string to split by line\n",
        "  lyrics = dataset[field].str.cat()\n",
        "  corpus = lyrics.split('\\n')\n",
        "  # Remove any trailing whitespace\n",
        "  for l in range(len(corpus)):\n",
        "    corpus[l] = corpus[l].rstrip()\n",
        "  # Remove any empty lines\n",
        "  corpus = [l for l in corpus if l != '']\n",
        "\n",
        "  return corpus"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "kIGedF3XjHj4",
        "outputId": "f51fe842-b3ea-4323-c8e1-7313981b0b09",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2000\n"
          ]
        }
      ],
      "source": [
        "def tokenize_corpus(corpus, num_words=-1):\n",
        "  # Fit a Tokenizer on the corpus\n",
        "  if num_words > -1:\n",
        "    tokenizer = Tokenizer(num_words=num_words)\n",
        "  else:\n",
        "    tokenizer = Tokenizer()\n",
        "  tokenizer.fit_on_texts(corpus)\n",
        "  return tokenizer\n",
        "\n",
        "# Read the dataset from csv - this time with 250 songs\n",
        "dataset = pd.read_csv('/tmp/songdata.csv', dtype=str)[:250]\n",
        "# Create the corpus using the 'text' column containing lyrics\n",
        "corpus = create_lyrics_corpus(dataset, 'text')\n",
        "# Tokenize the corpus\n",
        "tokenizer = tokenize_corpus(corpus, num_words=2000)\n",
        "total_words = tokenizer.num_words\n",
        "\n",
        "# There should be a lot more words now\n",
        "print(total_words)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "quoDmw_FkNBA"
      },
      "source": [
        "### Create Sequences and Labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "kkLAf3HmkPSo"
      },
      "outputs": [],
      "source": [
        "sequences = []\n",
        "for line in corpus:\n",
        "\ttoken_list = tokenizer.texts_to_sequences([line])[0]\n",
        "\tfor i in range(1, len(token_list)):\n",
        "\t\tn_gram_sequence = token_list[:i+1]\n",
        "\t\tsequences.append(n_gram_sequence)\n",
        "\n",
        "# Pad sequences for equal input length \n",
        "max_sequence_len = max([len(seq) for seq in sequences])\n",
        "sequences = np.array(pad_sequences(sequences, maxlen=max_sequence_len, padding='pre'))\n",
        "\n",
        "# Split sequences between the \"input\" sequence and \"output\" predicted word\n",
        "input_sequences, labels = sequences[:,:-1], sequences[:,-1]\n",
        "# One-hot encode the labels\n",
        "one_hot_labels = tf.keras.utils.to_categorical(labels, num_classes=total_words)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cECbqT-blMk-"
      },
      "source": [
        "### Train a (Better) Text Generation Model\n",
        "\n",
        "With more data, we'll cut off after 100 epochs to avoid keeping you here all day. You'll also want to change your runtime type to GPU if you haven't already (you'll need to re-run the above cells if you change runtimes)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "7nHOp6uWlP_P",
        "outputId": "af2b4009-e15c-4615-dd4c-89b4014fd667",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/100\n",
            "1480/1480 [==============================] - 35s 15ms/step - loss: 5.9801 - accuracy: 0.0465\n",
            "Epoch 2/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 5.6683 - accuracy: 0.0515\n",
            "Epoch 3/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 5.4765 - accuracy: 0.0684\n",
            "Epoch 4/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 5.3058 - accuracy: 0.0932\n",
            "Epoch 5/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 5.1513 - accuracy: 0.1167\n",
            "Epoch 6/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 4.9909 - accuracy: 0.1293\n",
            "Epoch 7/100\n",
            "1480/1480 [==============================] - 22s 15ms/step - loss: 4.8141 - accuracy: 0.1459\n",
            "Epoch 8/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 4.6599 - accuracy: 0.1619\n",
            "Epoch 9/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 4.5205 - accuracy: 0.1791\n",
            "Epoch 10/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 4.3936 - accuracy: 0.1930\n",
            "Epoch 11/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 4.2832 - accuracy: 0.2046\n",
            "Epoch 12/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 4.1823 - accuracy: 0.2168\n",
            "Epoch 13/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 4.0899 - accuracy: 0.2288\n",
            "Epoch 14/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 4.0100 - accuracy: 0.2382\n",
            "Epoch 15/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.9333 - accuracy: 0.2483\n",
            "Epoch 16/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.8620 - accuracy: 0.2572\n",
            "Epoch 17/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.7972 - accuracy: 0.2670\n",
            "Epoch 18/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.7356 - accuracy: 0.2737\n",
            "Epoch 19/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.6779 - accuracy: 0.2829\n",
            "Epoch 20/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.6226 - accuracy: 0.2911\n",
            "Epoch 21/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.5721 - accuracy: 0.2979\n",
            "Epoch 22/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.5271 - accuracy: 0.3048\n",
            "Epoch 23/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.4794 - accuracy: 0.3116\n",
            "Epoch 24/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.4329 - accuracy: 0.3191\n",
            "Epoch 25/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.3937 - accuracy: 0.3235\n",
            "Epoch 26/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.3511 - accuracy: 0.3284\n",
            "Epoch 27/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.3155 - accuracy: 0.3345\n",
            "Epoch 28/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.2768 - accuracy: 0.3395\n",
            "Epoch 29/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.2422 - accuracy: 0.3454\n",
            "Epoch 30/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.2110 - accuracy: 0.3502\n",
            "Epoch 31/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.1831 - accuracy: 0.3528\n",
            "Epoch 32/100\n",
            "1480/1480 [==============================] - 20s 14ms/step - loss: 3.1467 - accuracy: 0.3593\n",
            "Epoch 33/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.1233 - accuracy: 0.3636\n",
            "Epoch 34/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.0939 - accuracy: 0.3679\n",
            "Epoch 35/100\n",
            "1480/1480 [==============================] - 20s 14ms/step - loss: 3.0667 - accuracy: 0.3726\n",
            "Epoch 36/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.0396 - accuracy: 0.3765\n",
            "Epoch 37/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 3.0279 - accuracy: 0.3775\n",
            "Epoch 38/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.9924 - accuracy: 0.3840\n",
            "Epoch 39/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.9733 - accuracy: 0.3872\n",
            "Epoch 40/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.9439 - accuracy: 0.3921\n",
            "Epoch 41/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.9247 - accuracy: 0.3938\n",
            "Epoch 42/100\n",
            "1480/1480 [==============================] - 21s 15ms/step - loss: 2.9096 - accuracy: 0.3964\n",
            "Epoch 43/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.8871 - accuracy: 0.4002\n",
            "Epoch 44/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.8674 - accuracy: 0.4032\n",
            "Epoch 45/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.8515 - accuracy: 0.4074\n",
            "Epoch 46/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.8316 - accuracy: 0.4096\n",
            "Epoch 47/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.8063 - accuracy: 0.4151\n",
            "Epoch 48/100\n",
            "1480/1480 [==============================] - 20s 14ms/step - loss: 2.7952 - accuracy: 0.4168\n",
            "Epoch 49/100\n",
            "1480/1480 [==============================] - 20s 14ms/step - loss: 2.7768 - accuracy: 0.4198\n",
            "Epoch 50/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.7676 - accuracy: 0.4207\n",
            "Epoch 51/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.7525 - accuracy: 0.4243\n",
            "Epoch 52/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.7335 - accuracy: 0.4257\n",
            "Epoch 53/100\n",
            "1480/1480 [==============================] - 20s 14ms/step - loss: 2.7162 - accuracy: 0.4295\n",
            "Epoch 54/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.6993 - accuracy: 0.4345\n",
            "Epoch 55/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.6914 - accuracy: 0.4343\n",
            "Epoch 56/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.6793 - accuracy: 0.4371\n",
            "Epoch 57/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.6687 - accuracy: 0.4388\n",
            "Epoch 58/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.6456 - accuracy: 0.4438\n",
            "Epoch 59/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.6741 - accuracy: 0.4360\n",
            "Epoch 60/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.6204 - accuracy: 0.4462\n",
            "Epoch 61/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.6074 - accuracy: 0.4477\n",
            "Epoch 62/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.5951 - accuracy: 0.4503\n",
            "Epoch 63/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.5877 - accuracy: 0.4520\n",
            "Epoch 64/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.5773 - accuracy: 0.4543\n",
            "Epoch 65/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.5597 - accuracy: 0.4574\n",
            "Epoch 66/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.5723 - accuracy: 0.4563\n",
            "Epoch 67/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.5493 - accuracy: 0.4586\n",
            "Epoch 68/100\n",
            "1480/1480 [==============================] - 20s 14ms/step - loss: 2.5363 - accuracy: 0.4608\n",
            "Epoch 69/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.5195 - accuracy: 0.4650\n",
            "Epoch 70/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.5098 - accuracy: 0.4650\n",
            "Epoch 71/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.4988 - accuracy: 0.4683\n",
            "Epoch 72/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.4908 - accuracy: 0.4684\n",
            "Epoch 73/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.4864 - accuracy: 0.4711\n",
            "Epoch 74/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.5171 - accuracy: 0.4632\n",
            "Epoch 75/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.4732 - accuracy: 0.4716\n",
            "Epoch 76/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.4583 - accuracy: 0.4748\n",
            "Epoch 77/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.4474 - accuracy: 0.4766\n",
            "Epoch 78/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.4370 - accuracy: 0.4780\n",
            "Epoch 79/100\n",
            "1480/1480 [==============================] - 20s 14ms/step - loss: 2.4351 - accuracy: 0.4778\n",
            "Epoch 80/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.4282 - accuracy: 0.4793\n",
            "Epoch 81/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.4136 - accuracy: 0.4820\n",
            "Epoch 82/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.4093 - accuracy: 0.4836\n",
            "Epoch 83/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.4063 - accuracy: 0.4829\n",
            "Epoch 84/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3990 - accuracy: 0.4866\n",
            "Epoch 85/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3911 - accuracy: 0.4851\n",
            "Epoch 86/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3777 - accuracy: 0.4909\n",
            "Epoch 87/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3708 - accuracy: 0.4897\n",
            "Epoch 88/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3665 - accuracy: 0.4890\n",
            "Epoch 89/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3518 - accuracy: 0.4936\n",
            "Epoch 90/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3403 - accuracy: 0.4953\n",
            "Epoch 91/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3481 - accuracy: 0.4924\n",
            "Epoch 92/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3336 - accuracy: 0.4958\n",
            "Epoch 93/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3306 - accuracy: 0.4970\n",
            "Epoch 94/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3256 - accuracy: 0.4983\n",
            "Epoch 95/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3192 - accuracy: 0.4990\n",
            "Epoch 96/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3113 - accuracy: 0.5006\n",
            "Epoch 97/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.3018 - accuracy: 0.5021\n",
            "Epoch 98/100\n",
            "1480/1480 [==============================] - 21s 14ms/step - loss: 2.2944 - accuracy: 0.5029\n",
            "Epoch 99/100\n",
            "1480/1480 [==============================] - 20s 14ms/step - loss: 2.2952 - accuracy: 0.5026\n",
            "Epoch 100/100\n",
            "1480/1480 [==============================] - 20s 14ms/step - loss: 2.2833 - accuracy: 0.5052\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Embedding(total_words, 64, input_length=max_sequence_len-1))\n",
        "model.add(Bidirectional(LSTM(20)))\n",
        "model.add(Dense(total_words, activation='softmax'))\n",
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "history = model.fit(input_sequences, one_hot_labels, epochs=100, verbose=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MgvIz20nlQcq"
      },
      "source": [
        "### View the Training Graph"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "rOqmmarvlSLh",
        "outputId": "7258e191-9c46-4f43-e470-b775cda69d32",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9b3/8dcnCQkQQsISFhNiACMYUQTD4lpcqmite1u3VnHB2lq97bWtvW3t3ntrW3+1vVZrcd9wV6xURUVvFWWTPSyyJywhhCRkIfvn98eMNiKBCeRkksz7+XjkwZwzZ2Y+xxPnnfP9fs/3mLsjIiKxKy7aBYiISHQpCEREYpyCQEQkxikIRERinIJARCTGJUS7gNbq37+/Z2dnR7sMEZFOZeHChTvdPX1fz3W6IMjOzmbBggXRLkNEpFMxs00tPaemIRGRGKcgEBGJcQoCEZEYpyAQEYlxCgIRkRinIBARiXEKAhGRGBdoEJjZZDNbbWZrzez2fTx/jZkVm9ni8M/1QdYjItIZbSqp4q5Za1hTVBHI+wd2QZmZxQP3AF8ECoH5ZjbD3fP32vRpd785qDpERDqy+sYmNpVUsaWshm1le9hZWUu3+Dh6JsbT0OT8c9l25m3chRmkpyRx5MCUNq8hyCuLxwNr3X09gJlNBy4A9g4CEZGY4O4Ulu7ho82lLNpcxtLCMlZs3U1tQ1OLrxnWP5nvnz2Ci8ZkcFhaj0DqCjIIMoCCZsuFwIR9bHeJmZ0KrAG+6+4Fe29gZlOBqQBZWVkBlCoi0nbKq+tZtX03q4sqWF9cRXFFLcUVtWwsqWJHRS0APRPjGXVYKldNPJxRGb3J7NOTwand6d8riYYmp7qugfpG57DU7phZoPVGe66hV4Cn3L3WzG4EHgFO33sjd78fuB8gLy9P99YUkXbn7mwsqSatRzf6JCd+Zv0H60p4d00xq7ZXsKaogm3lNZ8+n5wYz8De3UlPSeLkI/ozJiuNsYf3YcTAFBLiW+6m7ZXUfl/PQX7SFmBIs+XM8LpPuXtJs8VpwJ0B1iMi0iqNTc7zCwt5a1URCzaWUlJVR2J8HGePGsQV47Mo31PPve+sZUlhOYnxcQwf0IuJw/oxYlAKIwalMHJQCoN6B/8X/aEKMgjmAzlmNpRQAFwGXNF8AzMb7O7bwovnAysDrEdE5FPuzpx1JfRMjCdnYMrn/gJfU1TBD59fyqLNZQzp24NJIwaQl92HNUUVPL+wkFeWbAXg8H49+e1Fx3Dx2Ay6d4uPxq4cssCCwN0bzOxm4HUgHnjQ3VeY2S+BBe4+A7jFzM4HGoBdwDVB1SMi8ok9dY3c9twSXl267dN1GWk9yOjTgwEpSSQlxDNjyRZ6JSXwp68dxwXHHfaZv+p/cPZIXl+xncSEOM7KHbjfJp7OwNw7V5N7Xl6e634EIhKp7eU1vJG/nay+PRmdmcae+kZueHQB+dt2859fPJIjB6awpqiCNUWVbC+vYUdFDSVVdZx51EB+8qWj6NcrKdq70CbMbKG75+3ruWh3FouItInquga2lu2hf68kUnt0o7Q61H7/6AebPjM8MzEhjqT4OB64Oo/TRw4E4KyjB0Wr7A5BQSAinVJTk/P+up28mV/Ews2lrNxWQWNTqIUjKSHUVFPf2MRFYzK58QvD2FlZy5KCcgpLq5lyUjZHDGj7C7M6KwWBiHQatQ2NrNxWwVsri3h+YSFby2vomRjP6Mw0bvrCcIYPSKakso4dFbXUNTRx5YQscsJX4h45MIUTh/eP8h50TAoCEekwiitqWVpYxvItu1m+tZzSqjoSE+JITIijpLKOVdt3U9/oxBmckpPOj849ii/mDuy0o3U6CgWBiERNU5PzzpodzF5VzAfrS1i7oxIAs9DUCgN7d6euoYnK2gZSuidw7clDOS4zjeMP78OA3t2jXH3XoSAQkXa3p66R5z4q5MH3NrBhZxXJifGMG9qXS4/P5PjD+5A7uDfJ7XhlbazTf2kRaXMllbW8tHgrb6zYzk2ThjNpxIBPnyvaXcPFf53DlrI9jM5M5S+Xj2HyqEF06+Rj8TszBYGIHJKFm3bx8uKtVNc10tTklFbX8d7andQ3OilJCdz0+Ec8ecMExmT1obqugesemU9ZdR2PXzeBk47o1+GnX4gFCgIRiUhjk7Om6N9DNAt2VfPAextYsKmUnonxpPXoRlyckZgQxzdOyOareUPom5zIJffO4dqH5/PsN0/kztdWkb91N9OuzuPkHI3g6SgUBCJyQAs3lXLHy8tZsXX3Z9ZnpPXgZ1/O5at5Q1ps03/k2vFccu8czvvLv6ipb+KO83I/vZBLOgYFgYjsk7uzrriSe99Zz/MfFTKod3d+e9ExpKeEplzomRjPhKF9DzjPztD+yTx0zTiumjaXy07MYspJ2e1QvbSGgkBEPtXU5CwqKOWNFUW8kV/Ehp1VdIs3bpo0nJtPO+KgR/KMHpLGgp+eSVKCxvt3RAoCkRjT1OTMXr2DRz/YxNayPRzeL5mh/XtSWdvArPwd7KysJSHOOGF4P649eShn5Q5kYBuM2VcIdFwKApEYsWFnFf9cvo3p8wrYvKuagb2TOCYjjc27qvi/j4vpFmdMGjmAs3IHMmnEAFJ7dIt2ydJOFAQiXVR1XQMfbSrjw/UlvLmyiFXbKwAYl92HH0wewdlH/3vsflOT0+Te6efVl4OjIBDpQkqr6nh12TZmLNnKR5tKaWhy4uOMsVlp/PS8XCaPGkRGWo/PvS4uzohD4/ljlYJApBOrrG1gSUEZiwvKmL9xF++HL+TKGdCLG04dxoShfcnL7tuuN0KXzke/HSKdgLt/7grcdcWVXHTP++yuaQBgWHoyV5+QzUVjM8gd3FtX7ErEFAQiHVhDYxP3vbuO+95dz68vHMWFYzKA0A1Xvvf0YuLijIemjGPskD6k9lTnrhwcBYFIB7W5pJrvPrOYhZtKSU9J4rZnl5DasxunjRjAX95ey5LCcv565VhOazahm8jBUBCIdBDuzsptFczdUMK8Dbt4d00x8XHG3Zcdx+kjB3DZ/R9y0+MLuX3ySO6ZvZaLx2Rw7jGDo122dAEKApEoqGtoolu8YWbsqKjhhY+28Mz8AtbvrAIgs08PvnTMYG49M4fMPj0BeHjKeC69bw4/fyWfjLQe/PyCo6O5C9KFKAhE2tHOylp+9MIyZuUXAZCYEEdDYxNNHhrff+MXhnFyTvo+h3impyTx2LUT+MnLy7n1jCPo3V19AtI2FAQi7eTtVUX84Lml7K5pYOqpw+jRLZ7ahia6d4vjvGMP44gBvQ74Hln9evLotePboVqJJQoCkYAV7KrmrllreHHRFkYOSuGJ6ycyYlBKtMsS+ZSCQCQATU3OhpIqHvtgE0/M3UScGd8+bTi3nJGjydekw1EQiLSRqtoGnpi7ibdX7WDFlt1U1DYQH2d8NW8It56Rw6DUQ5/BUyQICgKRQ1ReXc+T8zbz93+tZ1dVHaMyenPBmMM4JiOVE4f3Z0jfntEuUWS/FAQirbS0sIwnPtzM8q3lbCnbQ1l1PQCTRqRzyxk5jM3qE+UKRVpHQSASga1le3h/7U6enLeZRZvLSE6MJy+7L2Oy0sjs05MThvVj9JC0aJcpclAUBCIt2F1Tzx9eX83s1Tso2LUHCN1/92dfzuWS4zM1jl+6DAWByD7kb93Nt55YSEHpHs4YOYApJw5l/NC+5A7uTVycZvWUrkVBINJMbUMjL3y0hZ/PWEFaz25MnzqRcdl9o12WSKAUBBLzGhqb+Ofy7by2Yjvvri6msraBE4f34+7LxpCekhTt8kQCF2gQmNlk4G4gHpjm7v/TwnaXAM8B49x9QZA1iTQ3Z+1OfvFKPquLKujfK4kvjx7MF3MH8oUjBxCvJiCJEYEFgZnFA/cAXwQKgflmNsPd8/faLgW4FZgbVC0ie1u+pZy/vP0xr68oYkjfHtx31VjOyh2k9n+JSUGeEYwH1rr7egAzmw5cAOTvtd2vgN8B3w+wFhEaGpt4bcV2Hn5/Iws2lZKcGM9tZx3J9acMo3s3TfsgsSvIIMgACpotFwITmm9gZmOBIe7+qpkpCCQQ7s7rK7bzhzfWsHZHJVl9e/LT83L5Sp6GgIpAFDuLzSwOuAu4JoJtpwJTAbKysoItTLqMmvpG3sgv4oH3NrCkoIzh6cn89cqxnH30ILX/izQTZBBsAYY0W84Mr/tECjAKeMfMAAYBM8zs/L07jN39fuB+gLy8PA+wZunkdlbWsmBjKW+vKmLmsu1U1jaQ2acHd156LBePySAhPi7aJYp0OEEGwXwgx8yGEgqAy4ArPnnS3cuB/p8sm9k7wG0aNSSt1dTkPDxnI49/uOnTWz0mJ8ZzzjGDuXhsBhOH9lMnsMh+BBYE7t5gZjcDrxMaPvqgu68ws18CC9x9RlCfLbFjS9kebntmCR+sL2HC0L58bdwQ8rL7MCojVfP+i0Qo0D4Cd58JzNxr3R0tbDspyFqka6mqbWD6/AL+NGsNTe7cecmxfCUvk3Azo4i0gq4slk6luKKWB97bwJNzN7G7JnQF8O8uOVZz/oscAgWBdBr5W3dz9UPzKKmsZfKoQVx/yjDN/S/SBhQE0il8sK6EqY8uIDkpgVdvOYWjBveOdkkiXYaCQDqkxiZna9keNuysYvnWcv4062Oy+vXk0WvHc1haj2iXJ9KlKAikQ2lqcl5ctIU/vrGareU1n64fn92X+79xPGk9E6NYnUjXpCCQDmPO2p38+tWV5G/bzbGZqdx8eg5D+yczLD2ZASlJGhEkEhAFgURdeXU9v341n2cXFpLZpwd3X3YcXz72MF0EJtJOFAQSVbPyi/jxi8soqarjW5OGc8sZOZoJVKSdKQgkKjaXVPOLV1bw1qodjByUwoPXjGNURmq0yxKJSQoCaVc19Y3c+8467n13HQlxxo/OGcmUk4aSmKDJ4ESiRUEg7ebtVUX8fEY+m3dVc96xg/nJl3IZlNo92mWJxDwFgQSuvLqe7z+3hDfyixiensyT10/gxCP6H/iFItIuFAQSqKLdNVz94DzWFVfyw8kjue5kNQOJdDQKAgnMhp1VfP2BuZRW1fHQNeM5OUdnASIdkYJA2tzGnVW8vHgrj3ywEYCnpk7k2My0qNYkIi1TEEibmb9xF7+duZJFm8swg4lD+/Hri0YxPL1XtEsTkf1QEMghq6lv5K5Za/j7v9aTkdaDH50zki+PPkyTw4l0EgoCOSRriiq4+cmPWFNUyZUTsvivc48iOUm/ViKdif6PlYM2K7+I/5i+iJ5JCTw8ZRyTRgyIdkkichAUBNJq7s5f31nHH95YzTEZqdz/9TxdGCbSiSkIpFVKKmv54fPLeHNlERccdxi/u+RYTRIn0skpCCRis1fv4PvPLmX3nnp+el4u156UrXsEiHQBCgI5oKYm587XV3Pfu+sYMTCFx64br3sGi3QhCgLZr+q6Br779GJeX1HE5eOz+NmXc9UUJNLFKAikRUW7a7j+kQUs31qupiCRLkxBIPv06tJt/PilZdQ1NDHtG3mccdTAaJckIgFREMhnlFfXc8eM5by8eCujM1P541eP44gBmiJCpCtTEAgQujbg5cVb+fWr+ZRV1/O9Lx7JtyYNJyFeU0aLdHURBYGZvQA8APzT3ZuCLUna24adVfz4xWXMWVfC6MxUHp4yXvcPFokhkZ4R/BWYAvzZzJ4FHnL31cGVJe1l7Y4Kvvq3D6lvbOJXF47iivFZxMepQ1gklkQUBO7+JvCmmaUCl4cfFwB/Bx539/oAa5SAFOyq5sppc4mPM56/6WSG9k+OdkkiEgURNwCbWT/gGuB6YBFwNzAWmBVIZRKo7eU1XDHtQ2obmnj8ugkKAZEYFmkfwYvACOAx4Mvuvi381NNmtiCo4iQY28r3cOW0ueyqrOPJGyYyYlBKtEsSkSiKtI/gz+4+e19PuHteG9YjAdu4s4orp82lfE89D187ntFDdAtJkVgXadNQrpl9+o1hZn3M7FsB1SQBWb29gq/87QOq6xp46oaJjMvuG+2SRKQDiDQIbnD3sk8W3L0UuOFALzKzyWa22szWmtnt+3j+m2a2zMwWm9l7ZpYbeenSGsu3lPO1+z/AgGduPIFjMjU8VERCIg2CeGs2yYyZxQOJ+3tBeJt7gHOAXODyfXzRP+nux7j7ccCdwF0RVy4RW1xQxhV//5DkxASe/eYJ5AxUn4CI/FukQfAaoY7hM8zsDOCp8Lr9GQ+sdff17l4HTAcuaL6Bu+9utpgMeIT1SIQWbtrFVdPmktqzG0/fOJHD+2l0kIh8VqSdxT8EbgRuCi/PAqYd4DUZQEGz5UJgwt4bmdm3ge8ROsM4fV9vZGZTgakAWVlZEZYsc9bu5PpHFzCwd3eevGECg1N7RLskEemAIjojcPcmd7/X3S8N//zN3RvbogB3v8fdhxMKm5+0sM397p7n7nnp6elt8bFd3lsri7jm4fkM6dOTp6dOVAiISIsivY4gB/hvQm39n96l3N2H7edlW4AhzZYzw+taMh24N5J6ZP9eWbKV7z69mNzDevPIlPH0Sd5vd46IxLhI+wgeIvQl3QCcBjwKPH6A18wHcsxsqJklApcBM5pvEA6YT3wJ+DjCeqQFb60s4tbpixib1Ycnrp+gEBCRA4o0CHq4+1uAufsmd/85oS/uFrl7A3Az8DqwEnjG3VeY2S/N7PzwZjeb2QozW0yon+Dqg9oLAWBZYTk3P7mIow9L5aEp40jp3i3aJYlIJxBpZ3GtmcUBH5vZzYSaeA54txJ3nwnM3GvdHc0e39qKWmU/CkurufaR+fRNTuSBa/JITtKtJkQkMpGeEdwK9ARuAY4HrkJ/vXcYFTX1THloPjX1jTw8ZRwDUrof+EUiImEH/LMxfGHY19z9NqCS0H0JpINwd77/7FLW76zisWvH62IxEWm1A54RhIeJntwOtchBmPavDby2Yju3Tx7JiUf0j3Y5ItIJRdqQvMjMZgDPAlWfrHT3FwKpSiIyd30J//PaKiYfPYjrTxka7XJEpJOKNAi6AyV89spfBxQEUbK5pJqbn1pEVt+e/P4rx9JsKigRkVaJ9FaV6hfoQNYUVXDVtLnUNzZx31XHa5ioiBySSK8sfoh9TAjn7te2eUWyX0sKyrj6oXkkxsfx9NQTdHcxETlkkTYN/aPZ4+7ARcDWti9H9mfF1nKu+PuH9O2VyOPXTdBMoiLSJiJtGnq++bKZPQW8F0hFsk/VdQ1856lFJCcl8OyNJzIoVdcKiEjbONjLT3OAAW1ZiOzfL1/JZ8POKp64boJCQETaVKR9BBV8to9gO6Fpo6UdzFy2jenzC7hp0nBdKyAibS7SpiH1SEZJYWk1tz+/lNFD0vjeF4+Mdjki0gVFNNeQmV1kZqnNltPM7MLgyhKAPXWN3PjYQtzhz5cdR7f4SKeGEhGJXKTfLD9z9/JPFty9DPhZMCUJhOYQ+tELS8nftpu7Lz9OI4REJDCRBsG+ttM8xwF64L0NvLR4K7edNYLTRw6Mdjki0oVFGgQLzOwuMxse/rkLWBhkYbFszrqd/HbmSs49ZhDfmjQ82uWISBcXaRB8B6gDniZ0b+Ea4NtBFRXLyqrr+N7TS8jun8zvLx2tOYREJHCRjhqqAm4PuJaY5+78+MXl7Kys5aWrT9JdxkSkXUQ6amiWmaU1W+5jZq8HV1ZsenHRFl5dto3vnXUkozJSD/wCEZE2EGnTUP/wSCEA3L0UXVncpgp2VXPHyysYn92XG09Vv4CItJ9Ig6DJzLI+WTCzbPYxG6kcnLqGJm5+ahEG/PGro4mPU7+AiLSfSBuhfwy8Z2bvAgacAkwNrKoY89uZK1lSUMZ9V41lSN+e0S5HRGJMpJ3Fr5lZHqEv/0XAS8CeIAuLFa8s2crDczZy/clDmTxqcLTLEZEYFOmkc9cDtwKZwGJgIvABn711pbTSuuJKbn9+Kccf3ocfnjMy2uWISIyKtI/gVmAcsMndTwPGAGX7f4nsj7vzg+eWkpgQx/9eMUbzCIlI1ET67VPj7jUAZpbk7quAEcGV1fW9uGgLCzeV8qNzjmJwao9olyMiMSzSzuLC8HUELwGzzKwU2BRcWV1bRU09v525itFD0rj0+MxolyMiMS7SzuKLwg9/bmazgVTgtcCq6uLufvNjSqpqeeDqPOI0VFREoqzVcxi4+7tBFBIr1hRV8NCcjVw2bgijh6Qd+AUiIgFTD2U7cnd+8coKkhPjue0sdbGISMegIGhHr6/YzvtrS7jt7BH065UU7XJERAAFQbupqW/kV/9YychBKVwxPuvALxARaSea57id3PfuOraU7eGpGyaSoGsGRKQD0TdSOygsrebed9bxpWMHc8LwftEuR0TkMwINAjObbGarzWytmX3uxjZm9j0zyzezpWb2lpkdHmQ90fKbV1diBv917lHRLkVE5HMCCwIziwfuAc4BcoHLzSx3r80WAXnufizwHHBnUPVEy+zVO/jn8u185/QcMtJ0BbGIdDxBnhGMB9a6+3p3ryN0r+MLmm/g7rPdvTq8+CGhSe26jJr6Rn728gqGpydzwynDol2OiMg+BRkEGUBBs+XC8LqWXAf8c19PmNlUM1tgZguKi4vbsMRg3TN7LZt3VfOrC0eRmKDuGBHpmDrEt5OZXQXkAb/f1/Pufr+757l7Xnp6evsWd5DW7qjkvnfXcdGYDE4c3j/a5YiItCjI4aNbgCHNljPD6z7DzM4kdAe0L7h7bYD1tKtfvLKCHt3i1UEsIh1ekGcE84EcMxtqZonAZcCM5huY2Rjgb8D57r4jwFra1YfrS/jXxzu55Ywc0lN0BbGIdGyBBYG7NwA3A68DK4Fn3H2Fmf3SzM4Pb/Z7oBfwrJktNrMZLbxdp+Hu3DVrDQNSkrhqYpccDSsiXUygVxa7+0xg5l7r7mj2+MwgPz8a3l9bwrwNu/jF+UfTvVt8tMsRETmgDtFZ3FW4O3+ctZrDUrtz2fghB36BiEgHoCBoQ++sLmbR5jJuPj2HpASdDYhI56AgaCPuzp/eXENmnx66/aSIdCoKgjayuKCMJYXl3HjqMF08JiKdir6x2shjH24iOTGei8bqbEBEOhcFQRvYVVXHP5Zu4+KxmfRK0i0eRKRzURC0gWcWFFDX0MTXT9B1AyLS+SgIDlFjk/PE3E1MGNqXIwemRLscEZFWUxAcov9bU0zBrj06GxCRTktBcIge/WAj6SlJnH30oGiXIiJyUBQEh2B9cSXvrCnm8vFZdNMN6UWkk9K31yF48P0NdIuL4+uaXE5EOjEFwUEqrarjuYWFXDjmME01LSKdmoLgID05bzM19U1cd7LuRSwinZuC4CDUNjTy8JyNnJLTnxGDNGRURDo3BcFB+MeSbRRX1HLDKTobEJHOT0HQSu7OtPc2cOTAXpySo5vSi0jnpyBopQ/X72Lltt1ce9JQzCza5YiIHDIFQSs9MmcjaT27ceGYjGiXIiLSJhQErVBYWs0b+du5fHyW7kcsIl2GgqAVHvtwE2bGVbqATES6EAVBhPbUNTJ9XgFnHz2QjLQe0S5HRKTNKAgi9NLiLZTvqefqE7KjXYqISJtSEETA3Xn4/Y0cNbg344f2jXY5IiJtSkEQgZnLtrO6qIJrT8rWkFER6XIUBAdQU9/If/9zJSMHpXCxbkwvIl2QguAAHnp/I4Wle/jpebnEx+lsQES6HgXBfhRX1HLP7LWcedQATjpC00mISNekINiPu2atpqa+kf8696holyIiEhgFQQs2l1Tz9PwCvnFCNsPSe0W7HBGRwCgIWvBG/naaHKaclB3tUkREAqUgaMGbK4sYMTCFIX17RrsUEZFAKQj2oay6jvkbSzkzd0C0SxERCZyCYB/eWV1MY5Nz5lEDo12KiEjgAg0CM5tsZqvNbK2Z3b6P5081s4/MrMHMLg2yltZ4c2UR/XslMTozLdqliIgELrAgMLN44B7gHCAXuNzMcvfabDNwDfBkUHW0Vl1DE++uLuaMkQOI0wVkIhIDEgJ87/HAWndfD2Bm04ELgPxPNnD3jeHnmgKso1XmbdhFRW0DZ+aqWUhEYkOQTUMZQEGz5cLwulYzs6lmtsDMFhQXF7dJcS15c2URSQlxnKwriUUkRnSKzmJ3v9/d89w9Lz09PcjP4c2VRZx8RH96JOpWlCISG4IMgi3AkGbLmeF1HdaaokoKS/eoWUhEYkqQQTAfyDGzoWaWCFwGzAjw8w7Z26t2AHDaCF0/ICKxI7AgcPcG4GbgdWAl8Iy7rzCzX5rZ+QBmNs7MCoGvAH8zsxVB1ROJd1bv4KjBvRmU2j2aZYiItKsgRw3h7jOBmXutu6PZ4/mEmoyibndNPQs2lXLjqcOiXYqISLvqFJ3F7eH9j3fS2ORMUrOQiMQYBUHY7NU7SOmewNgsXU0sIrFFQUBo2Og7q4s5NSedhHj9JxGR2KJvPSB/2252VNQyaURw1yiIiHRUCgJCs40CfEFBICIxSEFAaNjoqIzeDEjRsFERiT0xHwTl1fUs3FTKpCM1WkhEYlPMB8En9yY+baSahUQkNsV0ELg7j36wiSMG9GJsVp9olyMiEhUxHQSLCspYtqWcq084HDPdhEZEYlNMB8GjczaSkpTAxWM7xCwXIiJREbNBUFxRy6vLtnHJ8ZkkJwU65ZKISIcWs0Hw1LzN1Dc63zjh8GiXIiISVTEZBPWNTTwxdxOnHpnOsPRe0S5HRCSqYi4Iahsa+c2rKynaXcvVOhsQEQn2fgQdzartu/mP6YtZtb2Cr088XHciExEhhoLg2QUF/PjF5fTu0Y0Hr8nj9JG6L7GICMRQEAxLT+b0kQP4zUWj6NcrKdrliIh0GDETBMcf3pfjv9432mWIiHQ4MddZLCIin6UgEBGJcQoCEZEYpyAQEYlxCgIRkRinIBARiXEKAhGRGKcgEBGJcebu0a6hVcysGNh0kC/vD+xsw3I6i1jc71jcZ4jN/Y7FfYbW7/fh7r7Pm7N3uiA4FGa2wN3zol1He4vF/Y7FfYbY3O9Y3Gdo2/1W05CISIxTEIiIxLhYC4L7o11AlMTifsfiPkNs7ncs7jO04X7HVB+BiIh8XqydEYiIyF4UBCIiMS5mgsDMJkV2ACwAAAU8SURBVJvZajNba2a3R7ueIJjZEDObbWb5ZrbCzG4Nr+9rZrPM7OPwv32iXWtbM7N4M1tkZv8ILw81s7nh4/20mSVGu8a2ZmZpZvacma0ys5VmdkKMHOvvhn+/l5vZU2bWvasdbzN70Mx2mNnyZuv2eWwt5M/hfV9qZmNb+3kxEQRmFg/cA5wD5AKXm1ludKsKRAPwn+6eC0wEvh3ez9uBt9w9B3grvNzV3AqsbLb8O+D/ufsRQClwXVSqCtbdwGvuPhIYTWj/u/SxNrMM4BYgz91HAfHAZXS94/0wMHmvdS0d23OAnPDPVODe1n5YTAQBMB5Y6+7r3b0OmA5cEOWa2py7b3P3j8KPKwh9MWQQ2tdHwps9AlwYnQqDYWaZwJeAaeFlA04Hngtv0hX3ORU4FXgAwN3r3L2MLn6swxKAHmaWAPQEttHFjre7/x+wa6/VLR3bC4BHPeRDIM3MBrfm82IlCDKAgmbLheF1XZaZZQNjgLnAQHffFn5qOzAwSmUF5U/AD4Cm8HI/oMzdG8LLXfF4DwWKgYfCTWLTzCyZLn6s3X0L8AdgM6EAKAcW0vWPN7R8bA/5+y1WgiCmmFkv4HngP9x9d/PnPDReuMuMGTaz84Ad7r4w2rW0swRgLHCvu48BqtirGairHWuAcLv4BYSC8DAgmc83oXR5bX1sYyUItgBDmi1nhtd1OWbWjVAIPOHuL4RXF31yqhj+d0e06gvAScD5ZraRUJPf6YTaztPCTQfQNY93IVDo7nPDy88RCoaufKwBzgQ2uHuxu9cDLxD6HejqxxtaPraH/P0WK0EwH8gJjyxIJNS5NCPKNbW5cNv4A8BKd7+r2VMzgKvDj68GXm7v2oLi7j9y90x3zyZ0XN929yuB2cCl4c261D4DuPt2oMDMRoRXnQHk04WPddhmYKKZ9Qz/vn+y3136eIe1dGxnAN8Ijx6aCJQ3a0KKjLvHxA9wLrAGWAf8ONr1BLSPJxM6XVwKLA7/nEuozfwt4GPgTaBvtGsNaP8nAf8IPx4GzAPWAs8CSdGuL4D9PQ5YED7eLwF9YuFYA78AVgHLgceApK52vIGnCPWB1BM6+7uupWMLGKFRkeuAZYRGVLXq8zTFhIhIjIuVpiEREWmBgkBEJMYpCEREYpyCQEQkxikIRERinIJAJMzMGs1scbOfNpuwzcyym88kKdKRJBx4E5GYscfdj4t2ESLtTWcEIgdgZhvN7E4zW2Zm88zsiPD6bDN7OzwH/FtmlhVeP9DMXjSzJeGfE8NvFW9mfw/Ppf+GmfUIb39L+B4SS81sepR2U2KYgkDk33rs1TT0tWbPlbv7McD/EprtFOAvwCPufizwBPDn8Po/A++6+2hC8/+sCK/PAe5x96OBMuCS8PrbgTHh9/lmUDsn0hJdWSwSZmaV7t5rH+s3Aqe7+/rwpH7b3b2fme0EBrt7fXj9Nnfvb2bFQKa71zZ7j2xgloduKoKZ/RDo5u6/NrPXgEpC00S85O6VAe+qyGfojEAkMt7C49aobfa4kX/30X2J0FwxY4H5zWbRFGkXCgKRyHyt2b8fhB/PITTjKcCVwL/Cj98CboJP76Wc2tKbmlkcMMTdZwM/BFKBz52ViARJf3mI/FsPM1vcbPk1d/9kCGkfM1tK6K/6y8PrvkPoDmHfJ3S3sCnh9bcC95vZdYT+8r+J0EyS+xIPPB4OCwP+7KFbToq0G/URiBxAuI8gz913RrsWkSCoaUhEJMbpjEBEJMbpjEBEJMYpCEREYpyCQEQkxikIRERinIJARCTG/X9JLL/GoFmmoAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def plot_graphs(history, string):\n",
        "  plt.plot(history.history[string])\n",
        "  plt.xlabel(\"Epochs\")\n",
        "  plt.ylabel(string)\n",
        "  plt.show()\n",
        "\n",
        "plot_graphs(history, 'accuracy')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ISLZZGlQlSxh"
      },
      "source": [
        "### Generate better lyrics!\n",
        "\n",
        "This time around, we should be able to get a more interesting output with less repetition."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "P96oVMk3lU7y",
        "outputId": "8d8f82f0-a955-4a85-f466-57979738d36e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "im feeling chills me and bobby and bobbys brother joe for a playing ground where we came to be thousand gritty darling heads heads long little fiddle with hours and unreal fine december my back together searching cause knowing movin tall movin on cinderella to go near the teacher he fire it up my heal god distant attraction me all black and the phone back past our world patience fault rather resist smiling door ha ha ha kissed mm dumb lot tear family resist broken stoppin cars fat cars temperature headyeah worlds citys misery lifetime proteccion game forgot taken choice disillusions twist scenes\n"
          ]
        }
      ],
      "source": [
        "seed_text = \"im feeling chills\"\n",
        "next_words = 100\n",
        "  \n",
        "for _ in range(next_words):\n",
        "\ttoken_list = tokenizer.texts_to_sequences([seed_text])[0]\n",
        "\ttoken_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')\n",
        "\tpredicted = np.argmax(model.predict(token_list), axis=-1)\n",
        "\toutput_word = \"\"\n",
        "\tfor word, index in tokenizer.word_index.items():\n",
        "\t\tif index == predicted:\n",
        "\t\t\toutput_word = word\n",
        "\t\t\tbreak\n",
        "\tseed_text += \" \" + output_word\n",
        "print(seed_text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "upgJKV8_oRU9"
      },
      "source": [
        "### Varying the Possible Outputs\n",
        "\n",
        "In running the above, you may notice that the same seed text will generate similar outputs. This is because the code is currently always choosing the top predicted class as the next word. What if you wanted more variance in the output? \n",
        "\n",
        "Switching from `model.predict_classes` to `model.predict_proba` will get us all of the class probabilities. We can combine this with `np.random.choice` to select a given predicted output based on a probability, thereby giving a bit more randomness to our outputs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "lZe9gaJeoGVP",
        "outputId": "af0ef444-4ad0-4874-ea6d-a8051d05abe6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "7\n"
          ]
        }
      ],
      "source": [
        "# Test the method with just the first word after the seed text\n",
        "seed_text = \"im feeling chills\"\n",
        "next_words = 100\n",
        "  \n",
        "token_list = tokenizer.texts_to_sequences([seed_text])[0]\n",
        "token_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')\n",
        "predicted_probs = model.predict(token_list)[0]\n",
        "predicted = np.random.choice([x for x in range(len(predicted_probs))], \n",
        "                             p=predicted_probs)\n",
        "# Running this cell multiple times should get you some variance in output\n",
        "print(predicted)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "ee7WKgRGrJy1",
        "outputId": "35d8aa31-f3e8-4cda-8a71-d75bf8e37a2e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "im feeling chills all alone then i laugh is so down the west but still the deep near find little hope soft and making another day thats rough out light needs fun tambourine girls tear family climbing memory forgot memory beg jungle musics depend neon crazy lay exciting up seventeen family monkey given fool fool shame on done to loved you go in girls to the terror melody left that night his diggin car glory broken of our back wild im told in all he town he did gone away on down there i throw a night yeah i believe yeah see you\n"
          ]
        }
      ],
      "source": [
        "# Use this process for the full output generation\n",
        "seed_text = \"im feeling chills\"\n",
        "next_words = 100\n",
        "  \n",
        "for _ in range(next_words):\n",
        "  token_list = tokenizer.texts_to_sequences([seed_text])[0]\n",
        "  token_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')\n",
        "  predicted_probs = model.predict(token_list)[0]\n",
        "  predicted = np.random.choice([x for x in range(len(predicted_probs))],\n",
        "                               p=predicted_probs)\n",
        "  output_word = \"\"\n",
        "  for word, index in tokenizer.word_index.items():\n",
        "    if index == predicted:\n",
        "      output_word = word\n",
        "      break\n",
        "  seed_text += \" \" + output_word\n",
        "print(seed_text)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "l10c04_nlp_optimizing_the_text_generation_model.ipynb",
      "toc_visible": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}