l10c03_nlp_constructing_text_generation_model.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "punL79CN7Ox6"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "cellView": "form",
        "id": "_ckMIh7O7s6D"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ph5eir3Pf-3z"
      },
      "source": [
        "# Constructing a Text Generation Model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S5Uhzt6vVIB2"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l10c03_nlp_constructing_text_generation_model.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l10c03_nlp_constructing_text_generation_model.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7GbGfr_oLCat"
      },
      "source": [
        "Using most of the techniques you've already learned, it's now possible to generate new text by predicting the next word that follows a given seed word. To practice this method, we'll use the [Kaggle Song Lyrics Dataset](https://www.kaggle.com/mousehead/songlyrics)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4aHK2CYygXom"
      },
      "source": [
        "## Import TensorFlow and related functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "2LmLTREBf5ng"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "# Other imports for processing data\n",
        "import string\n",
        "import numpy as np\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GmLTO_dpgge9"
      },
      "source": [
        "## Get the Dataset\n",
        "\n",
        "As noted above, we'll utilize the [Song Lyrics dataset](https://www.kaggle.com/mousehead/songlyrics) on Kaggle."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "4Bf5FVHfganK",
        "outputId": "180c0061-a823-4e73-bd65-856e4d1957d6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-12-20 21:39:20--  https://drive.google.com/uc?id=1LiJFZd41ofrWoBtW-pMYsfz1w8Ny0Bj8\n",
            "Resolving drive.google.com (drive.google.com)... 142.250.152.138, 142.250.152.101, 142.250.152.100, ...\n",
            "Connecting to drive.google.com (drive.google.com)|142.250.152.138|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Moved Temporarily\n",
            "Location: https://doc-04-ak-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/350c8g8qgs67729qsr71u2sn1t76e07g/1640036325000/11118900490791463723/*/1LiJFZd41ofrWoBtW-pMYsfz1w8Ny0Bj8 [following]\n",
            "Warning: wildcards not supported in HTTP.\n",
            "--2021-12-20 21:39:22--  https://doc-04-ak-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/350c8g8qgs67729qsr71u2sn1t76e07g/1640036325000/11118900490791463723/*/1LiJFZd41ofrWoBtW-pMYsfz1w8Ny0Bj8\n",
            "Resolving doc-04-ak-docs.googleusercontent.com (doc-04-ak-docs.googleusercontent.com)... 142.251.6.132, 2607:f8b0:4001:c5a::84\n",
            "Connecting to doc-04-ak-docs.googleusercontent.com (doc-04-ak-docs.googleusercontent.com)|142.251.6.132|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 72436445 (69M) [text/csv]\n",
            "Saving to: ‘/tmp/songdata.csv’\n",
            "\n",
            "/tmp/songdata.csv   100%[===================>]  69.08M  82.7MB/s    in 0.8s    \n",
            "\n",
            "2021-12-20 21:39:23 (82.7 MB/s) - ‘/tmp/songdata.csv’ saved [72436445/72436445]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget --no-check-certificate \\\n",
        "    https://drive.google.com/uc?id=1LiJFZd41ofrWoBtW-pMYsfz1w8Ny0Bj8 \\\n",
        "    -O /tmp/songdata.csv"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gu1BTzMIS1oy"
      },
      "source": [
        "## **First 10 Songs**\n",
        "\n",
        "Let's first look at just 10 songs from the dataset, and see how things perform."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fmb9rGaAUDO-"
      },
      "source": [
        "### Preprocessing\n",
        "\n",
        "Let's perform some basic preprocessing to get rid of punctuation and make everything lowercase. We'll then split the lyrics up by line and tokenize the lyrics."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "2AVAvyF_Vuh5"
      },
      "outputs": [],
      "source": [
        "def tokenize_corpus(corpus, num_words=-1):\n",
        "  # Fit a Tokenizer on the corpus\n",
        "  if num_words > -1:\n",
        "    tokenizer = Tokenizer(num_words=num_words)\n",
        "  else:\n",
        "    tokenizer = Tokenizer()\n",
        "  tokenizer.fit_on_texts(corpus)\n",
        "  return tokenizer\n",
        "\n",
        "def create_lyrics_corpus(dataset, field):\n",
        "  # Remove all other punctuation\n",
        "  dataset[field] = dataset[field].str.replace('[{}]'.format(string.punctuation), '')\n",
        "  # Make it lowercase\n",
        "  dataset[field] = dataset[field].str.lower()\n",
        "  # Make it one long string to split by line\n",
        "  lyrics = dataset[field].str.cat()\n",
        "  corpus = lyrics.split('\\n')\n",
        "  # Remove any trailing whitespace\n",
        "  for l in range(len(corpus)):\n",
        "    corpus[l] = corpus[l].rstrip()\n",
        "  # Remove any empty lines\n",
        "  corpus = [l for l in corpus if l != '']\n",
        "\n",
        "  return corpus"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "apcEXp7WhVBs",
        "outputId": "27ffa9cb-0072-49ae-cd31-0157e7f6f1ed",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'you': 1, 'i': 2, 'and': 3, 'a': 4, 'me': 5, 'the': 6, 'is': 7, 'my': 8, 'to': 9, 'ma': 10, 'it': 11, 'of': 12, 'im': 13, 'your': 14, 'love': 15, 'so': 16, 'as': 17, 'that': 18, 'in': 19, 'andante': 20, 'boomaboomerang': 21, 'make': 22, 'on': 23, 'oh': 24, 'for': 25, 'but': 26, 'new': 27, 'bang': 28, 'its': 29, 'be': 30, 'like': 31, 'know': 32, 'now': 33, 'how': 34, 'could': 35, 'youre': 36, 'sing': 37, 'never': 38, 'no': 39, 'chiquitita': 40, 'can': 41, 'we': 42, 'song': 43, 'had': 44, 'good': 45, 'youll': 46, 'she': 47, 'just': 48, 'girl': 49, 'again': 50, 'will': 51, 'take': 52, 'please': 53, 'let': 54, 'am': 55, 'eyes': 56, 'was': 57, 'always': 58, 'cassandra': 59, 'blue': 60, 'time': 61, 'dont': 62, 'were': 63, 'return': 64, 'once': 65, 'then': 66, 'sorry': 67, 'cryin': 68, 'over': 69, 'feel': 70, 'ever': 71, 'believe': 72, 'what': 73, 'do': 74, 'go': 75, 'all': 76, 'out': 77, 'think': 78, 'every': 79, 'leave': 80, 'look': 81, 'at': 82, 'way': 83, 'one': 84, 'music': 85, 'down': 86, 'our': 87, 'give': 88, 'learn': 89, 'more': 90, 'us': 91, 'would': 92, 'there': 93, 'before': 94, 'when': 95, 'with': 96, 'feeling': 97, 'play': 98, 'cause': 99, 'away': 100, 'here': 101, 'have': 102, 'yes': 103, 'baby': 104, 'get': 105, 'didnt': 106, 'see': 107, 'did': 108, 'closed': 109, 'realized': 110, 'crazy': 111, 'world': 112, 'lord': 113, 'shes': 114, 'kind': 115, 'without': 116, 'if': 117, 'touch': 118, 'strong': 119, 'making': 120, 'such': 121, 'found': 122, 'true': 123, 'stay': 124, 'together': 125, 'thought': 126, 'come': 127, 'they': 128, 'sweet': 129, 'tender': 130, 'sender': 131, 'tune': 132, 'humdehumhum': 133, 'gonna': 134, 'last': 135, 'leaving': 136, 'sleep': 137, 'only': 138, 'saw': 139, 'tell': 140, 'hes': 141, 'her': 142, 'sound': 143, 'tread': 144, 'lightly': 145, 'ground': 146, 'ill': 147, 'show': 148, 'life': 149, 'too': 150, 'used': 151, 'darling': 152, 'meant': 153, 'break': 154, 'end': 155, 'yourself': 156, 'little': 157, 'dumbedumdum': 158, 'bedumbedumdum': 159, 'youve': 160, 'dumbbedumbdumb': 161, 'bedumbbedumbdumb': 162, 'by': 163, 'theyre': 164, 'alone': 165, 'misunderstood': 166, 'day': 167, 'dawning': 168, 'some': 169, 'wanted': 170, 'none': 171, 'listen': 172, 'words': 173, 'warning': 174, 'darkest': 175, 'nights': 176, 'nobody': 177, 'knew': 178, 'fight': 179, 'caught': 180, 'really': 181, 'power': 182, 'dreams': 183, 'weave': 184, 'until': 185, 'final': 186, 'hour': 187, 'morning': 188, 'ship': 189, 'gone': 190, 'grieving': 191, 'still': 192, 'pain': 193, 'cry': 194, 'sun': 195, 'try': 196, 'face': 197, 'something': 198, 'sees': 199, 'makes': 200, 'fine': 201, 'who': 202, 'mine': 203, 'leaves': 204, 'walk': 205, 'hand': 206, 'well': 207, 'about': 208, 'things': 209, 'slow': 210, 'theres': 211, 'talk': 212, 'why': 213, 'up': 214, 'lousy': 215, 'packing': 216, 'ive': 217, 'gotta': 218, 'near': 219, 'keeping': 220, 'intention': 221, 'growing': 222, 'taking': 223, 'dimension': 224, 'even': 225, 'better': 226, 'thank': 227, 'god': 228, 'not': 229, 'somebody': 230, 'happy': 231, 'question': 232, 'smile': 233, 'mean': 234, 'much': 235, 'kisses': 236, 'around': 237, 'anywhere': 238, 'advice': 239, 'care': 240, 'use': 241, 'selfish': 242, 'tool': 243, 'fool': 244, 'showing': 245, 'boomerang': 246, 'throwing': 247, 'warm': 248, 'kiss': 249, 'surrender': 250, 'giving': 251, 'been': 252, 'door': 253, 'burning': 254, 'bridges': 255, 'being': 256, 'moving': 257, 'though': 258, 'behind': 259, 'are': 260, 'must': 261, 'sure': 262, 'stood': 263, 'hope': 264, 'this': 265, 'deny': 266, 'sad': 267, 'quiet': 268, 'truth': 269, 'heartaches': 270, 'scars': 271, 'dancing': 272, 'sky': 273, 'shining': 274, 'above': 275, 'hear': 276, 'came': 277, 'couldnt': 278, 'everything': 279, 'back': 280, 'long': 281, 'waitin': 282, 'cold': 283, 'chills': 284, 'bone': 285, 'youd': 286, 'wonderful': 287, 'means': 288, 'special': 289, 'smiles': 290, 'lucky': 291, 'fellow': 292, 'park': 293, 'holds': 294, 'squeezes': 295, 'walking': 296, 'hours': 297, 'talking': 298, 'plan': 299, 'easy': 300, 'gently': 301, 'summer': 302, 'evening': 303, 'breeze': 304, 'grow': 305, 'fingers': 306, 'soft': 307, 'light': 308, 'body': 309, 'velvet': 310, 'night': 311, 'soul': 312, 'slowly': 313, 'shimmer': 314, 'thousand': 315, 'butterflies': 316, 'float': 317, 'put': 318, 'rotten': 319, 'boy': 320, 'tough': 321, 'stuff': 322, 'saying': 323, 'need': 324, 'anymore': 325, 'enough': 326, 'standing': 327, 'creep': 328, 'felt': 329, 'cheap': 330, 'notion': 331, 'deep': 332, 'dumb': 333, 'mistake': 334, 'entitled': 335, 'another': 336, 'beg': 337, 'forgive': 338, 'an': 339, 'feels': 340, 'hoot': 341, 'holler': 342, 'mad': 343, 'under': 344, 'heel': 345, 'holy': 346, 'christ': 347, 'deal': 348, 'sick': 349, 'tired': 350, 'tedious': 351, 'ways': 352, 'aint': 353, 'walkin': 354, 'cutting': 355, 'tie': 356, 'wanna': 357, 'into': 358, 'eye': 359, 'myself': 360, 'counting': 361, 'pride': 362, 'unright': 363, 'neighbours': 364, 'ride': 365, 'burying': 366, 'past': 367, 'peace': 368, 'free': 369, 'sucker': 370, 'street': 371, 'singing': 372, 'shouting': 373, 'staying': 374, 'alive': 375, 'city': 376, 'dead': 377, 'hiding': 378, 'their': 379, 'shame': 380, 'hollow': 381, 'laughter': 382, 'while': 383, 'crying': 384, 'bed': 385, 'pity': 386, 'believed': 387, 'lost': 388, 'from': 389, 'start': 390, 'suffer': 391, 'sell': 392, 'secrets': 393, 'bargain': 394, 'playing': 395, 'smart': 396, 'aching': 397, 'hearts': 398, 'sailing': 399, 'father': 400, 'sister': 401, 'reason': 402, 'linger': 403, 'deeply': 404, 'future': 405, 'casting': 406, 'shadow': 407, 'else': 408, 'fate': 409, 'bags': 410, 'thorough': 411, 'knowing': 412, 'late': 413, 'wait': 414, 'watched': 415, 'harbor': 416, 'sunrise': 417, 'sails': 418, 'almost': 419, 'slack': 420, 'cool': 421, 'rain': 422, 'deck': 423, 'tiny': 424, 'figure': 425, 'rigid': 426, 'restrained': 427, 'filled': 428, 'whats': 429, 'wrong': 430, 'enchained': 431, 'own': 432, 'sorrow': 433, 'tomorrow': 434, 'hate': 435, 'shoulder': 436, 'best': 437, 'friend': 438, 'rely': 439, 'broken': 440, 'feather': 441, 'patch': 442, 'walls': 443, 'tumbling': 444, 'loves': 445, 'blown': 446, 'candle': 447, 'seems': 448, 'hard': 449, 'handle': 450, 'id': 451, 'thinking': 452, 'went': 453, 'house': 454, 'hardly': 455, 'guy': 456, 'closing': 457, 'front': 458, 'emptiness': 459, 'he': 460, 'disapeared': 461, 'his': 462, 'car': 463, 'stunned': 464, 'dreamed': 465, 'lifes': 466, 'part': 467, 'move': 468, 'feet': 469, 'pavement': 470, 'acted': 471, 'told': 472, 'lies': 473, 'meet': 474, 'other': 475, 'guys': 476, 'stupid': 477, 'blind': 478, 'smiled': 479, 'took': 480, 'said': 481, 'may': 482, 'couple': 483, 'men': 484, 'them': 485, 'brother': 486, 'joe': 487, 'seeing': 488, 'lot': 489, 'him': 490, 'nice': 491, 'sitting': 492, 'sittin': 493, 'memories': 494}\n",
            "495\n"
          ]
        }
      ],
      "source": [
        "# Read the dataset from csv - just first 10 songs for now\n",
        "dataset = pd.read_csv('/tmp/songdata.csv', dtype=str)[:10]\n",
        "# Create the corpus using the 'text' column containing lyrics\n",
        "corpus = create_lyrics_corpus(dataset, 'text')\n",
        "# Tokenize the corpus\n",
        "tokenizer = tokenize_corpus(corpus)\n",
        "\n",
        "total_words = len(tokenizer.word_index) + 1\n",
        "\n",
        "print(tokenizer.word_index)\n",
        "print(total_words)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v9x68iN_X6FK"
      },
      "source": [
        "### Create Sequences and Labels\n",
        "\n",
        "After preprocessing, we next need to create sequences and labels. Creating the sequences themselves is similar to before with `texts_to_sequences`, but also including the use of [N-Grams](https://towardsdatascience.com/introduction-to-language-models-n-gram-e323081503d9); creating the labels will now utilize those sequences as well as utilize one-hot encoding over all potential output words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "QmlTsUqfikVO"
      },
      "outputs": [],
      "source": [
        "sequences = []\n",
        "for line in corpus:\n",
        "\ttoken_list = tokenizer.texts_to_sequences([line])[0]\n",
        "\tfor i in range(1, len(token_list)):\n",
        "\t\tn_gram_sequence = token_list[:i+1]\n",
        "\t\tsequences.append(n_gram_sequence)\n",
        "\n",
        "# Pad sequences for equal input length \n",
        "max_sequence_len = max([len(seq) for seq in sequences])\n",
        "sequences = np.array(pad_sequences(sequences, maxlen=max_sequence_len, padding='pre'))\n",
        "\n",
        "# Split sequences between the \"input\" sequence and \"output\" predicted word\n",
        "input_sequences, labels = sequences[:,:-1], sequences[:,-1]\n",
        "# One-hot encode the labels\n",
        "one_hot_labels = tf.keras.utils.to_categorical(labels, num_classes=total_words)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "Zsmu3aEId49i",
        "outputId": "dffc9867-9e34-445f-9196-af93dca0c404",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "32\n",
            "97\n",
            "[  0   0   0   0   0   0   0   0   0   0   0   0   0  81  82 142 197  29\n",
            "   4]\n",
            "[  0   0   0   0   0   0   0   0   0   0   0   0  81  82 142 197  29   4\n",
            " 287]\n",
            "[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
            "[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
            " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
          ]
        }
      ],
      "source": [
        "# Check out how some of our data is being stored\n",
        "# The Tokenizer has just a single index per word\n",
        "print(tokenizer.word_index['know'])\n",
        "print(tokenizer.word_index['feeling'])\n",
        "# Input sequences will have multiple indexes\n",
        "print(input_sequences[5])\n",
        "print(input_sequences[6])\n",
        "# And the one hot labels will be as long as the full spread of tokenized words\n",
        "print(one_hot_labels[5])\n",
        "print(one_hot_labels[6])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-1TAJMlmfO8r"
      },
      "source": [
        "### Train a Text Generation Model\n",
        "\n",
        "Building an RNN to train our text generation model will be very similar to the sentiment models you've built previously. The only real change necessary is to make sure to use Categorical instead of Binary Cross Entropy as the loss function - we could use Binary before since the sentiment was only 0 or 1, but now there are hundreds of categories.\n",
        "\n",
        "From there, we should also consider using *more* epochs than before, as text generation can take a little longer to converge than sentiment analysis, *and* we aren't working with all that much data yet. I'll set it at 200 epochs here since we're only use part of the dataset, and training will tail off quite a bit over that many epochs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "G1YXuxIqfygN",
        "outputId": "954a2118-2a38-4b33-ab74-1b614759cc6b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/200\n",
            "62/62 [==============================] - 12s 17ms/step - loss: 6.0069 - accuracy: 0.0277\n",
            "Epoch 2/200\n",
            "62/62 [==============================] - 1s 18ms/step - loss: 5.4427 - accuracy: 0.0348\n",
            "Epoch 3/200\n",
            "62/62 [==============================] - 1s 16ms/step - loss: 5.3701 - accuracy: 0.0394\n",
            "Epoch 4/200\n",
            "62/62 [==============================] - 1s 15ms/step - loss: 5.3139 - accuracy: 0.0399\n",
            "Epoch 5/200\n",
            "62/62 [==============================] - 1s 17ms/step - loss: 5.2373 - accuracy: 0.0444\n",
            "Epoch 6/200\n",
            "62/62 [==============================] - 1s 17ms/step - loss: 5.1556 - accuracy: 0.0585\n",
            "Epoch 7/200\n",
            "62/62 [==============================] - 1s 17ms/step - loss: 5.0699 - accuracy: 0.0681\n",
            "Epoch 8/200\n",
            "62/62 [==============================] - 1s 18ms/step - loss: 4.9842 - accuracy: 0.0646\n",
            "Epoch 9/200\n",
            "62/62 [==============================] - 1s 14ms/step - loss: 4.8851 - accuracy: 0.0721\n",
            "Epoch 10/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 4.7851 - accuracy: 0.0838\n",
            "Epoch 11/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 4.6842 - accuracy: 0.0903\n",
            "Epoch 12/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 4.5817 - accuracy: 0.1236\n",
            "Epoch 13/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 4.4781 - accuracy: 0.1236\n",
            "Epoch 14/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 4.3736 - accuracy: 0.1569\n",
            "Epoch 15/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 4.2684 - accuracy: 0.1544\n",
            "Epoch 16/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 4.1550 - accuracy: 0.1771\n",
            "Epoch 17/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 4.0478 - accuracy: 0.1998\n",
            "Epoch 18/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 3.9367 - accuracy: 0.2154\n",
            "Epoch 19/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 3.8384 - accuracy: 0.2437\n",
            "Epoch 20/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 3.7279 - accuracy: 0.2684\n",
            "Epoch 21/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 3.6384 - accuracy: 0.2931\n",
            "Epoch 22/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 3.5488 - accuracy: 0.3098\n",
            "Epoch 23/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 3.4759 - accuracy: 0.3163\n",
            "Epoch 24/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 3.3953 - accuracy: 0.3350\n",
            "Epoch 25/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 3.2903 - accuracy: 0.3698\n",
            "Epoch 26/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 3.2201 - accuracy: 0.3744\n",
            "Epoch 27/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 3.1494 - accuracy: 0.3749\n",
            "Epoch 28/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 3.0714 - accuracy: 0.3900\n",
            "Epoch 29/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 3.0011 - accuracy: 0.4026\n",
            "Epoch 30/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 2.9409 - accuracy: 0.4072\n",
            "Epoch 31/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 2.8790 - accuracy: 0.4183\n",
            "Epoch 32/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 2.8132 - accuracy: 0.4390\n",
            "Epoch 33/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 2.7552 - accuracy: 0.4445\n",
            "Epoch 34/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 2.7000 - accuracy: 0.4531\n",
            "Epoch 35/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 2.6248 - accuracy: 0.4707\n",
            "Epoch 36/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 2.5728 - accuracy: 0.4758\n",
            "Epoch 37/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 2.5153 - accuracy: 0.4919\n",
            "Epoch 38/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 2.4678 - accuracy: 0.4960\n",
            "Epoch 39/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 2.4100 - accuracy: 0.5106\n",
            "Epoch 40/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 2.3645 - accuracy: 0.5182\n",
            "Epoch 41/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 2.3197 - accuracy: 0.5272\n",
            "Epoch 42/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 2.2751 - accuracy: 0.5288\n",
            "Epoch 43/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 2.2337 - accuracy: 0.5464\n",
            "Epoch 44/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 2.1922 - accuracy: 0.5484\n",
            "Epoch 45/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 2.1425 - accuracy: 0.5666\n",
            "Epoch 46/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 2.0889 - accuracy: 0.5827\n",
            "Epoch 47/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 2.0564 - accuracy: 0.5838\n",
            "Epoch 48/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 2.0183 - accuracy: 0.5954\n",
            "Epoch 49/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 2.0169 - accuracy: 0.5954\n",
            "Epoch 50/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 2.0139 - accuracy: 0.5918\n",
            "Epoch 51/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.9695 - accuracy: 0.6100\n",
            "Epoch 52/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.9094 - accuracy: 0.6216\n",
            "Epoch 53/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.8590 - accuracy: 0.6377\n",
            "Epoch 54/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.8123 - accuracy: 0.6387\n",
            "Epoch 55/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.7993 - accuracy: 0.6473\n",
            "Epoch 56/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.7676 - accuracy: 0.6504\n",
            "Epoch 57/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.7320 - accuracy: 0.6509\n",
            "Epoch 58/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.7074 - accuracy: 0.6574\n",
            "Epoch 59/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.6746 - accuracy: 0.6645\n",
            "Epoch 60/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.6573 - accuracy: 0.6665\n",
            "Epoch 61/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.6074 - accuracy: 0.6791\n",
            "Epoch 62/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.5640 - accuracy: 0.6897\n",
            "Epoch 63/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.5710 - accuracy: 0.6872\n",
            "Epoch 64/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.5448 - accuracy: 0.6917\n",
            "Epoch 65/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.6552 - accuracy: 0.6534\n",
            "Epoch 66/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.5073 - accuracy: 0.7003\n",
            "Epoch 67/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.4693 - accuracy: 0.7104\n",
            "Epoch 68/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.4288 - accuracy: 0.7220\n",
            "Epoch 69/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.3943 - accuracy: 0.7306\n",
            "Epoch 70/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.3712 - accuracy: 0.7341\n",
            "Epoch 71/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.3996 - accuracy: 0.7220\n",
            "Epoch 72/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.3569 - accuracy: 0.7331\n",
            "Epoch 73/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.4371 - accuracy: 0.7059\n",
            "Epoch 74/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.3332 - accuracy: 0.7301\n",
            "Epoch 75/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.2909 - accuracy: 0.7508\n",
            "Epoch 76/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.2508 - accuracy: 0.7598\n",
            "Epoch 77/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.2404 - accuracy: 0.7573\n",
            "Epoch 78/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.2107 - accuracy: 0.7644\n",
            "Epoch 79/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 1.1835 - accuracy: 0.7740\n",
            "Epoch 80/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.1614 - accuracy: 0.7760\n",
            "Epoch 81/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.1404 - accuracy: 0.7810\n",
            "Epoch 82/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.1178 - accuracy: 0.7851\n",
            "Epoch 83/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.0988 - accuracy: 0.7866\n",
            "Epoch 84/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.0782 - accuracy: 0.7911\n",
            "Epoch 85/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.0588 - accuracy: 0.8007\n",
            "Epoch 86/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.0523 - accuracy: 0.8002\n",
            "Epoch 87/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.0371 - accuracy: 0.8007\n",
            "Epoch 88/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.0164 - accuracy: 0.8047\n",
            "Epoch 89/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.0166 - accuracy: 0.8022\n",
            "Epoch 90/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.0394 - accuracy: 0.7906\n",
            "Epoch 91/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.0228 - accuracy: 0.8012\n",
            "Epoch 92/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.9826 - accuracy: 0.8108\n",
            "Epoch 93/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.9670 - accuracy: 0.8138\n",
            "Epoch 94/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.0085 - accuracy: 0.8012\n",
            "Epoch 95/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 1.0726 - accuracy: 0.7805\n",
            "Epoch 96/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.9946 - accuracy: 0.8047\n",
            "Epoch 97/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.9615 - accuracy: 0.8113\n",
            "Epoch 98/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.9284 - accuracy: 0.8158\n",
            "Epoch 99/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.9066 - accuracy: 0.8274\n",
            "Epoch 100/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.8892 - accuracy: 0.8330\n",
            "Epoch 101/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.8786 - accuracy: 0.8325\n",
            "Epoch 102/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.8563 - accuracy: 0.8345\n",
            "Epoch 103/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.8424 - accuracy: 0.8355\n",
            "Epoch 104/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.8446 - accuracy: 0.8340\n",
            "Epoch 105/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.8386 - accuracy: 0.8335\n",
            "Epoch 106/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.8351 - accuracy: 0.8391\n",
            "Epoch 107/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.8077 - accuracy: 0.8426\n",
            "Epoch 108/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.7881 - accuracy: 0.8466\n",
            "Epoch 109/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.7765 - accuracy: 0.8471\n",
            "Epoch 110/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.7652 - accuracy: 0.8507\n",
            "Epoch 111/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.7593 - accuracy: 0.8486\n",
            "Epoch 112/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.9090 - accuracy: 0.8098\n",
            "Epoch 113/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.8758 - accuracy: 0.8158\n",
            "Epoch 114/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.8152 - accuracy: 0.8380\n",
            "Epoch 115/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.8010 - accuracy: 0.8345\n",
            "Epoch 116/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.7798 - accuracy: 0.8426\n",
            "Epoch 117/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.7605 - accuracy: 0.8436\n",
            "Epoch 118/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.7485 - accuracy: 0.8451\n",
            "Epoch 119/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.7331 - accuracy: 0.8522\n",
            "Epoch 120/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.7181 - accuracy: 0.8481\n",
            "Epoch 121/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.7062 - accuracy: 0.8552\n",
            "Epoch 122/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.6938 - accuracy: 0.8557\n",
            "Epoch 123/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.6772 - accuracy: 0.8653\n",
            "Epoch 124/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.6671 - accuracy: 0.8648\n",
            "Epoch 125/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.6579 - accuracy: 0.8673\n",
            "Epoch 126/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.6496 - accuracy: 0.8668\n",
            "Epoch 127/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.6394 - accuracy: 0.8673\n",
            "Epoch 128/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.6322 - accuracy: 0.8673\n",
            "Epoch 129/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.6264 - accuracy: 0.8698\n",
            "Epoch 130/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.6294 - accuracy: 0.8653\n",
            "Epoch 131/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.6267 - accuracy: 0.8663\n",
            "Epoch 132/200\n",
            "62/62 [==============================] - 1s 17ms/step - loss: 0.6547 - accuracy: 0.8562\n",
            "Epoch 133/200\n",
            "62/62 [==============================] - 1s 18ms/step - loss: 0.6353 - accuracy: 0.8648\n",
            "Epoch 134/200\n",
            "62/62 [==============================] - 1s 17ms/step - loss: 0.6168 - accuracy: 0.8663\n",
            "Epoch 135/200\n",
            "62/62 [==============================] - 1s 18ms/step - loss: 0.6045 - accuracy: 0.8638\n",
            "Epoch 136/200\n",
            "62/62 [==============================] - 1s 17ms/step - loss: 0.5886 - accuracy: 0.8698\n",
            "Epoch 137/200\n",
            "62/62 [==============================] - 1s 17ms/step - loss: 0.5807 - accuracy: 0.8729\n",
            "Epoch 138/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5770 - accuracy: 0.8729\n",
            "Epoch 139/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5661 - accuracy: 0.8779\n",
            "Epoch 140/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5574 - accuracy: 0.8784\n",
            "Epoch 141/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5481 - accuracy: 0.8794\n",
            "Epoch 142/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5672 - accuracy: 0.8713\n",
            "Epoch 143/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5500 - accuracy: 0.8789\n",
            "Epoch 144/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5404 - accuracy: 0.8789\n",
            "Epoch 145/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5302 - accuracy: 0.8860\n",
            "Epoch 146/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.5228 - accuracy: 0.8829\n",
            "Epoch 147/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5176 - accuracy: 0.8865\n",
            "Epoch 148/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5141 - accuracy: 0.8824\n",
            "Epoch 149/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5077 - accuracy: 0.8845\n",
            "Epoch 150/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5024 - accuracy: 0.8880\n",
            "Epoch 151/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.5029 - accuracy: 0.8895\n",
            "Epoch 152/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.4957 - accuracy: 0.8865\n",
            "Epoch 153/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4905 - accuracy: 0.8900\n",
            "Epoch 154/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4854 - accuracy: 0.8885\n",
            "Epoch 155/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.4828 - accuracy: 0.8880\n",
            "Epoch 156/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4799 - accuracy: 0.8910\n",
            "Epoch 157/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4932 - accuracy: 0.8835\n",
            "Epoch 158/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4869 - accuracy: 0.8885\n",
            "Epoch 159/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4798 - accuracy: 0.8900\n",
            "Epoch 160/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.4724 - accuracy: 0.8900\n",
            "Epoch 161/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4608 - accuracy: 0.8935\n",
            "Epoch 162/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4609 - accuracy: 0.8920\n",
            "Epoch 163/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4571 - accuracy: 0.8935\n",
            "Epoch 164/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.4548 - accuracy: 0.8910\n",
            "Epoch 165/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4495 - accuracy: 0.8930\n",
            "Epoch 166/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4504 - accuracy: 0.8880\n",
            "Epoch 167/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4396 - accuracy: 0.8986\n",
            "Epoch 168/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4325 - accuracy: 0.8940\n",
            "Epoch 169/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4384 - accuracy: 0.8930\n",
            "Epoch 170/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4307 - accuracy: 0.8966\n",
            "Epoch 171/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4223 - accuracy: 0.8946\n",
            "Epoch 172/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4435 - accuracy: 0.8875\n",
            "Epoch 173/200\n",
            "62/62 [==============================] - 1s 12ms/step - loss: 0.4444 - accuracy: 0.8890\n",
            "Epoch 174/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4730 - accuracy: 0.8850\n",
            "Epoch 175/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4536 - accuracy: 0.8865\n",
            "Epoch 176/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4647 - accuracy: 0.8799\n",
            "Epoch 177/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4644 - accuracy: 0.8840\n",
            "Epoch 178/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4538 - accuracy: 0.8855\n",
            "Epoch 179/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4292 - accuracy: 0.8925\n",
            "Epoch 180/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4137 - accuracy: 0.8946\n",
            "Epoch 181/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4005 - accuracy: 0.8996\n",
            "Epoch 182/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3973 - accuracy: 0.8976\n",
            "Epoch 183/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3918 - accuracy: 0.8971\n",
            "Epoch 184/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3874 - accuracy: 0.8971\n",
            "Epoch 185/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3903 - accuracy: 0.8991\n",
            "Epoch 186/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3911 - accuracy: 0.8986\n",
            "Epoch 187/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3913 - accuracy: 0.8981\n",
            "Epoch 188/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4034 - accuracy: 0.8930\n",
            "Epoch 189/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3812 - accuracy: 0.9001\n",
            "Epoch 190/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3741 - accuracy: 0.9016\n",
            "Epoch 191/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3705 - accuracy: 0.9031\n",
            "Epoch 192/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3660 - accuracy: 0.8981\n",
            "Epoch 193/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3609 - accuracy: 0.9011\n",
            "Epoch 194/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3587 - accuracy: 0.9011\n",
            "Epoch 195/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3601 - accuracy: 0.9046\n",
            "Epoch 196/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3621 - accuracy: 0.8951\n",
            "Epoch 197/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3534 - accuracy: 0.9057\n",
            "Epoch 198/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3512 - accuracy: 0.9036\n",
            "Epoch 199/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.4253 - accuracy: 0.8809\n",
            "Epoch 200/200\n",
            "62/62 [==============================] - 1s 13ms/step - loss: 0.3919 - accuracy: 0.8905\n"
          ]
        }
      ],
      "source": [
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional\n",
        "\n",
        "model = Sequential()\n",
        "model.add(Embedding(total_words, 64, input_length=max_sequence_len-1))\n",
        "model.add(Bidirectional(LSTM(20)))\n",
        "model.add(Dense(total_words, activation='softmax'))\n",
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "history = model.fit(input_sequences, one_hot_labels, epochs=200, verbose=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXVFpoREhV6Y"
      },
      "source": [
        "### View the Training Graph"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "aeSNfS7uhch0",
        "outputId": "c99babef-ae16-4003-bc50-8a7dec503217",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhU5d3/8fc3ewIhLIEQdhAEIpsYUCquaKuouIta61KrtlZrn9qntY/9WZ/aRW0fbW1dqta1WkXrghZxV1wQCLKFfYckEBLISvaZ+/fHDDRAQgbImUkyn9d15crMmTOT75yZzGfu+5xz3+acQ0REoldMpAsQEZHIUhCIiEQ5BYGISJRTEIiIRDkFgYhIlIuLdAGHKj093Q0aNCjSZYiItCsLFy4sds71bOq2dhcEgwYNIicnJ9JliIi0K2a2ubnb1DUkIhLlFAQiIlFOQSAiEuUUBCIiUU5BICIS5RQEIiJRTkEgIhLlFAQiIm1AQWk1O8prAFi0pYQX5m0mv7Q6LH+73Z1QJiISaRU19XRKiCMmxvZZ7vM7YgzM9l2+sXg3n67ewcItpSzNK+XY/l35f+dm8eX6nWwq3s2WXVW8viifXqmJvHjDCVz/bA67dtcRY/CP7x3PN45K9/T5KAhEpMN6+ouNvLWkgF+ddwxj+3eltsHH7NztpCXHc8KQHiTFxx7S4zX4/Pzpg7U8/Mk60jsnkj2wG906JRAfY2wtqeaLdcX4naNrSgLdUxK4NLsfWZlduO6ZBdQ2+MlMS2JkZhfeXrqNmUsK8AfnBUuIjWHauD7MXFzAuX/5nJp6H09enc1PZizmta/zFQQiEp3qfX4Ky2vo1y3lgNvySqp45JP1FFXUMjwjlfVFlcTGGGdmZbC2sJLEuBhOG9GL381aic/vuOjRLxnVpwvFlXV7u1t6dErg+euPJ6tPlxZrqaip54H31/De8kLyS6uZNrYPPudYta2csup6GvyObikJTJ/Qn5SEOEqr6lhfVMlv/r0SMxjWqzNPXj2BAT0Cz2Xh5hJeX5THlJEZTB6aTowZsTHGUT0784d3V3Pr6UM5IyuDU4f34uNVO/D7HeuKKhnWq/MBrY3WYO1tqsrs7GynsYZE2r66Bj8JcYHdkMWVtXRPSSAmxnDO8dnaYmYuKWBFQTnjBnRlSHonnv5iE4nxMYzum0ZqUhwfrtzB9vIa/nz5sQzr1Zk/f7CWhVtK2LW7Dp/fkRAXQ5+0JDbtrGJA9xR21zawM9id4ncQG2N0SYrjtZtPZEbOVnLzy/A7x/dOGoJzjjtfz6Wuwc9Zo3qza3cdN548hJGZXSivqadXatLe51FV18C1Ty1g4ZYSThvei0uO68dZo3q3+Pydczw3dzMfr97B/ZeM2ecxm+P3O+Zv2kX2wG7Excbw5uJ8bntpMb+5YBT/+9ZyfvatEdxw8pDDej3MbKFzLrvJ2xQEItLa3lycz09fWcKkYJfGnDVFjB/QlbNHZTJ7+XYWbi6hW0o8I3p3YeGWEuoa/EwY1I0uSfGsLqygtKqe0X3TqGnwkZtfhpnROTGOU4/uSWbXJJLjY7n4uH5kpiXvDZx6n5+leWUc1bMTOZtKuOffK/jZt0ZwzpjMJmvcUFTJd/4+n7LqeuJjjZKqwO96n+OX54zkxKHpPD5nA19t2ElhMJDOG9snnJuRsqp6xv/mfXx+R6/URGb/+GS6d0o4rMdSEIhIq6n3+Zm/cRfFlbXklVRT7/Nzw0mBb6kvLdjKjvIanvx8I8MzUimqrKXB5+fCY/sxc0k+xZV1DOnZiWu/MYjpE/qTGBdLcWUtheU1ZGV2OaDbo6y6nmuemk965wTuvXgM6Z0TW/W5+P0OB1TX+3jq841U1flYX1TJ+ysKiTHonBjH5GHpXDy+H1NGZrTq3w7V9L/NZd7GXTx//UROGtbkKNIhURCIyBHx+x3by2soqarjf17PZcnW0n1uPzMrg5p6H5+tLQZg0pAePHFNNsnBnbGxMcbu2gZKq+vp2zU57PUfigafn9/NWoXDcduUYXRNObxv4K1laV4pG4t3c/64vkf0OAoCEdmrwefny/U7SYqPZeLg7gdd1znH64vyeeST9azbUQlAalIcd593DGP7p9E7LZlXcrbyv2+tAOD+i8dw0fi+xMaYJzs15fAdLAh01JBIB7Msr4xV28upbfDz9tICiipqSU2Kp6KmntoGPxU1DZRV1wNw7phMeqYm0q9bCtdPHkxZdT2zlm0jN7+MkZldWLejkme+3ERWZhfuPi+LTolxnDg0nT6NvtVf+41B1Db4SUuO57IJ/SP1tOUIKAhEOpAVBeVc/OiX1Pn8AAzp2YnhGalU1jbQp2sSSfGxJMTGcOrwnqzYVsEjH6/DDOp9jj5pSTz66XqW5pWRHB9Ldf0WAK6fPJg7p4484OSpPcyM759yVNieo7Q+dQ2JdADlNfVsKt7N7TOWUFpdz3PfnUhCXAxD0jsdtIumpt6HGZz/1y9YXViBc/DXK4/lnNGZ5OaXs6OiJmI7SaV1qWtIpA2ra/Djd+6Qz3KdnbuNIT070+BzXPnkV5RW1WMGz1w3kZGZLZ8kBez9m3+8dCyXPjaXG08ewrljAodIju6XBqQdUk3SPikIRCKgqKKWNYUV5OaX8ficDVTUNjB5aDr3XxLaIZJ//Wgtf3xvDbExRkp8LJ2T4vj9t0czLCOVob06H3I9o/qmseiuMw85jKRjUBCIhFFFTT2PfrKev3++kdqGQD/+5KHpDOyRwgvztvDp6iIuPq7fAfcrq67nw5WFrC+qZNGWUr5cv5Pzx/Whc2IcOZtKeOw7xzE4vdMR1aYQiF4KApEwcM7x6sI87pu9iuLKOi4Y14fLJvQnMy2ZwemdaPD5eWVhHqsLK/bep6y6ntKqOnqmJnLlE1+xvKCc2BhjeEYqt5w2lB+fMYy4WI0kL0dOQSDSCsqq6qn1+aioaWDOmiJWb6+gvKaeUX3TGNSjE3PWFPHSgq1MGNSNv18zgbH9u+5z/7jYGIb27Myq7f8JgjtfX8bbS7fRv3sy+SXVPPLt8ZwxMmPv+D0irUVBIHKEZuRs5Zev5+49ZBMgvXMCKQlxzFq2fe+ym089itu/OZzYZg7DHNE7lS/WB87Mran38dGqHQzPSGVHRQ13TzuGqaObHjNH5EgpCESOwJOfbeA3/17J5KHpfGtUb2LNOGlYOv27B4YbLq2qY1tZDfGxMS3uxB3eO5XXFuVTWlXHoq2lVNX5uGPqCE4b3iscT0WimIJA5DAVlFbzx/dWc8bIXjx21XFN9td3TUkIeayao3unArB6ewXvryikU0Is3ziqR6vWLNIUdTaKHAbnHL+dtRLn4O5px7TKTtsRwSBYtb2CD1cWcvLRPUmM05E84j0FgUgTcjbt4ocvfs0D76854LadlbXc/MLX/HvpNn5w6lFNzqB1OHp3SaJLUhwPfbiWwvLakCY/EWkNnnYNmdlZwJ+BWOBJ59y9+90+AHgW6Bpc5w7n3CwvaxJpyb8W5nH7K0uIjTH8znH2qN57z9TNL63myie+YltpDT87azg3ndx6Y+yYGSN6d2H+pl18b/JgzhsT3klQJHp51iIws1jgYeBsIAu4wsyy9lvtl8AM59yxwOXAI17VI9KYz+94Yd5mvvfsAp6fu4nK2gYANu/czV1v5nL84O58/vPTSE2M4/7Zq4DAyWCXPTaXXbvreOmmE7j51KHNHgF0uO6YOoK/Xnksvzw3q9lB3kRam5ctgonAOufcBgAzewk4H1jRaB0H7BkUJQ0o8LAeiWJVdQ3c8uIi8kqqSI6PpbC8lu3lNaR3TuSDlTt4d3khz353Ij+ZEWgJPDh9HJlpyfzg1KHcN3sVX28pIb+kmvzSap6/fiLjB3TzpM7xA7p59tgizfEyCPoCWxtdzwOO32+du4H3zOxWoBNwRlMPZGY3AjcCDBgwoNULlY7vnrdX8vHqHUwZ0Ys6n2Ngj06cmZXBuWMyeeqLTdzz9gpuen4hCzeX8MBlY/eOt/+dSQN58IM1zFxcwO7aBtKS4/lGcB5ekY4i0oePXgE845z7PzObBDxvZqOcc/7GKznnHgceh8Aw1BGoU9op5xz/+Goz/5y/hZtOGcIvzh55wDrXTBrIKzlb+SB4pM6Fx/5nSsDOiXGccnRPZudux+8cJw1Lb/XuIJFI8zII8oHG0xX1Cy5r7HrgLADn3FwzSwLSgR0e1iVRoKC0mn8v3caHqwr5asMuJg9N5/Yzhze5blxsDL+7aDT3vrOK314w6oDx+6eO7s37KwoBOPnow588XKSt8jIIFgDDzGwwgQC4HLhyv3W2AFOAZ8xsJJAEFHlYk3RwDT4/f5uzgb98tJaaej/9uyfzq/OyuGbSoIPufB0/oBszbprU5G1TRmaQEBtDnc/PKQoC6YA8CwLnXIOZ3QK8S+DQ0Kecc8vN7NdAjnNuJnA78ISZ/ReBHcfXuvY2ZZq0Cc451hRWcufry8jZXMJZx/Tmf6aOZECPIz/Gv0tSPGdmZZBfWk1Gl6RWqFakbdFUldLu5ZdWc+1T81m7o5JOCbH87qLRnD+ub8t3PAQ19T78zpGSEOndaiKHR1NVSofl8zt+8vJiCkqrueeCUXwrK4NeHnxr16Qt0pEpCKRde/jjdczbuIs/XDKGS7P7t3wHETmAgkDanZcXbGFjcRUpCbE88P4aLhjXh0uamN5RREKjIJB25fO1xdzx2jL27No6Y2QGf7h07AGHfIpI6BQE0m7srKzlxy8vYmjPzjx17QRy88s4fWQv4jVvr8gRURBIu/H6onyKK+t49rsT6d89Ze8sYCJyZPRVStqNd3K3k5XZhWP6pEW6FJEORUEg7cL2shoWbi5h6mhN1iLS2hQE0iY45/h41Q4u+9tcXpi3GYBnv9zE52uLAXgndxsAZ4/OjFiNIh2V9hFIm3D/u6t59JP1xMUYy/PL6JIUz69mLmfioO5MHpbOO7nbGZ6RylE9O0e6VJEORy0Cibh/L93Go5+s57Lsfrxz20nUNvi59Z+LAFi8tZQdFYFuoTOzMiJcqUjHpCCQiCosr+Hn/1rK+AFdueeCUQzLSOXqSYMAuOqEAdT5/Pz1o3X4/E5DQIt4RF1DElH3vbOKugY/D04fR2JcYDyfX0wdwUXj+zKgRwovztvCP+dvITUxjmMHdI1wtSIdk1oEEjELN+/itUX5fO+kwQzs0Wnv8vjYGEb1TaNLUjyj+qZR73OcODRdJ46JeET/WRIRNfU+fv6vZWSmJfHD04Y2u94JQ3oAmhlMxEsKAomIB99fw7odldx78Rg6JTbfQ3n2qN4M6J7CGSN7hbE6keiifQQSdhU19Tz5+UYuPa5fi1M/HjugG3N+dlqYKhOJTmoRSFg45/h6Swn1Pj8bi3fj8zumjNThoCJtgVoE4pn80mpu/sdCsvqkkVdSxWdri/nthaPoHOwKGpzeqYVHEJFwUBCIJ5xz/PL1ZazaXsGawkrMICk+htz8Mnp3SQZgYCtMLC8iR05BIJ6YuaSAj1cXcde5WUyf0B8H3PBsDiu3VVBT76dPWpLmARZpIxQE0uqq63z8ftYqxvZL45pvDCI2JjB72PDeqczI2YpzjkHqFhJpM7SzWFrdM19uYnt5Df8zdeTeEAAY0TuVqjofuQXlCgKRNkQtAmk1pVV1vLt8O498so4pI3pxfPBksD1GZHYBwOd3DO6hIBBpKxQE0iq27Kzi8sfnUlBWQ//uyfxi6sgD1jk6ozNm4BxqEYi0IQoCOWJ7QqCq3seMmyYxYVA3zOyA9VIS4hjQPYXNO6sYnK4jhkTaCu0jkMNWU+9j5bZyrnjiK6rqfbzwveOZOLh7kyGwx4jeqcQYmnhepA1Ri0AO2ZKtpfz143V8sLIQ56BrSjz/uP74kCaVnz6hP/27pewdclpEIk9BIIdkeUEZ0x+fS3J8LDeeNISBPToxeWg6A0I8Oez0ERmcPkJDS4i0JQoCCcmu3XUs2lLCr2Yup2tyAm/dOpmeqYmRLktEWoGCQFpUXefj7D/PobC8luT4WF684XiFgEgHoiCQFr21pIDC8loenD6W00dkkJYcH+mSRKQVKQjkoJxzPDt3E8MzUrlgXN+DHhEkIu2TDh+Vg/p6SwnLC8q5+hsDFQIiHZSCQJrlnOMP766mW0o8F4zrG+lyRMQjCgJp1qxl2/lqwy5++q3hB51XWETaN0+DwMzOMrPVZrbOzO5oZp3LzGyFmS03sxe9rEdC1+Dz87tZK8nK7MLlEwZEuhwR8ZBnX/PMLBZ4GDgTyAMWmNlM59yKRusMA34BnOicKzGzXl7VI4dmztoi8kurueu8rH2GkhaRjsfLFsFEYJ1zboNzrg54CTh/v3VuAB52zpUAOOd2eFiPHIJXF+bRvVMCpw1XNot0dF52/PYFtja6ngccv986RwOY2RdALHC3c272/g9kZjcCNwIMGKBuCi8s2lLC83M3A3Di0HQ+WLGDb58wgIQ47UYS6egivQcwDhgGnAr0A+aY2WjnXGnjlZxzjwOPA2RnZ7twF9nRVdU18MMXvqaipoHE+BheW5QPwMXj+0W4MhEJBy+DIB/o3+h6v+CyxvKAec65emCjma0hEAwLPKxL9vPYJ+spKKthxk2TOG5gN95YlM/28hqO6dMl0qWJSBh4GQQLgGFmNphAAFwOXLnfOm8AVwBPm1k6ga6iDR7WJI28uTifP763mrySaqaN7cPEwd0BuPg4tQREoolnQeCcazCzW4B3CfT/P+WcW25mvwZynHMzg7d908xWAD7gv51zO72qSf6jrsHPPW+vpFtKPP91xtFcM2lQpEsSkQjxdB+Bc24WMGu/ZXc1uuyAnwR/JIzeyd1GcWUtf7x0DKfqyCCRqKZDQqLUc3M3M6hHCicP6xnpUkQkwhQEUWhFQTkLN5dw1QkDidHJYiJRT0EQhV5dmEd8rOnwUBEBFARRp97n583F+ZwxMoNunRIiXY6ItAEhBYGZvWZm55iZgqOd+2R1ETt313GJDhEVkaBQP9gfIXAOwFozu9fMhntYk3igqq6Bn726hNtnLCa9cyInH62dxCISEFIQOOc+cM59GxgPbAI+MLMvzew6M9MEtu3AqwvzmJGTx5SRGTx1bTbxsWrciUhAyOcRmFkP4CrgO8Ai4AVgMnANgbGCpA2bkbOVrMwuPDh9XKRLEZE2JtR9BK8DnwEpwHnOuWnOuZedc7cCnb0sUI7c8oIycvPLmT6hf8sri0jUCbVF8JBz7uOmbnDOZbdiPdKK/H7HBysLeXbuJhLiYjh/XJ9IlyQibVCoHcVZZtZ1zxUz62ZmN3tUk7SSP324lhufX0jOphJuPvUouqbocFEROVCoLYIbnHMP77kSnFbyBgJHE0kb9PnaYv7y0VouHt+P3180WhPMiEizQv10iDWzvWMRBOcj1tfLNmp9USW3/vNrhvbszD0XHKMQEJGDCrVFMBt42cz+Frx+U3CZtDHFlbVc/ff5xMYYT16TTUpCpCehE5G2LtRPiZ8T+PD/QfD6+8CTnlQkR+SRj9ezvbyGN24+kYE9OkW6HBFpB0IKAuecH3g0+CNtVGlVHS8t2MK0sX0Y3S8t0uWISDsRUhCY2TDg90AWkLRnuXNuiEd1yWF4bu5mqup83HSKXhYRCV2oXUNPA78CHgROA65DI5e2GTX1Pu6bvYrn5m5myohejOitSedFJHShfpgnO+c+BMw5t9k5dzdwjndlyaF49JP1PP3FJi6f0J8HLtMQEiJyaEJtEdQGh6BeG5yQPh8NLdEm1DX4eWHeFk4f0YvfXjg60uWISDsUaovgNgLjDP0IOI7A4HPXeFWUhG7PJPRXTxoY6VJEpJ1qsUUQPHlsunPup0Algf0D0gaU19Tz+JwNmoReRI5Iiy0C55yPwHDT0oZsLN7NuQ99zqrtFdz+zeGahF5EDluo+wgWmdlM4BVg956FzrnXPKlKWvTnD9awa3cdL994AtmDuke6HBFpx0INgiRgJ3B6o2UOUBBEQFl1Pe/kbufS7H4KARE5YqGeWaz9Am3IzCUF1Db4uSxbE82IyJEL9czipwm0APbhnPtuq1ckB+X3O15esIURvVMZ3VfDSIjIkQu1a+jtRpeTgAuBgtYvR1ry6Kfryc0v576LR9NoZHARkcMWatfQvxpfN7N/Ap97UpE0a3buNv743mqmje2jbiERaTWHO1j9MKBXaxYiB/fYp+u5b/YqxvTryr1qDYhIKwp1H0EF++4j2E5gjgIJg43Fu7n3nVWcPao3D04fR1J8bKRLEpEOJNSuoVSvC5HmzVq2DYBfnpulEBCRVhfSWENmdqGZpTW63tXMLvCuLGnsndxtjOvflb5dkyNdioh0QKEOOvcr51zZnivOuVIC8xOIx7buqiI3v5ypo3tHuhQR6aBCDYKm1tOs6GEwc0ngKN2zR2VGuBIR6ahCDYIcM3vAzI4K/jwALPSyMIGiiloe+3Q9k4em0797SqTLEZEOKtQguBWoA14GXgJqgB+2dCczO8vMVpvZOjO74yDrXWxmzsyyQ6wnKvx+1kpq6n3cPe2YSJciIh1YqEcN7Qaa/SBvSnAeg4eBM4E8YIGZzXTOrdhvvVQCE9/MO5TH78hKdtdx18zlvLWkgFtOG8rQXpoMTkS8E+pRQ++bWddG17uZ2bst3G0isM45t8E5V0egJXF+E+vdA9xHoJUhwI9eWsTs3G385Myj+fEZwyJdjoh0cKF2DaUHjxQCwDlXQstnFvcFtja6nhdctpeZjQf6O+f+fbAHMrMbzSzHzHKKiopCLLl9+mxtEZ+tLeaOs0fyoynDiIsN9SUSETk8oX7K+M1swJ4rZjaIJkYjPRRmFgM8ANze0rrOucedc9nOueyePTvulIx+v+Ped1bRr1syV50woOU7iIi0glAPAb0T+NzMPgUMOAm4sYX75AONR0brF1y2RyowCvgkOG5Ob2CmmU1zzuWEWFeH8uGqHSwvKOeBy8aSGKcziEUkPELdWTw7eETPjcAi4A2guoW7LQCGmdlgAgFwOXBlo8csA9L3XDezT4CfRmsIADzx2Qb6dk1m2tg+kS5FRKJIqIPOfY/AkT39gMXACcBc9p26ch/OuQYzuwV4F4gFnnLOLTezXwM5zrmZR1p8R7I0r5T5G3fxy3NGar+AiIRVqF1DtwETgK+cc6eZ2Qjgdy3dyTk3C5i137K7mln31BBr6TCcc8xcUsCfPlhLQWk1qYlxTJ+geQZEJLxCDYIa51yNmWFmic65VWY23NPKosD9767m0U/WM6ZfGldPGsiZWb1JTYqPdFkiEmVCDYK84HkEbwDvm1kJsNm7sqLDm4vyOXV4T/5+zQRiYzTRjIhERqg7iy8MXrzbzD4G0oDZnlUVBXZW1lJQVsN1Jw5WCIhIRB3yCKLOuU+9KCTaLC8oB+CYvl0iXImIRDsdnhIhuQWB6R2O6ZPWwpoiIt5SEETI8vxyBnRPIS1ZO4dFJLIUBBGSW1DGKHULiUgboCCIgLLqejbvrFK3kIi0CQqCCFiaFxjI9Zg+ahGISOQpCCLg7SXb6JQQy/GDe0S6FBERBUG41dT7mLVsG2eNyiQ5QSOMikjkKQjC7IOVhVTUNnDR+L4trywiEgYKgjCqa/Dz3JebyUxL4oQh6hYSkbZBQRAmtQ0+fvCPhczftIvbpgzTsBIi0mYoCMLkha+28OGqHfzmglFcPlHTUIpI26EgCAPnHP+Yt5ljB3TlqhMGRrocEZF9KAjCYO6GnWwo2s1VxysERKTtURCEwQvztpCWHM85YzIjXYqIyAEUBB6rqffx4cpCzh/Xh6R4nTcgIm2PgsBjOZtKqKn3c9rwXpEuRUSkSQoCj322toj4WOP4Id0jXYqISJMUBB77dE0R2QO7k5JwyJPBiYiEhYLAQzvKa1i1vYKTjk6PdCkiIs1SEHjo83XFAJw8rGeEKxERaZ6CwENz1hTRo1MCWZmad0BE2i4FgUf8fsfn64qZPCydGI0rJCJtmILAIyu3l1NcWaduIRFp8xQEHpmzJrB/4KRh2lEsIm2bgsAjn60tYkTvVHp1SYp0KSIiB6Ug8EBNvY+czSVMHqrWgIi0fQoCDyzNK6Ouwc/xmoVMRNoBBYEH5m/cCcCEQd0iXImISMsUBB6Yv6mE4RmpdE1JiHQpIiItUhC0sgafn4WbdjFxsAaZE5H2QUHQylZuq2B3nY8JCgIRaScUBK1sztoiACYOUhCISPvgaRCY2VlmttrM1pnZHU3c/hMzW2FmS83sQzNr15P6Nvj8vDhvC5OG9KB3ms4fEJH2wbMgMLNY4GHgbCALuMLMsvZbbRGQ7ZwbA7wK3O9VPeHw/opC8kurue7EQZEuRUQkZF62CCYC65xzG5xzdcBLwPmNV3DOfeycqwpe/Qro52E9nnvqi430757MlJEZkS5FRCRkXgZBX2Bro+t5wWXNuR54p6kbzOxGM8sxs5yioqJWLLH15JVUsWBTCd8+fiCxGm1URNqRNrGz2MyuArKBPzR1u3PucedctnMuu2fPtjma50erdgDwzSy1BkSkffFyIt18oH+j6/2Cy/ZhZmcAdwKnOOdqPazHUx+u3MHg9E4M6dk50qWIiBwSL1sEC4BhZjbYzBKAy4GZjVcws2OBvwHTnHM7PKzFU7trG5i7fidTRvSKdCkiIofMsyBwzjUAtwDvAiuBGc655Wb2azObFlztD0Bn4BUzW2xmM5t5uDbts7XF1Pn8nD5SQSAi7Y+XXUM452YBs/Zbdlejy2d4+ffD5aNVhaQmxTFBJ5GJSDvUJnYWt2d+v+OjVUWccnRP4mO1OUWk/dEn1xFakldKcWUtZ+jcARFppxQER+ijVTuIMTjl6LZ5WKuISEs83UfQkc3I2cozX2xi5+5asgd2p1snzT0gIu2TWgSH6a0lBawrqqSwvJZzxmRGuhwRkcOmFsFhcM6xNK+Mi8f3446zRpCapM0oIu2XPsEOw5ZdVZRV1zOmXxppKfGRLkdE5Iioa+gwLM0rA2B037QIVyIicuQUBIdhWX4ZCXExHJ2RGulSRESOmILgMCzNK2VkZhcS4rT5RKT90yfZIdiys4rn5m5iWV4ZY9QtJCIdhHYWH4LfzqPjatwAAAmsSURBVFrBu8sLAThxaI8IVyMi0joUBIdgydYypo7uze8vHKOjhUSkw1DXUIh2lNewvbyG8QO6KQREpENREIRoWX7gkNGx/btGuBIRkdalIAjR0rwyYgyyMrtEuhQRkValIAjR0rxShvbqTKdE7VYRkY5FQRAC5xzL8ssY3VfdQiLS8SgIQrBgUwnFlXWM6adzB0Sk41EQtGDBpl1c9/R8BvZI0XDTItIhKQgOoqqugVtfXESvLknMuGkS6Z0TI12SiEir057Pg3j0k/VsL6/h1e9PIqNLUqTLERHxhFoEzVi9vYK/zdnABeP6kD2oe6TLERHxjFoEjSzcXMLP/7WUqaN6MyMnj67J8fxi6shIlyUi4ikFQVB5TT23vbSIkt11PPTROlIT45ihLiERiQIKgqB73lrBtrIaZtw0icS4GJLiYxjaSxPPiEjHpyAA1hdV8urXedxw0hCOG9gt0uWIiISVdhYDj3y8nsS4GG48eUikSxERCbuoD4IVBeW8sTifKyYO0HkCIhKVorJrqLC8hjcX55NXUs1LC7bSNTmem04+KtJliYhERNQFwZuL8/l/b+RSXtNAQlwMZ47M4O5px9AzVa0BEYlOURUE/5y/hV+8tozjBnbj/kvGMCS9E2YW6bJERCIqaoLgjUX5/M/ryzhteE/+9p1sEuKifveIiAgQRTuL+3RN5syRGTx61XEKARGRRqKmRTBxcHcmDtaYQSIi+9NXYxGRKOdpEJjZWWa22szWmdkdTdyeaGYvB2+fZ2aDvKxHREQO5FkQmFks8DBwNpAFXGFmWfutdj1Q4pwbCjwI3OdVPSIi0jQvWwQTgXXOuQ3OuTrgJeD8/dY5H3g2ePlVYIrpeE4RkbDyMgj6AlsbXc8LLmtyHedcA1AG9Nj/gczsRjPLMbOcoqIij8oVEYlO7WJnsXPucedctnMuu2fPnpEuR0SkQ/EyCPKB/o2u9wsua3IdM4sD0oCdHtYkIiL78TIIFgDDzGywmSUAlwMz91tnJnBN8PIlwEfOOedhTSIish/z8nPXzKYCfwJigaecc781s18DOc65mWaWBDwPHAvsAi53zm1o4TGLgM2HWVI6UHyY9/VaW61NdR0a1XXo2mptHa2ugc65JvvWPQ2CtsbMcpxz2ZGuoylttTbVdWhU16Frq7VFU13tYmexiIh4R0EgIhLloi0IHo90AQfRVmtTXYdGdR26tlpb1NQVVfsIRETkQNHWIhARkf0oCEREolzUBEFLQ2KHsY7+Zvaxma0ws+Vmdltw+d1mlm9mi4M/UyNQ2yYzWxb8+znBZd3N7H0zWxv83S3MNQ1vtE0Wm1m5mf04UtvLzJ4ysx1mlttoWZPbyAIeCr7nlprZ+DDX9QczWxX826+bWdfg8kFmVt1o2z0W5rqafe3M7BfB7bXazL7lVV0Hqe3lRnVtMrPFweVh2WYH+Xzw9j3mnOvwPwROaFsPDAESgCVAVoRqyQTGBy+nAmsIDNN9N/DTCG+nTUD6fsvuB+4IXr4DuC/Cr+N2YGCkthdwMjAeyG1pGwFTgXcAA04A5oW5rm8CccHL9zWqa1Dj9SKwvZp87YL/B0uARGBw8H82Npy17Xf7/wF3hXObHeTzwdP3WLS0CEIZEjssnHPbnHNfBy9XACs5cFTWtqTxUOHPAhdEsJYpwHrn3OGeWX7EnHNzCJwF31hz2+h84DkX8BXQ1cwyw1WXc+49FxjVF+ArAuN9hVUz26s55wMvOedqnXMbgXUE/nfDXpuZGXAZ8E+v/n4zNTX3+eDpeyxagiCUIbHDzgIzsh0LzAsuuiXYvHsq3F0wQQ54z8wWmtmNwWUZzrltwcvbgYwI1LXH5ez7jxnp7bVHc9uoLb3vvkvgm+Meg81skZl9amYnRaCepl67trS9TgIKnXNrGy0L6zbb7/PB0/dYtARBm2NmnYF/AT92zpUDjwJHAeOAbQSapeE22Tk3nsCscj80s5Mb3+gCbdGIHG9sgYELpwGvBBe1he11gEhuo+aY2Z1AA/BCcNE2YIBz7ljgJ8CLZtYljCW1ydduP1ew75eOsG6zJj4f9vLiPRYtQRDKkNhhY2bxBF7kF5xzrwE45wqdcz7nnB94Ag+bxM1xzuUHf+8AXg/WULinqRn8vSPcdQWdDXztnCsM1hjx7dVIc9so4u87M7sWOBf4dvADhGDXy87g5YUE+uKPDldNB3ntIr69YO+Q+BcBL+9ZFs5t1tTnAx6/x6IlCEIZEjssgn2PfwdWOuceaLS8cb/ehUDu/vf1uK5OZpa65zKBHY257DtU+DXAm+Gsq5F9vqFFenvtp7ltNBO4OnhkxwlAWaPmvefM7CzgZ8A051xVo+U9LTCnOGY2BBgGHHTU31auq7nXbiZwuZklmtngYF3zw1VXI2cAq5xzeXsWhGubNff5gNfvMa/3greVHwJ719cQSPI7I1jHZALNuqXA4uDPVALDcS8LLp8JZIa5riEEjthYAizfs40ITB36IbAW+ADoHoFt1onAhEVpjZZFZHsRCKNtQD2B/tjrm9tGBI7keDj4nlsGZIe5rnUE+o/3vM8eC657cfA1Xgx8DZwX5rqafe2AO4PbazVwdrhfy+DyZ4Dv77duWLbZQT4fPH2PaYgJEZEoFy1dQyIi0gwFgYhIlFMQiIhEOQWBiEiUUxCIiEQ5BYFIkJn5bN+RTlttlNrg6JWRPNdBpFlxkS5ApA2pds6Ni3QRIuGmFoFIC4Lj0t9vgbka5pvZ0ODyQWb2UXDwtA/NbEBweYYFxv9fEvz5RvChYs3sieA48++ZWXJw/R8Fx59famYvRehpShRTEIj8R/J+XUPTG91W5pwbDfwV+FNw2V+AZ51zYwgM6PZQcPlDwKfOubEExrtfHlw+DHjYOXcMUErgbFUIjC9/bPBxvu/VkxNpjs4sFgkys0rnXOcmlm8CTnfObQgOCLbdOdfDzIoJDI9QH1y+zTmXbmZFQD/nXG2jxxgEvO+cGxa8/nMg3jn3GzObDVQCbwBvOOcqPX6qIvtQi0AkNK6Zy4eittFlH//ZR3cOgfFixgMLgqNfioSNgkAkNNMb/Z4bvPwlgZFsAb4NfBa8/CHwAwAzizWztOYe1MxigP7OuY+BnwNpwAGtEhEv6ZuHyH8kW3Cy8qDZzrk9h5B2M7OlBL7VXxFcdivwtJn9N1AEXBdcfhvwuJldT+Cb/w8IjHLZlFjgH8GwMOAh51xpqz0jkRBoH4FIC4L7CLKdc8WRrkXEC+oaEhGJcmoRiIhEObUIRESinIJARCTKKQhERKKcgkBEJMopCEREotz/B/QEKGYQpT00AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def plot_graphs(history, string):\n",
        "  plt.plot(history.history[string])\n",
        "  plt.xlabel(\"Epochs\")\n",
        "  plt.ylabel(string)\n",
        "  plt.show()\n",
        "\n",
        "plot_graphs(history, 'accuracy')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1rAgRpxYhjpB"
      },
      "source": [
        "### Generate new lyrics!\n",
        "\n",
        "It's finally time to generate some new lyrics from the trained model, and see what we get. To do so, we'll provide some \"seed text\", or an input sequence for the model to start with. We'll also decide just how long of an output sequence we want - this could essentially be infinite, as the input plus the previous output will be continuously fed in for a new output word (at least up to our max sequence length)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "DC7zfcgviDTp",
        "outputId": "2fa10bb3-524f-456e-b86f-26c6e61ddc8e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "im feeling chills of your music and i to talk enough handle anywhere anywhere weave weave question music music behind again again cause i see again hour hour body here again again again again again body again be are sorrow leaves again body be bang have at again body again again again again again again again again and look at from the eye from body please from have life body here again again again again again again again again again behind body again body again be only before have eyes touch other park pain pain will end but a break mistake figure park\n"
          ]
        }
      ],
      "source": [
        "seed_text = \"im feeling chills\"\n",
        "next_words = 100\n",
        "  \n",
        "for _ in range(next_words):\n",
        "\ttoken_list = tokenizer.texts_to_sequences([seed_text])[0]\n",
        "\ttoken_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')\n",
        "\tpredicted = np.argmax(model.predict(token_list), axis=-1)\n",
        "\toutput_word = \"\"\n",
        "\tfor word, index in tokenizer.word_index.items():\n",
        "\t\tif index == predicted:\n",
        "\t\t\toutput_word = word\n",
        "\t\t\tbreak\n",
        "\tseed_text += \" \" + output_word\n",
        "print(seed_text)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "l10c03_nlp_constructing_text_generation_model.ipynb",
      "toc_visible": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}