Read_Make_Inputs_transformer.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Read_Make_Inputs_transformer.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "s_qNSzzyaCbD"
      ],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/UtpalMattoo/NLP-Attention-Transformers/blob/master/Read_Make_Inputs_transformer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s_qNSzzyaCbD"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "jmjh290raIky"
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J0Qjg6vuaHNt"
      },
      "source": [
        "# Transformer model for language understanding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AOpGoE2T-YXS"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/text/transformer\">\n",
        "    <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />\n",
        "    View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/text/transformer.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
        "    Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/text/transformer.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
        "    View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/text/transformer.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M-f8TnGpE_ex"
      },
      "source": [
        "This tutorial trains a <a href=\"https://arxiv.org/abs/1706.03762\" class=\"external\">Transformer model</a> to translate Portuguese to English. This is an advanced example that assumes knowledge of [text generation](text_generation.ipynb) and [attention](nmt_with_attention.ipynb).\n",
        "\n",
        "The core idea behind the Transformer model is *self-attention*—the ability to attend to different positions of the input sequence to compute a representation of that sequence. Transformer creates stacks of self-attention layers and is explained below in the sections *Scaled dot product attention* and *Multi-head attention*.\n",
        "\n",
        "A transformer model handles variable-sized input using stacks of self-attention layers instead of [RNNs](text_classification_rnn.ipynb) or [CNNs](../images/intro_to_cnns.ipynb). This general architecture has a number of advantages:\n",
        "\n",
        "* It make no assumptions about the temporal/spatial relationships across the data. This is ideal for processing a set of objects (for example, [StarCraft units](https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/#block-8)).\n",
        "* Layer outputs can be calculated in parallel, instead of a series like an RNN.\n",
        "* Distant items can affect each other's output without passing through many RNN-steps, or convolution layers (see [Scene Memory Transformer](https://arxiv.org/pdf/1903.03878.pdf) for example).\n",
        "* It can learn long-range dependencies. This is a challenge in many sequence tasks.\n",
        "\n",
        "The downsides of this architecture are:\n",
        "\n",
        "* For a time-series, the output for a time-step is calculated from the *entire history* instead of only the inputs and current hidden-state. This _may_ be less efficient.   \n",
        "* If the input *does* have a  temporal/spatial relationship, like text, some positional encoding must be added or the model will effectively see a bag of words. \n",
        "\n",
        "After training the model in this notebook, you will be able to input a Portuguese sentence and return the English translation.\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/tutorials/transformer/attention_map_portuguese.png\" width=\"800\" alt=\"Attention heatmap\">"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XFG0NDRu5mYQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fae6cb97-1800-42d8-e112-28acc3b13245"
      },
      "source": [
        "!pip install -q tfds-nightly\n",
        "\n",
        "# Pin matplotlib version to 3.2.2 since in the latest version\n",
        "# transformer.ipynb fails with the following error:\n",
        "# https://stackoverflow.com/questions/62953704/valueerror-the-number-of-fixedlocator-locations-5-usually-from-a-call-to-set\n",
        "!pip install matplotlib==3.2.2"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: matplotlib==3.2.2 in /usr/local/lib/python3.6/dist-packages (3.2.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.2.2) (0.10.0)\n",
            "Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.2.2) (1.19.4)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.2.2) (1.3.1)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.2.2) (2.4.7)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib==3.2.2) (2.8.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib==3.2.2) (1.15.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dKSOOalvbhhs"
      },
      "source": [
        "# tf.data: Build TensorFlow input pipelines\n",
        "# https://www.tensorflow.org/guide/data\n",
        "# https://www.tensorflow.org/api_docs/python/tf/data/experimental\n"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JjJJyJTZYebt"
      },
      "source": [
        "import tensorflow_datasets as tfds\n",
        "import tensorflow as tf\n",
        "\n",
        "import time\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fd1NWMxjfsDd"
      },
      "source": [
        "## Setup input pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t4_Qt8W1hJE_"
      },
      "source": [
        "Use [TFDS](https://www.tensorflow.org/datasets) to load the [Portugese-English translation dataset](https://github.com/neulab/word-embeddings-for-nmt) from the [TED Talks Open Translation Project](https://www.ted.com/participate/translate).\n",
        "\n",
        "This dataset contains approximately 50000 training examples, 1100 validation examples, and 2000 test examples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8q9t4FmN96eN"
      },
      "source": [
        "examples, metadata = tfds.load('ted_hrlr_translate/pt_to_en', with_info=True,\n",
        "                               as_supervised=True)\n",
        "train_examples, val_examples = examples['train'], examples['validation']"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RCEKotqosGfq"
      },
      "source": [
        "Create a custom subwords tokenizer from the training dataset. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KVBg5Q8tBk5z",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b09d1081-5823-40c1-a3ac-73e502f10eb3"
      },
      "source": [
        "# https://www.tensorflow.org/datasets/api_docs/python/tfds/deprecated/text/SubwordTextEncoder#build_from_corpus\n",
        "# https://colab.research.google.com/github/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_for_deep_learning/l10c01_nlp_lstms_with_reviews_subwords_dataset.ipynb#scrollTo=33AthPiALFZK\n",
        "# SubwordTextEncoder.build_from_corpus() will create a tokenizer for us. You could also use this \n",
        "# functionality to get subwords from a much larger corpus of text as well\n",
        "\n",
        "tokenizer_en = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(\n",
        "    (en.numpy() for pt, en in train_examples), target_vocab_size=2**13)\n",
        "\n",
        "tokenizer_pt = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(\n",
        "    (pt.numpy() for pt, en in train_examples), target_vocab_size=2**13)\n",
        "\n",
        "print (type(tokenizer_en))"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'tensorflow_datasets.core.deprecated.text.subword_text_encoder.SubwordTextEncoder'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sV3rRQsMBues"
      },
      "source": [
        "# i=11\n",
        "# j=0\n",
        "# for data in train_examples:\n",
        "#   if (j < i):\n",
        "#     a, b = data\n",
        "#     print (f\"a is: {a} and \\n b is: {b}\\n\") #a and b are byte strings\n",
        "#     j+=1"
      ],
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4DYWukNFkGQN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e22d9d2c-d8de-456b-9a37-339decd9dc22"
      },
      "source": [
        "\"\"\"\n",
        "e quando melhoramos a procura , tiramos a única vantagem da impressão , que é a serendipidade .\n",
        "[8214, 6, 40, 4092, 57, 3, 1687, 1, 6155, 12, 3, 461, 6770, 19, 5227, 1088, 97, 1, 5, 8, 3, 4213, 3408, 7256, 1670, 2, 8215]\n",
        "mas e se estes fatores fossem ativos ?\n",
        "[8214, 25, 6, 16, 138, 7800, 2004, 2445, 8073, 29, 8215]\n",
        "mas eles não tinham a curiosidade de me testar .\n",
        "[8214, 25, 66, 13, 342, 3, 5278, 7990, 4, 38, 3625, 8072, 2, 8215]\n",
        "e esta rebeldia consciente é a razão pela qual eu , como agnóstica , posso ainda ter fé .\n",
        "[8214, 6, 54, 3906, 2682, 156, 2646, 7990, 8, 3, 496, 139, 216, 354, 1, 21, 1712, 243, 4206, 1, 375, 130, 75, 1960, 2, 8215]\n",
        "`` `` '' podem usar tudo sobre a mesa no meu corpo . ''\n",
        "[8214, 149, 74, 258, 123, 60, 3, 4088, 22, 73, 806, 37, 8215]\n",
        "\"\"\"\n",
        "\n",
        "sample_string = 'and when you improve searchability , you actually take away the one advantage of print , which is serendipity .'\n",
        "\n",
        "tokenized_string = tokenizer_en.encode(sample_string)\n",
        "print ('Tokenized string is {}'.format(tokenized_string))\n",
        "\n",
        "original_string = tokenizer_en.decode(tokenized_string)\n",
        "print ('The original string: {}'.format(original_string))\n",
        "\n",
        "assert original_string == sample_string"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tokenized string is [4, 59, 15, 1792, 6561, 3060, 7952, 1, 15, 103, 134, 378, 3, 47, 6122, 6, 5311, 1, 91, 13, 1849, 559, 1609, 894, 2]\n",
            "The original string: and when you improve searchability , you actually take away the one advantage of print , which is serendipity .\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o9KJWJjrsZ4Y"
      },
      "source": [
        "The tokenizer encodes the string by breaking it into subwords if the word is not in its dictionary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bf2ntBxjkqK6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6e4c263f-5e5c-447d-ff8f-f61e63a3b594"
      },
      "source": [
        "for ts in tokenized_string:\n",
        "  print ('{} ----> {}'.format(ts, tokenizer_en.decode([ts])))"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "4 ----> and \n",
            "59 ----> when \n",
            "15 ----> you \n",
            "1792 ----> improve \n",
            "6561 ----> search\n",
            "3060 ----> abilit\n",
            "7952 ----> y\n",
            "1 ---->  , \n",
            "15 ----> you \n",
            "103 ----> actually \n",
            "134 ----> take \n",
            "378 ----> away \n",
            "3 ----> the \n",
            "47 ----> one \n",
            "6122 ----> advantage \n",
            "6 ----> of \n",
            "5311 ----> print\n",
            "1 ---->  , \n",
            "91 ----> which \n",
            "13 ----> is \n",
            "1849 ----> ser\n",
            "559 ----> end\n",
            "1609 ----> ip\n",
            "894 ----> ity\n",
            "2 ---->  .\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bcRp7VcQ5m6g"
      },
      "source": [
        "BUFFER_SIZE = 20000\n",
        "BATCH_SIZE = 64"
      ],
      "execution_count": 66,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kGi4PoVakxdc"
      },
      "source": [
        "Add a start and end token to the input and target. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mqIV4EOcAsPB"
      },
      "source": [
        "def encode(lang1, lang2):\r\n",
        "\r\n",
        "  lst1 = [tokenizer_pt.vocab_size]\r\n",
        "  lst2 = [tokenizer_en.vocab_size]\r\n",
        "\r\n",
        "  # print (f\"Protugese source sentence - = {lang1.numpy()}\")\r\n",
        "  lang1 = [tokenizer_pt.vocab_size] + tokenizer_pt.encode(\r\n",
        "      lang1) + [tokenizer_pt.vocab_size+1]\r\n",
        " \r\n",
        "  # print (f\"Protugese tokenized vector - beginning and end of vector is vocab_size and vocab_size+1 for Portugese = {lang1}\\n\")\r\n",
        "  \r\n",
        "  # print (f\"English source sentence - = {lang2.numpy()}\")\r\n",
        "  lang2 = [tokenizer_en.vocab_size] + tokenizer_en.encode(\r\n",
        "      lang2) + [tokenizer_en.vocab_size+1]\r\n",
        "\r\n",
        "  # print (f\"English tokenized vector - beginning and end of vector is vocab_size and vocab_size+1 for English = {lang2}\\n\")\r\n",
        "  \r\n",
        "  return lang1, lang2\r\n"
      ],
      "execution_count": 101,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tx1sFbR-9fRs"
      },
      "source": [
        "You want to use `Dataset.map` to apply this function to each element of the dataset.  `Dataset.map` runs in graph mode.\n",
        "\n",
        "* Graph tensors do not have a value. \n",
        "* In graph mode you can only use TensorFlow Ops and functions. \n",
        "\n",
        "So you can't `.map` this function directly: You need to wrap it in a `tf.py_function`. The `tf.py_function` will pass regular tensors (with a value and a `.numpy()` method to access it), to the wrapped python function."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6JrGp5Gek6Ql"
      },
      "source": [
        "Note: To keep this example small and relatively fast, drop examples with a length of over 40 tokens."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nFQj2MoO8WH0"
      },
      "source": [
        "MAX_LEN = 30"
      ],
      "execution_count": 102,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9mk9AZdZ5bcS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0a99b6ea-39cd-445a-a2ed-7b2005fa475d"
      },
      "source": [
        "# https://www.tensorflow.org/api_docs/python/tf/data/Dataset\n",
        "# https://www.tensorflow.org/tutorials/load_data/numpy\n",
        "# https://stackoverflow.com/questions/62436302/accessing-data-in-tensorflow-prefetchdataset\n",
        "\n",
        "print (f\"number of training examples: {len(train_examples)}\")\n",
        "print (f\"type of training examples: {type(train_examples)}\\n\") # <class 'tensorflow.python.data.ops.dataset_ops.PrefetchDataset'>\n",
        "lst = list(train_examples) # convert to list and then to tf.data.Dataset.from_tensor_slices\n",
        "tf_slices_train_examples = tf.data.Dataset.from_tensor_slices(lst)\n",
        "\n",
        "source_pt = []\n",
        "target_en = []\n",
        "result_pt = []\n",
        "result_en = []\n",
        "\n",
        "#Finally, access tensor elements in tensor of type tf.data.Dataset.from_tensor_slices\n",
        "for example_no, source_target in enumerate(tf_slices_train_examples.take(100)):\n",
        "\n",
        "  source = source_target[0].numpy().decode('UTF-8')\n",
        "  target = source_target[1].numpy().decode('UTF-8')\n",
        "  if (len(source) < MAX_LEN) and (len(target) < MAX_LEN):\n",
        "    source_pt.append(source)\n",
        "    target_en.append(target)\n",
        "    pt, en = encode(source, target)\n",
        "    result_pt.append(pt)\n",
        "    result_en.append(en)\n",
        "  else: \n",
        "    continue"
      ],
      "execution_count": 103,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "number of training examples: 51785\n",
            "type of training examples: <class 'tensorflow.python.data.ops.dataset_ops.PrefetchDataset'>\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UamEoLgnYmBR"
      },
      "source": [
        "from tensorflow import keras\r\n",
        "\r\n",
        "# for item in  zip(source_pt, result_pt, target_en, result_en):\r\n",
        "#    if (len(item[1]) >  max_source_pt_len):      \r\n",
        "#       max_source_pt_len = len(item[1])\r\n",
        "\r\n",
        "def pad_sequence(result_pt, result_en):\r\n",
        "  padded_inputs_pt = tf.keras.preprocessing.sequence.pad_sequences(result_pt, padding=\"post\")\r\n",
        "  padded_inputs_en = tf.keras.preprocessing.sequence.pad_sequences(result_en, padding=\"post\")\r\n",
        "  return padded_inputs_pt, padded_inputs_en"
      ],
      "execution_count": 104,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nDId2PhKNeM9",
        "outputId": "29777a12-77c8-422e-8d59-6fa351a6cd00"
      },
      "source": [
        "padded_inputs_pt, padded_inputs_en = pad_sequence(result_pt, result_en)\r\n",
        "print (\"Padded input sequences:\\n\")\r\n",
        "print (padded_inputs_pt)\r\n",
        "print (\"\\nPadded output sequences:\\n\")\r\n",
        "print (padded_inputs_en)"
      ],
      "execution_count": 105,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Padded input sequences:\n",
            "\n",
            "[[8214    3 4675   53    9 1537    2 8215]\n",
            " [8214   27   56 5569 8055    2 8215    0]\n",
            " [8214   42   13  170 2610    2 8215    0]\n",
            " [8214  212    2 8215    0    0    0    0]\n",
            " [8214  273    1  179 7240 8055    2 8215]\n",
            " [8214   67  107  173 8215    0    0    0]\n",
            " [8214    3  313   95  410    2 8215    0]\n",
            " [8214   54   53    3   69  486    2 8215]\n",
            " [8214  372    2 8215    0    0    0    0]]\n",
            "\n",
            "Padded output sequences:\n",
            "\n",
            "[[8087    3 3978   20 2981    2 8088    0    0    0    0]\n",
            " [8087   12   20  952 7931    2 8088    0    0    0    0]\n",
            " [8087   16   97   36 1537    2 8088    0    0    0    0]\n",
            " [8087  153   51    2 8088    0    0    0    0    0    0]\n",
            " [8087  307    1  204    1 7936    8   78 5054    2 8088]\n",
            " [8087   94  136  192 8088    0    0    0    0    0    0]\n",
            " [8087    3  338   20   79    2 8088    0    0    0    0]\n",
            " [8087   18   10   20   32  422    2 8088    0    0    0]\n",
            " [8087  153   51    2 8088    0    0    0    0    0    0]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a_b4ou4TYqUN"
      },
      "source": [
        "## Masking"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s42Uydjkv0hF"
      },
      "source": [
        "Mask all the pad tokens in the batch of sequence. It ensures that the model does not treat padding as the input. The mask indicates where pad value `0` is present: it outputs a `1` at those locations, and a `0` otherwise."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U2i8-e1s8ti9"
      },
      "source": [
        "def create_padding_mask(seq):\n",
        "  # print (\"Padded input:\\n\")\n",
        "  # print (seq)\n",
        "  # print ()\n",
        "  # print (\"Masked paddings:\\n\")\n",
        "  seq = tf.cast(tf.math.equal(seq, 0), tf.int32)\n",
        "  # why add a new axis when a padding mask can is returned by tf.cast(tf.math.equal(seq, 0), tf.float32)\n",
        "  # return seq[:, tf.newaxis, tf.newaxis, :]  # (batch_size, 1, 1, seq_len)\n",
        "  return seq[:, :]  # (batch_size, 1, 1, seq_len)"
      ],
      "execution_count": 106,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A7BYeBCNvi7n",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "445f7927-69ed-4d17-e689-d137695be38c"
      },
      "source": [
        "x = tf.constant([[8087, 307, 1, 204,    1, 7936,    8,   78, 5054,    2, 8088],\n",
        "                 [8087,   18,   10,   20,   32,  422,    2, 8088,    0,    0,    0], \n",
        "                 [8087,  153,   51,    2, 8088,    0,    0,    0,    0,    0,    0]])\n",
        "create_padding_mask(x)"
      ],
      "execution_count": 107,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(3, 11), dtype=int32, numpy=\n",
              "array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
              "       [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\n",
              "       [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]], dtype=int32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 107
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dVxS8OPI9uI0"
      },
      "source": [
        "def create_look_ahead_mask(size):\n",
        "  # tf.matrix_band_part(input, -1, 0) ==> Lower triangular part.\n",
        "  # https://www.tensorflow.org/api_docs/python/tf/linalg/band_part\n",
        "  \n",
        "  # print (tf.linalg.band_part(tf.ones((size, size), dtype=tf.int32), -1, 0)) # Uncomment later\n",
        " \n",
        "  # switch to the uppper triangular part\n",
        "  mask = 1 - tf.linalg.band_part(tf.ones((size, size), dtype=tf.int32), -1, 0)\n",
        "  return mask  # (seq_len, seq_len) tf.cast(x, tf.int32)"
      ],
      "execution_count": 131,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yxKGuXxaBeeE",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9849bf08-cdc8-4948-c6f7-a38473e15e35"
      },
      "source": [
        "x = tf.random.uniform((10, 4))\n",
        "print (x)\n",
        "temp = create_look_ahead_mask(x.shape[1])\n",
        "temp"
      ],
      "execution_count": 132,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[[0.1812272  0.46394026 0.63007116 0.7669574 ]\n",
            " [0.95795393 0.84440243 0.96232843 0.933537  ]\n",
            " [0.62386644 0.596776   0.72025025 0.04517007]\n",
            " [0.93267477 0.5643449  0.09496391 0.51921177]\n",
            " [0.618559   0.82524896 0.9240786  0.36227167]\n",
            " [0.14654422 0.36420035 0.27860534 0.5808252 ]\n",
            " [0.649814   0.3746668  0.61872554 0.21667993]\n",
            " [0.28737855 0.01136065 0.14243972 0.874161  ]\n",
            " [0.920475   0.54663754 0.11380601 0.50058675]\n",
            " [0.01067126 0.47789025 0.6782414  0.97009134]], shape=(10, 4), dtype=float32)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(4, 4), dtype=int32, numpy=\n",
              "array([[0, 1, 1, 1],\n",
              "       [0, 0, 1, 1],\n",
              "       [0, 0, 0, 1],\n",
              "       [0, 0, 0, 0]], dtype=int32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 132
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YgkDE7hzo8r5"
      },
      "source": [
        "## CREATE MASKS"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oxGJtoDuYIHL"
      },
      "source": [
        "Since the target sequences are padded, it is important to apply a padding mask when calculating the loss."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZOJUSB1T8GjM"
      },
      "source": [
        "def create_masks(inp, tar):\n",
        "  print (\"*************************In create_masks ****************************\")\n",
        "\n",
        "  # Encoder padding mask\n",
        "  # print (\"calling create_padding_mask(inp) for encoder \")\n",
        "  enc_padding_mask = create_padding_mask(inp)\n",
        "  # print (enc_padding_mask)\n",
        "  # print (enc_padding_mask.shape)\n",
        "\n",
        "  # Used in the 2nd attention block in the decoder.\n",
        "  # This padding mask is used to mask the encoder outputs.\n",
        "  # print (\"calling create_padding_mask(inp) for decoder \")\n",
        "  dec_padding_mask = create_padding_mask(inp)\n",
        "  # print (dec_padding_mask)\n",
        "  # print (dec_padding_mask.shape)\n",
        "\n",
        "  # Used in the 1st attention block in the decoder.\n",
        "  # It is used to pad and mask future tokens in the input received by \n",
        "  # the decoder.\n",
        "\n",
        "  # print (tf.shape(tar))  \n",
        "  # print (tf.shape(tar)[1])\n",
        "\n",
        "  # print (\"Calling create_look_ahead_mask(tf.shape(tar)[1])\")\n",
        "  look_ahead_mask = create_look_ahead_mask(tf.shape(tar)[1])\n",
        "  # print (tf.shape(look_ahead_mask))\n",
        "\n",
        "  # print (\"calling create_padding_mask(tar)\")\n",
        "  dec_target_padding_mask = create_padding_mask(tar)\n",
        "  look_ahead_mask = tf.constant(look_ahead_mask[:-1, :], dtype=tf.int32)  \n",
        "  # print (f\"\\n{tf.shape(look_ahead_mask)} and matrix is {look_ahead_mask}\")\n",
        "  combined_mask = tf.maximum(dec_target_padding_mask, look_ahead_mask)\n",
        "  \n",
        "  return enc_padding_mask, combined_mask, dec_padding_mask"
      ],
      "execution_count": 133,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LKpoA6q1sJFj"
      },
      "source": [
        "EPOCHS = 20"
      ],
      "execution_count": 134,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iJwmp9OE29oj"
      },
      "source": [
        "def train_step(inp, tar_inp_parm):\n",
        "  # print (f\"Shape passed in:\\n Input/Portugese shape: {inp.shape} \\n Output/English shape: {tar_inp_parm.shape}\" )\n",
        "  tar_inp = tar_inp_parm[:, :-1]\n",
        "  tar_real = tar_inp_parm[:, 1:]  \n",
        "  enc_padding_mask, combined_mask, dec_padding_mask = create_masks(inp, tar_inp)  \n",
        "  return enc_padding_mask, combined_mask, dec_padding_mask  "
      ],
      "execution_count": 135,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qM2PDWGDJ_8V"
      },
      "source": [
        "Portuguese is used as the input language and English is the target language."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5buvMlnvyrFm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5279c049-a998-4f58-9416-7a8216066b72"
      },
      "source": [
        "# https://www.tensorflow.org/api_docs/python/tf/data/Dataset\n",
        "# https://www.tensorflow.org/tutorials/load_data/numpy\n",
        "# https://stackoverflow.com/questions/62436302/accessing-data-in-tensorflow-prefetchdataset\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "print (f\"number of training examples: {len(train_examples)}\")\n",
        "print (f\"type of training examples: {type(train_examples)}\\n\") # <class 'tensorflow.python.data.ops.dataset_ops.PrefetchDataset'>\n",
        "lst = list(train_examples) # convert to list and then to tf.data.Dataset.from_tensor_slices\n",
        "tf_slices_train_examples = tf.data.Dataset.from_tensor_slices(lst)\n",
        "\n",
        "source_pt = []\n",
        "target_en = []\n",
        "result_pt = []\n",
        "result_en = []\n",
        "\n",
        "#Finally, access tensor elements in tensor of type tf.data.Dataset.from_tensor_slices\n",
        "for example_no, source_target in enumerate(tf_slices_train_examples.take(100)):\n",
        "\n",
        "  source = source_target[0].numpy().decode('UTF-8')\n",
        "  target = source_target[1].numpy().decode('UTF-8')\n",
        "  if (len(source) < MAX_LEN) and (len(target) < MAX_LEN):\n",
        "    source_pt.append(source)\n",
        "    target_en.append(target)\n",
        "    pt, en = encode(source, target)\n",
        "    result_pt.append(pt)\n",
        "    result_en.append(en)\n",
        "  else: \n",
        "    continue\n",
        "\n",
        "padded_inputs_pt, padded_inputs_en = pad_sequence(result_pt, result_en)"
      ],
      "execution_count": 136,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "number of training examples: 51785\n",
            "type of training examples: <class 'tensorflow.python.data.ops.dataset_ops.PrefetchDataset'>\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dP77HGyDWfsM",
        "outputId": "be257c9e-ece1-4fda-9b9a-dc63d6faaea3"
      },
      "source": [
        "print (\"\\nSource without padding or masked padded tokens:\\n\")\r\n",
        "tf.print (result_pt)\r\n",
        "print (\"\\nTarget without padding, masking padded tokens or look ahead masks:\\n\")\r\n",
        "tf.print (result_en)\r\n",
        "print (\"\\nSource with padding. No masked padded tokens:\\n\")\r\n",
        "print (tf.constant(padded_inputs_pt))\r\n",
        "print (\"\\nTarget with padding. No masked padded tokens or Look ahead masks:\\n\")\r\n",
        "print (tf.constant(padded_inputs_en))\r\n",
        "enc_padding_mask, combined_mask, dec_padding_mask = train_step(padded_inputs_pt, padded_inputs_en)\r\n",
        "print (\"*******************************************\")\r\n",
        "print (f\"Encoder Padding Mask: \\n {enc_padding_mask}\")\r\n",
        "print (f\"Combined Mask (Decoder Padding Mask + Look ahead mask): \\n {combined_mask}\")\r\n",
        "print (f\"Decoder Padding Mask (same as Encoder Padding Mask): \\n {dec_padding_mask}\")\r\n",
        "\r\n",
        "\"\"\"\r\n",
        "\r\n",
        "# Call Embedding Layer\r\n",
        "# Embed input vectors\r\n",
        "# Pass embedded vectors to Encoder to create encodings (keys and values)\r\n",
        "# Multiply enc_padding mask with a vector of -inf\r\n",
        "# Add resulting vector to embedded vectors\r\n",
        "\r\n",
        "\"\"\""
      ],
      "execution_count": 137,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Source without padding or masked padded tokens:\n",
            "\n",
            "[[8214, 3, 4675, 53, 9, 1537, 2, 8215],\n",
            " [8214, 27, 56, 5569, 8055, 2, 8215],\n",
            " [8214, 42, 13, 170, 2610, 2, 8215],\n",
            " [8214, 212, 2, 8215],\n",
            " [8214, 273, 1, 179, 7240, 8055, 2, 8215],\n",
            " [8214, 67, 107, 173, 8215],\n",
            " [8214, 3, 313, 95, 410, 2, 8215],\n",
            " [8214, 54, 53, 3, 69, 486, 2, 8215],\n",
            " [8214, 372, 2, 8215]]\n",
            "\n",
            "Target without padding, masking padded tokens or look ahead masks:\n",
            "\n",
            "[[8087, 3, 3978, 20, 2981, 2, 8088],\n",
            " [8087, 12, 20, 952, 7931, 2, 8088],\n",
            " [8087, 16, 97, 36, 1537, 2, 8088],\n",
            " [8087, 153, 51, 2, 8088],\n",
            " [8087, 307, 1, 204, 1, 7936, 8, 78, 5054, 2, 8088],\n",
            " [8087, 94, 136, 192, 8088],\n",
            " [8087, 3, 338, 20, 79, 2, 8088],\n",
            " [8087, 18, 10, 20, 32, 422, 2, 8088],\n",
            " [8087, 153, 51, 2, 8088]]\n",
            "\n",
            "Source with padding. No masked padded tokens:\n",
            "\n",
            "tf.Tensor(\n",
            "[[8214    3 4675   53    9 1537    2 8215]\n",
            " [8214   27   56 5569 8055    2 8215    0]\n",
            " [8214   42   13  170 2610    2 8215    0]\n",
            " [8214  212    2 8215    0    0    0    0]\n",
            " [8214  273    1  179 7240 8055    2 8215]\n",
            " [8214   67  107  173 8215    0    0    0]\n",
            " [8214    3  313   95  410    2 8215    0]\n",
            " [8214   54   53    3   69  486    2 8215]\n",
            " [8214  372    2 8215    0    0    0    0]], shape=(9, 8), dtype=int32)\n",
            "\n",
            "Target with padding. No masked padded tokens or Look ahead masks:\n",
            "\n",
            "tf.Tensor(\n",
            "[[8087    3 3978   20 2981    2 8088    0    0    0    0]\n",
            " [8087   12   20  952 7931    2 8088    0    0    0    0]\n",
            " [8087   16   97   36 1537    2 8088    0    0    0    0]\n",
            " [8087  153   51    2 8088    0    0    0    0    0    0]\n",
            " [8087  307    1  204    1 7936    8   78 5054    2 8088]\n",
            " [8087   94  136  192 8088    0    0    0    0    0    0]\n",
            " [8087    3  338   20   79    2 8088    0    0    0    0]\n",
            " [8087   18   10   20   32  422    2 8088    0    0    0]\n",
            " [8087  153   51    2 8088    0    0    0    0    0    0]], shape=(9, 11), dtype=int32)\n",
            "*************************In create_masks ****************************\n",
            "*******************************************\n",
            "Encoder Padding Mask: \n",
            " [[0 0 0 0 0 0 0 0]\n",
            " [0 0 0 0 0 0 0 1]\n",
            " [0 0 0 0 0 0 0 1]\n",
            " [0 0 0 0 1 1 1 1]\n",
            " [0 0 0 0 0 0 0 0]\n",
            " [0 0 0 0 0 1 1 1]\n",
            " [0 0 0 0 0 0 0 1]\n",
            " [0 0 0 0 0 0 0 0]\n",
            " [0 0 0 0 1 1 1 1]]\n",
            "Combined Mask (Decoder Padding Mask + Look ahead mask): \n",
            " [[0 1 1 1 1 1 1 1 1 1]\n",
            " [0 0 1 1 1 1 1 1 1 1]\n",
            " [0 0 0 1 1 1 1 1 1 1]\n",
            " [0 0 0 0 1 1 1 1 1 1]\n",
            " [0 0 0 0 0 1 1 1 1 1]\n",
            " [0 0 0 0 0 1 1 1 1 1]\n",
            " [0 0 0 0 0 0 0 1 1 1]\n",
            " [0 0 0 0 0 0 0 0 1 1]\n",
            " [0 0 0 0 0 1 1 1 1 1]]\n",
            "Decoder Padding Mask (same as Encoder Padding Mask): \n",
            " [[0 0 0 0 0 0 0 0]\n",
            " [0 0 0 0 0 0 0 1]\n",
            " [0 0 0 0 0 0 0 1]\n",
            " [0 0 0 0 1 1 1 1]\n",
            " [0 0 0 0 0 0 0 0]\n",
            " [0 0 0 0 0 1 1 1]\n",
            " [0 0 0 0 0 0 0 1]\n",
            " [0 0 0 0 0 0 0 0]\n",
            " [0 0 0 0 1 1 1 1]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "06oKj-XvfoxO",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "d0c0f5ec-2c8d-4f25-a743-e472e0cc433c"
      },
      "source": [
        "\"\"\"\r\n",
        "x = 1 - tf.linalg.band_part(tf.ones((10, 10)), -1, 0)\r\n",
        "x = tf.cast(x, tf.int32)\r\n",
        "print (x)\r\n",
        "y = tf.constant(x[:-1, :], dtype=tf.int32)\r\n",
        "print (y)\r\n",
        "dec_target_padding_mask = tf.constant([[0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\r\n",
        " [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 0, 0, 0, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]], dtype=tf.int32)\r\n",
        "combined_mask = tf.maximum(dec_target_padding_mask, y)\r\n",
        "print (combined_mask)\r\n",
        "\"\"\""
      ],
      "execution_count": 120,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'\\nx = 1 - tf.linalg.band_part(tf.ones((10, 10)), -1, 0)\\nx = tf.cast(x, tf.int32)\\nprint (x)\\ny = tf.constant(x[:-1, :], dtype=tf.int32)\\nprint (y)\\ndec_target_padding_mask = tf.constant([[0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\\n [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\\n [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\\n [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\\n [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\\n [0, 0, 0, 0, 0, 0, 0, 0, 1, 1],\\n [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]], dtype=tf.int32)\\ncombined_mask = tf.maximum(dec_target_padding_mask, y)\\nprint (combined_mask)\\n'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 120
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NSK1kwYcfla9"
      },
      "source": [
        "dec_target_padding_mask = tf.constant([[0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\r\n",
        " [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 0, 0, 1, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 0, 0, 0, 1, 1],\r\n",
        " [0, 0, 0, 0, 0, 1, 1, 1, 1, 1]], dtype=tf.int32)"
      ],
      "execution_count": 244,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jNVZX57pgiEI",
        "outputId": "96cf043e-d98c-4597-a3c9-7aa81d0a7369"
      },
      "source": [
        "combined_mask = tf.maximum(dec_target_padding_mask, y)\r\n",
        "print (combined_mask)"
      ],
      "execution_count": 245,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[[0 1 1 1 1 1 1 1 1 1]\n",
            " [0 0 1 1 1 1 1 1 1 1]\n",
            " [0 0 0 1 1 1 1 1 1 1]\n",
            " [0 0 0 0 1 1 1 1 1 1]\n",
            " [0 0 0 0 0 1 1 1 1 1]\n",
            " [0 0 0 0 0 1 1 1 1 1]\n",
            " [0 0 0 0 0 0 0 1 1 1]\n",
            " [0 0 0 0 0 0 0 0 1 1]\n",
            " [0 0 0 0 0 1 1 1 1 1]], shape=(9, 10), dtype=int32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XYvuWZV5iwTC"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}