nmt_with_attention.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of nmt_with_attention.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/UtpalMattoo/BERT-Transformers/blob/master/Copy_of_nmt_with_attention.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s_qNSzzyaCbD"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "jmjh290raIky"
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dYj59gynA8mH"
      },
      "source": [
        "[Utpal 09/30/2020]: \n",
        "\n",
        "1. You can continue \"Training\" after mounting/restoring checkpoints\n",
        "\n",
        "2. Just saving checkpoints for 1 EPOCH (see poor translation quality at end), required 93GB of storage \n",
        "\n",
        "3. Saving checkpoints for 2 Epochs required 185GB\n",
        "\n",
        "4. Saving checkpoints vs saving model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J0Qjg6vuaHNt"
      },
      "source": [
        "# Neural machine translation with attention"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AOpGoE2T-YXS"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/text/nmt_with_attention\">\n",
        "    <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />\n",
        "    View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/text/nmt_with_attention.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
        "    Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/text/nmt_with_attention.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
        "    View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/tutorials/text/nmt_with_attention.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CiwtNgENbx2g"
      },
      "source": [
        "This notebook trains a sequence to sequence (seq2seq) model for Spanish to English translation. This is an advanced example that assumes some knowledge of sequence to sequence models.\n",
        "\n",
        "After training the model in this notebook, you will be able to input a Spanish sentence, such as *\"¿todavia estan en casa?\"*, and return the English translation: *\"are you still at home?\"*\n",
        "\n",
        "The translation quality is reasonable for a toy example, but the generated attention plot is perhaps more interesting. This shows which parts of the input sentence has the model's attention while translating:\n",
        "\n",
        "<img src=\"https://tensorflow.org/images/spanish-english.png\" alt=\"spanish-english attention plot\">\n",
        "\n",
        "Note: This example takes approximately 10 minutes to run on a single P100 GPU."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tnxXKDjq3jEL"
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.ticker as ticker\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "import unicodedata\n",
        "import re\n",
        "import numpy as np\n",
        "import os\n",
        "import io\n",
        "import time"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wfodePkj3jEa"
      },
      "source": [
        "## Download and prepare the dataset\n",
        "\n",
        "We'll use a language dataset provided by http://www.manythings.org/anki/. This dataset contains language translation pairs in the format:\n",
        "\n",
        "```\n",
        "May I borrow this book?\t¿Puedo tomar prestado este libro?\n",
        "```\n",
        "\n",
        "There are a variety of languages available, but we'll use the English-Spanish dataset. For convenience, we've hosted a copy of this dataset on Google Cloud, but you can also download your own copy. After downloading the dataset, here are the steps we'll take to prepare the data:\n",
        "\n",
        "1. Add a *start* and *end* token to each sentence.\n",
        "2. Clean the sentences by removing special characters.\n",
        "3. Create a word index and reverse word index (dictionaries mapping from word → id and id → word).\n",
        "4. Pad each sentence to a maximum length."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kRVATYOgJs1b",
        "outputId": "176c4f0a-5f5a-4fa9-c5d3-60594992b839",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "source": [
        "# Download the file\n",
        "path_to_zip = tf.keras.utils.get_file(\n",
        "    'spa-eng.zip', origin='http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip',\n",
        "    extract=True)\n",
        "\n",
        "path_to_file = os.path.dirname(path_to_zip)+\"/spa-eng/spa.txt\""
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip\n",
            "2646016/2638744 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rd0jw-eC3jEh"
      },
      "source": [
        "# Converts the unicode file to ascii\n",
        "# https://medium.com/concerning-pharo/an-implementation-of-unicode-normalization-7c6719068f43\n",
        "\n",
        "# What is the best way to remove accents (normalize) in a Python unicode string?\n",
        "# https://stackoverflow.com/questions/517923/what-is-the-best-way-to-remove-accents-normalize-in-a-python-unicode-string\n",
        "\n",
        "# UAX #44: Unicode Character Database\n",
        "# http://www.unicode.org/reports/tr44/#GC_Values_Table\n",
        "\n",
        "\n",
        "def unicode_to_ascii_1(s):\n",
        "  return ''.join(c for c in unicodedata.normalize('NFD', s)) # \"\"\"if unicodedata.category(c) != 'Mn'\"\"\" \n",
        "    \n",
        "def unicode_to_ascii_2(s):\n",
        "  return ''.join(c for c in unicodedata.normalize('NFD', s)\n",
        "            if unicodedata.category(c) != 'Mn')\n",
        "\n",
        "def preprocess_sentence(w, x):\n",
        "  if (x == 1):\n",
        "     w = unicode_to_ascii_1(w.lower().strip())\n",
        "  else:\n",
        "     w = unicode_to_ascii_2(w.lower().strip())\n",
        "  \n",
        "  # creating a space between a word and the punctuation following it\n",
        "  # eg: \"he is a boy.\" => \"he is a boy .\"\n",
        "  # Reference:- https://stackoverflow.com/questions/3645931/python-padding-punctuation-with-white-spaces-keeping-punctuation\n",
        "  w = re.sub(r\"([?.!,¿])\", r\" \\1 \", w)\n",
        "  w = re.sub(r'[\" \"]+', \" \", w)\n",
        "\n",
        "  # replacing everything with space except (a-z, A-Z, \".\", \"?\", \"!\", \",\")\n",
        "  w = re.sub(r\"[^a-zA-Z?.!,¿]+\", \" \", w)\n",
        "\n",
        "  w = w.strip()\n",
        "\n",
        "  # adding a start and an end token to the sentence\n",
        "  # so that the model know when to start and stop predicting.\n",
        "  w = '<start> ' + w + ' <end>'\n",
        "  return w"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "opI2GzOt479E",
        "outputId": "d7bcdc54-af2e-49a1-c5eb-461aff78b414",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 108
        }
      },
      "source": [
        "en_sentence = u\"May I borrow this book?\"\n",
        "sp_sentence = u\"¿Puedo tomar prestado este libro?\"\n",
        "print(preprocess_sentence(en_sentence, 1))\n",
        "print(preprocess_sentence(sp_sentence, 1).encode('utf-8'))\n",
        "print (\"*******What if you don't check for: Mn,Nonspacing_Mark,a nonspacing combining mark (zero advance width)****\")\n",
        "print(preprocess_sentence(en_sentence, 2))\n",
        "print(preprocess_sentence(sp_sentence, 2).encode('utf-8'))"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<start> may i borrow this book ? <end>\n",
            "b'<start> \\xc2\\xbf puedo tomar prestado este libro ? <end>'\n",
            "*******What if you don't check for: Mn,Nonspacing_Mark,a nonspacing combining mark (zero advance width)****\n",
            "<start> may i borrow this book ? <end>\n",
            "b'<start> \\xc2\\xbf puedo tomar prestado este libro ? <end>'\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OHn4Dct23jEm"
      },
      "source": [
        "# 1. Remove the accents\n",
        "# 2. Clean the sentences\n",
        "# 3. Return word pairs in the format: [ENGLISH, SPANISH]\n",
        "def create_dataset(path, num_examples):\n",
        "  lines = io.open(path, encoding='UTF-8').read().strip().split('\\n')\n",
        "\n",
        "  # Option 1:\n",
        "  # word_pairs = [(preprocess_sentence(w, 2) for w in l.split('\\t')) for l in lines[:num_examples]]\n",
        "  \n",
        "  # Option 2:\n",
        "  word_pairs = []\n",
        "  for l in lines[:num_examples]:\n",
        "      word_pairs.append(preprocess_sentence(w, 2) for w in l.split('\\t'))\n",
        "  \n",
        "  # Option 3:\n",
        "  # word_pairs = []\n",
        "  # final = []\n",
        "  # for l in lines[:num_examples]:\n",
        "  #   i=0\n",
        "  #   for w in l.split('\\t'):\n",
        "  #     word_pairs.append(preprocess_sentence(w, 2))     \n",
        "  #   final.append(word_pairs)\n",
        "  # print (final[-1])\n",
        "  return zip(*word_pairs)"
      ],
      "execution_count": 117,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cTbSbBz55QtF",
        "outputId": "fd96cef8-a22d-4f9a-bea2-486633472526",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 73
        }
      },
      "source": [
        "\"\"\"\n",
        "Ex:\n",
        "  word_pairs = [('e1','s1'), ('e2', 's2'), ('e3', 's3'), ('e4', 's4')]\n",
        "  e, s = zip(*word_pairs)\n",
        "  print (e)\n",
        "  print (s)\n",
        "Output:\n",
        "('e1', 'e2', 'e3', 'e4')\n",
        "('s1', 's2', 's3', 's4')\n",
        "\"\"\"\n",
        "\n",
        "en, sp = create_dataset(path_to_file, None)\n",
        "print(en[-1])\n",
        "print(sp[-1])"
      ],
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<start> if you want to sound like a native speaker , you must be willing to practice saying the same sentence over and over in the same way that banjo players practice the same phrase over and over until they can play it correctly and at the desired tempo . <end>\n",
            "<start> si quieres sonar como un hablante nativo , debes estar dispuesto a practicar diciendo la misma frase una y otra vez de la misma manera en que un musico de banjo practica el mismo fraseo una y otra vez hasta que lo puedan tocar correctamente y en el tiempo esperado . <end>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bIOn8RCNDJXG"
      },
      "source": [
        "# https://stackoverflow.com/questions/51956000/what-does-keras-tokenizer-method-exactly-do\n",
        "# The same tokenizer can be used for any language. So, then how do similar words in different languages have similar vectors??\n",
        "\n",
        "def tokenize(lang):\n",
        "  lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(\n",
        "      filters='')\n",
        "  lang_tokenizer.fit_on_texts(lang)\n",
        "\n",
        "  tensor = lang_tokenizer.texts_to_sequences(lang)\n",
        "\n",
        "  tensor = tf.keras.preprocessing.sequence.pad_sequences(tensor,\n",
        "                                                         padding='post')\n",
        "\n",
        "  return tensor, lang_tokenizer"
      ],
      "execution_count": 121,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eAY9k49G3jE_"
      },
      "source": [
        "def load_dataset(path, num_examples=None):\n",
        "  # creating cleaned input, output pairs\n",
        "  targ_lang, inp_lang = create_dataset(path, num_examples)\n",
        "  print (targ_lang[0])\n",
        "  print (inp_lang[0])\n",
        "  input_tensor, inp_lang_tokenizer = tokenize(inp_lang)\n",
        "  print (input_tensor.shape)\n",
        "  print (input_tensor[0])\n",
        "  target_tensor, targ_lang_tokenizer = tokenize(targ_lang)\n",
        "  print (target_tensor.shape)\n",
        "  print (target_tensor[0])\n",
        "  return input_tensor, target_tensor, inp_lang_tokenizer, targ_lang_tokenizer"
      ],
      "execution_count": 122,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GOi42V79Ydlr"
      },
      "source": [
        "### Limit the size of the dataset to experiment faster (optional)\n",
        "\n",
        "Training on the complete dataset of >100,000 sentences will take a long time. To train faster, we can limit the size of the dataset to 30,000 sentences (of course, translation quality degrades with less data):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cnxC7q-j3jFD",
        "outputId": "ceff2c7c-b725-4858-88c9-655fed88ba51",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 199
        }
      },
      "source": [
        "# Try experimenting with the size of that dataset\n",
        "num_examples = 30000\n",
        "input_tensor, target_tensor, inp_lang, targ_lang = load_dataset(path_to_file, num_examples)\n",
        "\n",
        "# Calculate max_length of the target tensors\n",
        "max_length_targ, max_length_inp = target_tensor.shape[1], input_tensor.shape[1]\n",
        "print (target_tensor.shape)\n",
        "print (input_tensor.shape)\n",
        "print (max_length_inp)\n",
        "print (max_length_targ)"
      ],
      "execution_count": 123,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<start> go . <end>\n",
            "<start> ve . <end>\n",
            "(30000, 16)\n",
            "[  1 135   3   2   0   0   0   0   0   0   0   0   0   0   0   0]\n",
            "(30000, 11)\n",
            "[ 1 36  3  2  0  0  0  0  0  0  0]\n",
            "(30000, 11)\n",
            "(30000, 16)\n",
            "16\n",
            "11\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4QILQkOs3jFG",
        "outputId": "a7d0f289-d3c0-4ac8-9900-dfa06fe97cd2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        }
      },
      "source": [
        "# Creating training and validation sets using an 80-20 split\n",
        "input_tensor_train, input_tensor_val, target_tensor_train, target_tensor_val = train_test_split(input_tensor, target_tensor, test_size=0.2)\n",
        "print (input_tensor_train.shape)\n",
        "print (target_tensor_train.shape)\n",
        "\n",
        "# # Show length\n",
        "print(len(input_tensor_train), len(target_tensor_train), len(input_tensor_val), len(target_tensor_val))"
      ],
      "execution_count": 124,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(24000, 16)\n",
            "(24000, 11)\n",
            "24000 24000 6000 6000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lJPmLZGMeD5q"
      },
      "source": [
        "def convert(lang, tensor):\n",
        "  for t in tensor:\n",
        "    if t!=0:\n",
        "      print (\"%d ----> %s\" % (t, lang.index_word[t]))"
      ],
      "execution_count": 125,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VXukARTDd7MT",
        "outputId": "ba4df9ee-85a1-4845-a9d2-3a860d267c92",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 290
        }
      },
      "source": [
        "print (\"Input Language; index to word mapping\")\n",
        "convert(inp_lang, input_tensor_train[1])\n",
        "print ()\n",
        "print (\"Target Language; index to word mapping\")\n",
        "convert(targ_lang, target_tensor_train[1])"
      ],
      "execution_count": 126,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input Language; index to word mapping\n",
            "1 ----> <start>\n",
            "26 ----> yo\n",
            "234 ----> conozco\n",
            "1360 ----> personas\n",
            "3 ----> .\n",
            "2 ----> <end>\n",
            "\n",
            "Target Language; index to word mapping\n",
            "1 ----> <start>\n",
            "4 ----> i\n",
            "43 ----> know\n",
            "329 ----> people\n",
            "3 ----> .\n",
            "2 ----> <end>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rgCLkfv5uO3d"
      },
      "source": [
        "### Create a tf.data dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TqHsArVZ3jFS",
        "outputId": "ec26cc7e-b6a3-4024-9f12-096da694f7f1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "BUFFER_SIZE = len(input_tensor_train)\n",
        "BATCH_SIZE = 64\n",
        "steps_per_epoch = len(input_tensor_train)//BATCH_SIZE\n",
        "print (len(input_tensor_train))\n",
        "print (steps_per_epoch)\n",
        "embedding_dim = 256\n",
        "units = 1024\n",
        "vocab_inp_size = len(inp_lang.word_index)+1\n",
        "print (\"vocab_inp_size = {} \".format(vocab_inp_size))\n",
        "vocab_tar_size = len(targ_lang.word_index)+1\n",
        "print (\"vocab_tar_size = {} \".format(vocab_tar_size))\n",
        "\n",
        "# https://www.tensorflow.org/guide/data#dataset_structure\n",
        "# https://www.tensorflow.org/api_docs/python/tf/data/Dataset\n",
        "dataset = tf.data.Dataset.from_tensor_slices((input_tensor_train, target_tensor_train)).shuffle(BUFFER_SIZE)\n",
        "print (dataset.element_spec)\n",
        "print (\"***********\")\n",
        "# https://www.tensorflow.org/api_docs/python/tf/data/Dataset#batch\n",
        "dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)\n",
        "\n",
        "# Some test code to understand class tf.data.Dataset\n",
        "\n",
        "UnderstandDatasetTakeEtc = True\n",
        "EPOCHS = 10\n",
        "for epoch in range(EPOCHS):\n",
        "    start = time.time()\n",
        "    print (dataset.take(steps_per_epoch))\n",
        "    print (enumerate(dataset.take(steps_per_epoch)))   \n",
        "    for (batch) in enumerate(dataset.take(steps_per_epoch)):\n",
        "       if ((epoch==0) and (UnderstandDatasetTakeEtc)): \n",
        "            print (\"***********\")\n",
        "            print (batch) #this will print the curent batch (batch 0) but also the input and \n",
        "                          #the target tensor - the one returned variable (batch) will \n",
        "                          #also include the other contents and not just the batch number\n",
        "            UnderstandDatasetTakeEtc = False \n",
        "            \n",
        "epoch=0\n",
        "for (batch, (inp, targ)) in enumerate(dataset.take(steps_per_epoch)):\n",
        "     if (epoch==0) and (batch % 125 == 0):\n",
        "        print ('Epoch is = {}\\n Steps per epoch is = {}\\n Batch is = {}\\n InputShape = {}\\n TargShape = {}\\n'\n",
        "                                                     .format(epoch,\n",
        "                                                             steps_per_epoch,\n",
        "                                                             batch,\n",
        "                                                             inp.get_shape().as_list(),\n",
        "                                                             targ.get_shape().as_list()))\n",
        "        "
      ],
      "execution_count": 133,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "24000\n",
            "375\n",
            "vocab_inp_size = 9414 \n",
            "vocab_tar_size = 4935 \n",
            "(TensorSpec(shape=(16,), dtype=tf.int32, name=None), TensorSpec(shape=(11,), dtype=tf.int32, name=None))\n",
            "***********\n",
            "<TakeDataset shapes: ((64, 16), (64, 11)), types: (tf.int32, tf.int32)>\n",
            "<enumerate object at 0x7fa2fadb4438>\n",
            "***********\n",
            "(0, (<tf.Tensor: shape=(64, 16), dtype=int32, numpy=\n",
            "array([[  1,   6,  11, ...,   0,   0,   0],\n",
            "       [  1,   6,  46, ...,   0,   0,   0],\n",
            "       [  1, 586,  18, ...,   0,   0,   0],\n",
            "       ...,\n",
            "       [  1,  63, 430, ...,   0,   0,   0],\n",
            "       [  1, 340,  32, ...,   0,   0,   0],\n",
            "       [  1,  78,  63, ...,   0,   0,   0]], dtype=int32)>, <tf.Tensor: shape=(64, 11), dtype=int32, numpy=\n",
            "array([[   1,   52,   26,  110,   11,  467,    7,    2,    0,    0,    0],\n",
            "       [   1,   60,   11,   44,   31,  812,    7,    2,    0,    0,    0],\n",
            "       [   1,  116,   17,   32,    6,   63,    3,    2,    0,    0,    0],\n",
            "       [   1,    5,  274, 2823,    3,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,   21,  226,  760,  138,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,    4,   25,   53,   91,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,    4, 1211,  676,    3,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,   53,  197,    3,    2,    0,    0,    0,    0,    0,    0],\n",
            "       [   1,   42,    6,  302,    5,    7,    2,    0,    0,    0,    0],\n",
            "       [   1,    4,  239,    6,   29,  214,    3,    2,    0,    0,    0],\n",
            "       [   1,    4,   63,  140,  655,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,    4,   26, 2492,    3,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,  111,   15,   13,  151,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,    5, 1442,  126,    3,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,  111, 1413,    3,    2,    0,    0,    0,    0,    0,    0],\n",
            "       [   1,    4,   35,   31,  247,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,   14,  240,   13,  428,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,   28,  785,    5,    3,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,   46,   11,   40,  549,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,   27,   11,    9, 4182,  350,    3,    2,    0,    0,    0],\n",
            "       [   1,    4,   29,   15,  111,  718,    3,    2,    0,    0,    0],\n",
            "       [   1,    4,   18,   34,  558,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,   27,  145,   15,   21, 2699,    3,    2,    0,    0,    0],\n",
            "       [   1,   22,    6,   47,   15,   72,   10,    7,    2,    0,    0],\n",
            "       [   1,   25,    4,  557,    7,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,   13,  104,  137,  232,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,  459,   17,   31,  713,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,   52,   22,    4,  185,    9, 1551,    7,    2,    0,    0],\n",
            "       [   1,   30,   12, 1722,   17,   37,    2,    0,    0,    0,    0],\n",
            "       [   1,   71,   11,   21,  471,    7,    2,    0,    0,    0,    0],\n",
            "       [   1,    4,   76,   79,    9,   33,  139,    3,    2,    0,    0],\n",
            "       [   1,   30,   12,  900,   37,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,    4,  164,   12,  639,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,   16,   43,   60,   42,   19,    3,    2,    0,    0,    0],\n",
            "       [   1,   28,   23,   34,  198,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,   10,   11,   34,    9, 2482,    3,    2,    0,    0,    0],\n",
            "       [   1,   24,    6,   33,   57,    7,    2,    0,    0,    0,    0],\n",
            "       [   1,   27, 1147,   48,  109,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,    4, 1036,    9,  146,  168,    3,    2,    0,    0,    0],\n",
            "       [   1,    6,  268,   78,  224,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,    5,   76,  142,   13,  322,    3,    2,    0,    0,    0],\n",
            "       [   1,  441,    5,   26,  377,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,   27,    8, 1427,    3,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,   19,    8,    9, 1513,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,   10,   11,   69,   15,   40,   89,    3,    2,    0,    0],\n",
            "       [   1,   22,    6,  103,   77,   17,    7,    2,    0,    0,    0],\n",
            "       [   1,   21,  146,  372, 2640,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,    5,  305,   55,    9,  578,    3,    2,    0,    0,    0],\n",
            "       [   1,    4,   62,  405,    3,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,   40,  630,    3,    2,    0,    0,    0,    0,    0,    0],\n",
            "       [   1,    4,   43,  139,    3,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,   28,  213, 1231,    3,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,   10,   26,   48,  343,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,    4,   70,  173,   50,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,    4,   30,   12,   64,   10,    3,    2,    0,    0,    0],\n",
            "       [   1,   28,  896,    3,    2,    0,    0,    0,    0,    0,    0],\n",
            "       [   1,   10,  176,   12, 1115,    3,    2,    0,    0,    0,    0],\n",
            "       [   1,   22,    6,   35,   15,  295,    7,    2,    0,    0,    0],\n",
            "       [   1,  122, 1993,   37,    2,    0,    0,    0,    0,    0,    0],\n",
            "       [   1,    4,   18,  129,   15,  283,    3,    2,    0,    0,    0],\n",
            "       [   1,  177,   19,    3,    2,    0,    0,    0,    0,    0,    0],\n",
            "       [   1,   16,   77, 3831,    3,    2,    0,    0,    0,    0,    0],\n",
            "       [   1,  252,   49,    4,   64,   13,  259,    3,    2,    0,    0],\n",
            "       [   1,   66,   84,   25,  256,   81,    3,    2,    0,    0,    0]],\n",
            "      dtype=int32)>))\n",
            "<TakeDataset shapes: ((64, 16), (64, 11)), types: (tf.int32, tf.int32)>\n",
            "<enumerate object at 0x7fa2fadb4318>\n",
            "<TakeDataset shapes: ((64, 16), (64, 11)), types: (tf.int32, tf.int32)>\n",
            "<enumerate object at 0x7fa2fadb4558>\n",
            "<TakeDataset shapes: ((64, 16), (64, 11)), types: (tf.int32, tf.int32)>\n",
            "<enumerate object at 0x7fa2fadb4438>\n",
            "<TakeDataset shapes: ((64, 16), (64, 11)), types: (tf.int32, tf.int32)>\n",
            "<enumerate object at 0x7fa2fadb4318>\n",
            "<TakeDataset shapes: ((64, 16), (64, 11)), types: (tf.int32, tf.int32)>\n",
            "<enumerate object at 0x7fa2fadb4558>\n",
            "<TakeDataset shapes: ((64, 16), (64, 11)), types: (tf.int32, tf.int32)>\n",
            "<enumerate object at 0x7fa2fadb4438>\n",
            "<TakeDataset shapes: ((64, 16), (64, 11)), types: (tf.int32, tf.int32)>\n",
            "<enumerate object at 0x7fa2fadb4318>\n",
            "<TakeDataset shapes: ((64, 16), (64, 11)), types: (tf.int32, tf.int32)>\n",
            "<enumerate object at 0x7fa2fadb4558>\n",
            "<TakeDataset shapes: ((64, 16), (64, 11)), types: (tf.int32, tf.int32)>\n",
            "<enumerate object at 0x7fa2fadb4438>\n",
            "Epoch is = 0\n",
            " Steps per epoch is = 375\n",
            " Batch is = 0\n",
            " InputShape = [64, 16]\n",
            " TargShape = [64, 11]\n",
            "\n",
            "Epoch is = 0\n",
            " Steps per epoch is = 375\n",
            " Batch is = 125\n",
            " InputShape = [64, 16]\n",
            " TargShape = [64, 11]\n",
            "\n",
            "Epoch is = 0\n",
            " Steps per epoch is = 375\n",
            " Batch is = 250\n",
            " InputShape = [64, 16]\n",
            " TargShape = [64, 11]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qc6-NK1GtWQt",
        "outputId": "a7135542-a87e-45a2-ce9a-549aa82e6bbb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "example_input_batch, example_target_batch = next(iter(dataset))\n",
        "example_input_batch.shape, example_target_batch.shape"
      ],
      "execution_count": 135,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([64, 16]), TensorShape([64, 11]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 135
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TNfHIF71ulLu"
      },
      "source": [
        "## Write the encoder and decoder model\n",
        "\n",
        "Implement an encoder-decoder model with attention which you can read about in the TensorFlow [Neural Machine Translation (seq2seq) tutorial](https://github.com/tensorflow/nmt). This example uses a more recent set of APIs. This notebook implements the [attention equations](https://github.com/tensorflow/nmt#background-on-the-attention-mechanism) from the seq2seq tutorial. The following diagram shows that each input words is assigned a weight by the attention mechanism which is then used by the decoder to predict the next word in the sentence. The below picture and formulas are an example of attention mechanism from [Luong's paper](https://arxiv.org/abs/1508.04025v5). \n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_mechanism.jpg\" width=\"500\" alt=\"attention mechanism\">\n",
        "\n",
        "The input is put through an encoder model which gives us the encoder output of shape *(batch_size, max_length, hidden_size)* and the encoder hidden state of shape *(batch_size, hidden_size)*.\n",
        "\n",
        "Here are the equations that are implemented:\n",
        "\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_equation_0.jpg\" alt=\"attention equation 0\" width=\"800\">\n",
        "<img src=\"https://www.tensorflow.org/images/seq2seq/attention_equation_1.jpg\" alt=\"attention equation 1\" width=\"800\">\n",
        "\n",
        "This tutorial uses [Bahdanau attention](https://arxiv.org/pdf/1409.0473.pdf) for the encoder. Let's decide on notation before writing the simplified form:\n",
        "\n",
        "* FC = Fully connected (dense) layer\n",
        "* EO = Encoder output\n",
        "* H = hidden state\n",
        "* X = input to the decoder\n",
        "\n",
        "And the pseudo-code:\n",
        "\n",
        "* `score = FC(tanh(FC(EO) + FC(H)))`\n",
        "* `attention weights = softmax(score, axis = 1)`. Softmax by default is applied on the last axis but here we want to apply it on the *1st axis*, since the shape of score is *(batch_size, max_length, hidden_size)*. `Max_length` is the length of our input. Since we are trying to assign a weight to each input, softmax should be applied on that axis.\n",
        "* `context vector = sum(attention weights * EO, axis = 1)`. Same reason as above for choosing axis as 1.\n",
        "* `embedding output` = The input to the decoder X is passed through an embedding layer.\n",
        "* `merged vector = concat(embedding output, context vector)`\n",
        "* This merged vector is then given to the GRU\n",
        "\n",
        "The shapes of all the vectors at each step have been specified in the comments in the code:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nZ2rI24i3jFg"
      },
      "source": [
        "class Encoder(tf.keras.Model):\n",
        "  def __init__(self, vocab_size, embedding_dim, enc_units, batch_sz):\n",
        "    super(Encoder, self).__init__()\n",
        "    self.batch_sz = batch_sz\n",
        "    self.enc_units = enc_units\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.enc_units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "    print (\"self.batch_sz = {}, self.enc_units = {}, \\\n",
        "            self.embedding.input_dim = {}, self.embedding.output_dim = {},\\\n",
        "            vocab_size = {}, embedding_dim = {}\".format(self.batch_sz, \\\n",
        "                                                self.enc_units, \\\n",
        "                                                self.embedding.input_dim,\\\n",
        "                                                self.embedding.output_dim,\\\n",
        "                                                vocab_size,\n",
        "                                                embedding_dim))\n",
        "                                \n",
        "  def call(self, x, hidden):\n",
        "    # print (x.shape)\n",
        "    x = self.embedding(x)\n",
        "    # print (x.shape)\n",
        "    # print (\"hidden shape is = {}\". format(hidden.shape))\n",
        "    output, state = self.gru(x, initial_state = hidden)\n",
        "    return output, state\n",
        "\n",
        "  def initialize_hidden_state(self):\n",
        "    return tf.zeros((self.batch_sz, self.enc_units))"
      ],
      "execution_count": 136,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "60gSVh05Jl6l",
        "outputId": "8f1707a3-9d66-467d-8ffd-9afbd92e3ad7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 110
        }
      },
      "source": [
        "encoder = Encoder(vocab_inp_size, embedding_dim, units, BATCH_SIZE)\n",
        "\n",
        "# sample input\n",
        "sample_hidden = encoder.initialize_hidden_state()\n",
        "print (sample_hidden.shape)\n",
        "sample_output, sample_hidden = encoder(example_input_batch, sample_hidden)\n",
        "print ('Encoder output shape: (batch size, sequence length, units) {}'.format(sample_output.shape))\n",
        "print ('Encoder Hidden state shape: (batch size, units) {}'.format(sample_hidden.shape))"
      ],
      "execution_count": 139,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "self.batch_sz = 64, self.enc_units = 1024,             self.embedding.input_dim = 9414, self.embedding.output_dim = 256,            vocab_size = 9414, embedding_dim = 256\n",
            "(64, 1024)\n",
            "Encoder output shape: (batch size, sequence length, units) (64, 16, 1024)\n",
            "Encoder Hidden state shape: (batch size, units) (64, 1024)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "umohpBN2OM94"
      },
      "source": [
        "# http://juditacs.github.io/2018/12/27/masked-attention.html\n",
        "# https://github.com/abisee/pointer-generator/issues/88\n",
        "# https://medium.com/analytics-vidhya/masking-in-transformers-self-attention-mechanism-bad3c9ec235c\n",
        "\n",
        "class BahdanauAttention(tf.keras.layers.Layer):\n",
        "  def __init__(self, units):\n",
        "    super(BahdanauAttention, self).__init__()\n",
        "    self.W1 = tf.keras.layers.Dense(units)\n",
        "    self.W2 = tf.keras.layers.Dense(units)\n",
        "    self.V = tf.keras.layers.Dense(1)\n",
        "\n",
        "  def call(self, query, values):\n",
        "    \n",
        "    # query hidden state shape == (batch_size, hidden size)\n",
        "    # query_with_time_axis shape == (batch_size, 1, hidden size)\n",
        "    # values shape == (batch_size, max_len, hidden size)\n",
        "    # we are doing this to broadcast addition along the time axis to calculate the score\n",
        "    # print (\"in call function\")\n",
        "    # print (query.shape)    \n",
        "    query_with_time_axis = tf.expand_dims(query, 1)\n",
        "\n",
        "    # score shape == (batch_size, max_length, 1)\n",
        "    # we get 1 at the last axis because we are applying score to self.V\n",
        "    # the shape of the tensor before applying self.V is (batch_size, max_length, units)\n",
        "    # print (query_with_time_axis.shape)\n",
        "    # print (values.shape)\n",
        "    score = self.V(tf.nn.tanh(\n",
        "        self.W1(query_with_time_axis) + self.W2(values)))\n",
        "    \n",
        "    # print (score.shape)\n",
        "    # attention_weights shape == (batch_size, max_length, 1)\n",
        "    attention_weights = tf.nn.softmax(score, axis=1)\n",
        "\n",
        "    # context_vector shape after sum == (batch_size, hidden_size)\n",
        "    context_vector = attention_weights * values\n",
        "    context_vector = tf.reduce_sum(context_vector, axis=1)\n",
        "\n",
        "    return context_vector, attention_weights"
      ],
      "execution_count": 142,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k534zTHiDjQU",
        "outputId": "d42cafc8-cbfa-4feb-facc-1f8e29f7f921",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "source": [
        "attention_layer = BahdanauAttention(10)\n",
        "attention_result, attention_weights = attention_layer(sample_hidden, sample_output)\n",
        "\n",
        "print(\"Attention result shape: (batch size, units) {}\".format(attention_result.shape))\n",
        "print(\"Attention weights shape: (batch_size, sequence_length, 1) {}\".format(attention_weights.shape))"
      ],
      "execution_count": 143,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Attention result shape: (batch size, units) (64, 1024)\n",
            "Attention weights shape: (batch_size, sequence_length, 1) (64, 16, 1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yJ_B3mhW3jFk"
      },
      "source": [
        "class Decoder(tf.keras.Model):\n",
        "  def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz):\n",
        "    super(Decoder, self).__init__()\n",
        "    self.batch_sz = batch_sz\n",
        "    self.dec_units = dec_units\n",
        "    self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
        "    self.gru = tf.keras.layers.GRU(self.dec_units,\n",
        "                                   return_sequences=True,\n",
        "                                   return_state=True,\n",
        "                                   recurrent_initializer='glorot_uniform')\n",
        "    self.fc = tf.keras.layers.Dense(vocab_size)\n",
        "\n",
        "    # used for attention\n",
        "    self.attention = BahdanauAttention(self.dec_units)\n",
        "\n",
        "  def call(self, x, hidden, enc_output):\n",
        "    # enc_output shape == (batch_size, max_length, hidden_size)\n",
        "    context_vector, attention_weights = self.attention(hidden, enc_output)\n",
        "\n",
        "    # x shape after passing through embedding == (batch_size, 1, embedding_dim)\n",
        "    x = self.embedding(x)\n",
        "\n",
        "    # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)\n",
        "    x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)\n",
        "    \n",
        "    # passing the concatenated vector to the GRU\n",
        "    output, state = self.gru(x)\n",
        "    print (output.shape)\n",
        "    print (output.shape[2])\n",
        "    # output shape == (batch_size * 1, hidden_size)\n",
        "    output = tf.reshape(output, (-1, output.shape[2]))\n",
        "    print (output.shape)\n",
        "    # output shape == (batch_size, vocab)\n",
        "    x = self.fc(output)  # Note: see how a hidden shape of (1024, ) is converted into an vocab embedding of shape (4935, )\n",
        "    print (x.shape)\n",
        "    return x, state, attention_weights"
      ],
      "execution_count": 192,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P5UY8wko3jFp",
        "outputId": "658320ec-13d3-4a37-c98e-51c420614390",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 108
        }
      },
      "source": [
        "decoder = Decoder(vocab_tar_size, embedding_dim, units, BATCH_SIZE)\n",
        "\n",
        "sample_decoder_output, _, _ = decoder(tf.random.uniform((BATCH_SIZE, 1)),\n",
        "                                      sample_hidden, sample_output)\n",
        "\n",
        "print ('Decoder output shape: (batch_size, vocab size) {}'.format(sample_decoder_output.shape))"
      ],
      "execution_count": 193,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "(64, 1, 1024)\n",
            "1024\n",
            "(64, 1024)\n",
            "(64, 4935)\n",
            "Decoder output shape: (batch_size, vocab size) (64, 4935)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ch_71VbIRfK"
      },
      "source": [
        "## Define the optimizer and the loss function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WmTHr5iV3jFr"
      },
      "source": [
        "optimizer = tf.keras.optimizers.Adam()\n",
        "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
        "    from_logits=True, reduction='none')\n",
        "\n",
        "def loss_function(real, pred):\n",
        "  mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
        "  loss_ = loss_object(real, pred)\n",
        "\n",
        "  mask = tf.cast(mask, dtype=loss_.dtype)\n",
        "  loss_ *= mask\n",
        "\n",
        "  return tf.reduce_mean(loss_)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DMVWzzsfNl4e"
      },
      "source": [
        "## Checkpoints (Object-based saving)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zj8bXQTgNwrF",
        "outputId": "8d513590-6a59-4ae4-bb14-fa951a9d7efd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "from google.colab import drive\n",
        "checkpoint_dir = '/content/gdrive'\n",
        "drive.mount (checkpoint_dir, force_remount=True)\n",
        "checkpoint_prefix = os.path.join(checkpoint_dir, 'My Drive/Training/ckpt_nmt')\n",
        "print (checkpoint_prefix)\n",
        "\n",
        "checkpoint = tf.train.Checkpoint(optimizer=optimizer,\n",
        "                                  encoder=encoder,\n",
        "                                  decoder=decoder)\n",
        "\n",
        "# checkpoint_dir = './training_checkpoints'\n",
        "# checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt\")\n",
        "# checkpoint = tf.train.Checkpoint(optimizer=optimizer,\n",
        "#                                  encoder=encoder,\n",
        "#                                  decoder=decoder)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/gdrive\n",
            "/content/gdrive/My Drive/Training/ckpt_nmt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pp8mhy4TnRbI"
      },
      "source": [
        "# from google.colab import drive\n",
        "# drive.flush_and_unmount()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hpObfY22IddU"
      },
      "source": [
        "## Training\n",
        "\n",
        "1. Pass the *input* through the *encoder* which return *encoder output* and the *encoder hidden state*.\n",
        "2. The encoder output, encoder hidden state and the decoder input (which is the *start token*) is passed to the decoder.\n",
        "3. The decoder returns the *predictions* and the *decoder hidden state*.\n",
        "4. The decoder hidden state is then passed back into the model and the predictions are used to calculate the loss.\n",
        "5. Use *teacher forcing* to decide the next input to the decoder.\n",
        "6. *Teacher forcing* is the technique where the *target word* is passed as the *next input* to the decoder.\n",
        "7. The final step is to calculate the gradients and apply it to the optimizer and backpropagate."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sC9ArXSsVfqn"
      },
      "source": [
        "# https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3\n",
        "# https://machinelearningmastery.com/how-does-attention-work-in-encoder-decoder-recurrent-neural-networks/\n",
        "# https://stats.stackexchange.com/questions/241985/understanding-lstm-units-vs-cells\n",
        "# https://stats.stackexchange.com/questions/179101/structure-of-recurrent-neural-network-lstm-gru/275518#275518\n",
        "# https://colab.research.google.com/drive/1rPk3ohrmVclqhH7uQ7qys4oznDdAhpzF#scrollTo=jUTNr15JBkSG\n",
        "\n",
        "\n",
        "@tf.function\n",
        "def train_step(inp, targ, enc_hidden):\n",
        "  loss = 0\n",
        "\n",
        "  with tf.GradientTape() as tape:\n",
        "    enc_output, enc_hidden = encoder(inp, enc_hidden)\n",
        "\n",
        "    dec_hidden = enc_hidden\n",
        "\n",
        "    dec_input = tf.expand_dims([targ_lang.word_index['<start>']] * BATCH_SIZE, 1)\n",
        "\n",
        "    # Teacher forcing - feeding the target as the next input\n",
        "    for t in range(1, targ.shape[1]):\n",
        "      # passing enc_output to the decoder\n",
        "      predictions, dec_hidden, _ = decoder(dec_input, dec_hidden, enc_output)\n",
        "\n",
        "      loss += loss_function(targ[:, t], predictions)\n",
        "\n",
        "      # using teacher forcing\n",
        "      dec_input = tf.expand_dims(targ[:, t], 1)\n",
        "\n",
        "  batch_loss = (loss / int(targ.shape[1]))\n",
        "\n",
        "  variables = encoder.trainable_variables + decoder.trainable_variables\n",
        "\n",
        "  gradients = tape.gradient(loss, variables)\n",
        "\n",
        "  optimizer.apply_gradients(zip(gradients, variables))\n",
        "\n",
        "  return batch_loss"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ihxq1B1nmKkC",
        "outputId": "c228f57d-0874-453e-c06a-02db82b1e883",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "EPOCHS = 2\n",
        "print_msg = True\n",
        "\n",
        "for epoch in range(EPOCHS):\n",
        "  start = time.time()\n",
        "\n",
        "  enc_hidden = encoder.initialize_hidden_state()\n",
        "  print (enc_hidden)\n",
        "  total_loss = 0\n",
        "\n",
        "# https://www.tensorflow.org/api_docs/python/tf/data/Dataset#take\n",
        "  for (batch, (inp, targ)) in enumerate(dataset.take(steps_per_epoch)):\n",
        "    if ((epoch == 0) and (batch % 125 == 0) and (print_msg)):\n",
        "        x, y = (inp, targ)\n",
        "        print (\"*************************************\\n\")\n",
        "        print ('Epoch is = {}\\n Steps per epoch is = {}\\n Batch is = {}\\n InputShape = {}\\n TargShape = {}\\n  Input batch size = {}\\n \\\n",
        "                Target batch size = {} \\n Input length is = {} \\n Target length is = {} \\n Input is = {} \\n Target is = {} \\n'\n",
        "                                                     .format(epoch,\n",
        "                                                             steps_per_epoch,\n",
        "                                                             batch,\n",
        "                                                             inp.get_shape().as_list(),\n",
        "                                                             targ.get_shape().as_list(),\n",
        "                                                             len(x),\n",
        "                                                             len(y),\n",
        "                                                             len(x[0]),\n",
        "                                                             len(y[0]),\n",
        "                                                             x[0],                                                             \n",
        "                                                             y[0]))\n",
        "    \n",
        "    batch_loss = train_step(inp, targ, enc_hidden)\n",
        "    total_loss += batch_loss\n",
        "\n",
        "    if batch % 100 == 0:\n",
        "      print('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1,\n",
        "                                                   batch,\n",
        "                                                   batch_loss.numpy()))\n",
        "    \n",
        "  # saving (checkpoint) the model every 2 epochs\n",
        "  # if (epoch + 1) % 2 == 0:\n",
        "    checkpoint.save(file_prefix = checkpoint_prefix)\n",
        "\n",
        "  print('Epoch {} Loss {:.4f}'.format(epoch + 1,\n",
        "                                      total_loss / steps_per_epoch))\n",
        "  print('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))\n",
        "  "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[[0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " ...\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]], shape=(64, 1024), dtype=float32)\n",
            "*************************************\n",
            "\n",
            "Epoch is = 0\n",
            " Steps per epoch is = 375\n",
            " Batch is = 0\n",
            " InputShape = [64, 16]\n",
            " TargShape = [64, 11]\n",
            "  Input batch size = 64\n",
            "                 Target batch size = 64 \n",
            " Input length is = 16 \n",
            " Target length is = 11 \n",
            " Input is = [   1   90  733 1199    3    2    0    0    0    0    0    0    0    0\n",
            "    0    0] \n",
            " Target is = [   1    4  106 1022    3    2    0    0    0    0    0] \n",
            "\n",
            "Epoch 1 Batch 0 Loss 2.2216\n",
            "Epoch 1 Batch 100 Loss 1.8491\n",
            "*************************************\n",
            "\n",
            "Epoch is = 0\n",
            " Steps per epoch is = 375\n",
            " Batch is = 125\n",
            " InputShape = [64, 16]\n",
            " TargShape = [64, 11]\n",
            "  Input batch size = 64\n",
            "                 Target batch size = 64 \n",
            " Input length is = 16 \n",
            " Target length is = 11 \n",
            " Input is = [   1   18 3451    3    2    0    0    0    0    0    0    0    0    0\n",
            "    0    0] \n",
            " Target is = [  1  16  23 718   3   2   0   0   0   0   0] \n",
            "\n",
            "Epoch 1 Batch 200 Loss 1.7609\n",
            "*************************************\n",
            "\n",
            "Epoch is = 0\n",
            " Steps per epoch is = 375\n",
            " Batch is = 250\n",
            " InputShape = [64, 16]\n",
            " TargShape = [64, 11]\n",
            "  Input batch size = 64\n",
            "                 Target batch size = 64 \n",
            " Input length is = 16 \n",
            " Target length is = 11 \n",
            " Input is = [   1  216   12 3703    3    2    0    0    0    0    0    0    0    0\n",
            "    0    0] \n",
            " Target is = [   1    4 1232  233    3    2    0    0    0    0    0] \n",
            "\n",
            "Epoch 1 Batch 300 Loss 1.5800\n",
            "Epoch 1 Loss 1.7791\n",
            "Time taken for 1 epoch 2119.942216873169 sec\n",
            "\n",
            "tf.Tensor(\n",
            "[[0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " ...\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]\n",
            " [0. 0. 0. ... 0. 0. 0.]], shape=(64, 1024), dtype=float32)\n",
            "Epoch 2 Batch 0 Loss 1.4346\n",
            "Epoch 2 Batch 100 Loss 1.2370\n",
            "Epoch 2 Batch 200 Loss 1.2682\n",
            "Epoch 2 Batch 300 Loss 1.2312\n",
            "Epoch 2 Loss 1.3116\n",
            "Time taken for 1 epoch 1964.2011487483978 sec\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mU3Ce8M6I3rz"
      },
      "source": [
        "## Translate\n",
        "\n",
        "* The evaluate function is similar to the training loop, except we don't use *teacher forcing* here. The input to the decoder at each time step is its previous predictions along with the hidden state and the encoder output.\n",
        "* Stop predicting when the model predicts the *end token*.\n",
        "* And store the *attention weights for every time step*.\n",
        "\n",
        "Note: The encoder output is calculated only once for one input."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2PpVcHAO1ARe"
      },
      "source": [
        "# New Section"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EbQpyYs13jF_"
      },
      "source": [
        "def evaluate(sentence):\n",
        "  attention_plot = np.zeros((max_length_targ, max_length_inp))\n",
        "\n",
        "  sentence = preprocess_sentence(sentence)\n",
        "\n",
        "  inputs = [inp_lang.word_index[i] for i in sentence.split(' ')]\n",
        "  inputs = tf.keras.preprocessing.sequence.pad_sequences([inputs],\n",
        "                                                         maxlen=max_length_inp,\n",
        "                                                         padding='post')\n",
        "  inputs = tf.convert_to_tensor(inputs)\n",
        "\n",
        "  result = ''\n",
        "\n",
        "  hidden = [tf.zeros((1, units))]\n",
        "  enc_out, enc_hidden = encoder(inputs, hidden)\n",
        "\n",
        "  dec_hidden = enc_hidden\n",
        "  dec_input = tf.expand_dims([targ_lang.word_index['<start>']], 0)\n",
        "\n",
        "  for t in range(max_length_targ):\n",
        "    predictions, dec_hidden, attention_weights = decoder(dec_input,\n",
        "                                                         dec_hidden,\n",
        "                                                         enc_out)\n",
        "\n",
        "    # storing the attention weights to plot later on\n",
        "    attention_weights = tf.reshape(attention_weights, (-1, ))\n",
        "    attention_plot[t] = attention_weights.numpy()\n",
        "\n",
        "    predicted_id = tf.argmax(predictions[0]).numpy()\n",
        "\n",
        "    result += targ_lang.index_word[predicted_id] + ' '\n",
        "\n",
        "    if targ_lang.index_word[predicted_id] == '<end>':\n",
        "      return result, sentence, attention_plot\n",
        "\n",
        "    # the predicted ID is fed back into the model\n",
        "    dec_input = tf.expand_dims([predicted_id], 0)\n",
        "\n",
        "  return result, sentence, attention_plot"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s5hQWlbN3jGF"
      },
      "source": [
        "# function for plotting the attention weights\n",
        "def plot_attention(attention, sentence, predicted_sentence):\n",
        "  fig = plt.figure(figsize=(10,10))\n",
        "  ax = fig.add_subplot(1, 1, 1)\n",
        "  ax.matshow(attention, cmap='viridis')\n",
        "\n",
        "  fontdict = {'fontsize': 14}\n",
        "\n",
        "  ax.set_xticklabels([''] + sentence, fontdict=fontdict, rotation=90)\n",
        "  ax.set_yticklabels([''] + predicted_sentence, fontdict=fontdict)\n",
        "\n",
        "  ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
        "  ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
        "\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sl9zUHzg3jGI"
      },
      "source": [
        "def translate(sentence):\n",
        "  result, sentence, attention_plot = evaluate(sentence)\n",
        "\n",
        "  print('Input: %s' % (sentence))\n",
        "  print('Predicted translation: {}'.format(result))\n",
        "\n",
        "  attention_plot = attention_plot[:len(result.split(' ')), :len(sentence.split(' '))]\n",
        "  plot_attention(attention_plot, sentence.split(' '), result.split(' '))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n250XbnjOaqP"
      },
      "source": [
        "## Restore the latest checkpoint and test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UJpT9D5_OgP6",
        "outputId": "4bc58863-3411-42f5-99cd-9a833a100264",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "# restoring the latest checkpoint in checkpoint_dir\n",
        "checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.training.tracking.util.InitializationOnlyStatus at 0x7f62afe52278>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 152
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WrAM0FDomq3E",
        "outputId": "7c7c1fa3-5e0c-48e3-fb02-8cde86e3601b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 678
        }
      },
      "source": [
        "translate(u'hace mucho frio aqui.')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input: <start> hace mucho frio aqui . <end>\n",
            "Predicted translation: it s a good beer . <end> \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAJwCAYAAAC08grWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZzlB1nn+++TdBYTCAgohIzIGggiS2hBBE0UZlDxojK4IEsAhzACF7iIIoNeuMxFZRNRcIaoAyKgCAM3LA4OCsomgwEXMuwGQgJCiAZIIEBInvvH7zRUF90hwXQ9p6ve79erX1T9zqlTT/3o9PnUb63uDgDAhEOmBwAAdi4hAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESJroKpuVlVvqKrvnJ4FALaSEFkPpyQ5OcmDh+cAgC1Vbno3q6oqyUeSvD7J/5Hk+t196ehQALBFbBGZd3KSqyd5ZJIvJ/nh0WkAYAsJkXmnJHl5d38+yR+vPgeAHcGumUFVdXSSf0pyj+5+c1XdNslfJzm2uz89Ox0AHHi2iMz690nO7+43J0l3/12SDyb56dGpADjoVdXRVfWAqrrG9CyXR4jMun+SF21a9qIkD9z6UQDYZn4yyfOzvNesLbtmhlTVtyX5cJITuvuDG5b/myxn0dyyuz8wNB5roKpuneSxSW6ZpJO8J8nTu/vM0cGAg0JVvTHJdZN8vrt3T8+zP0IE1lBV3TPJK5K8OclbVovvsvpzr+5+9dRswPqrqhsm+UCSOyR5e5ITu/s9kzPtjxAZVFU3SHJO7+P/hKq6QXd/dGAs1kBV/UOSV3b3Ezctf3KSH+3u28xMBhwMqupXkpzc3Xetqlck+WB3P256rn1xjMisDyf5ls0Lq+raq8fYuY5P8of7WP6HSW6+xbMAB58H5Kv/hrw4yX1XF9BcO0JkVmXZ97/Z1ZJ8YYtnYb2cl+T2+1h++ySf3OJZgINIVX1PkmOTvHy16NVJjkpyt7GhLseu6QF2oqr6rdWHneTXqurzGx4+NMs+vb/b8sFYJ7+b5HlVddMkb1stu3OWg1efPjYVcDA4Jcnp3X1RknT3l6rqT7Kckfn6ycH2xTEiA1ZHMifJSVkuYPalDQ9/KctZM8/YeDYNO8tqE+qjk/x8kuuvFn88S4T81r6OKwKoqiOSfCLJfbr7dRuW3yXJnyW57p5AWRdCZMjqjeZPkjy4uy+cnof1VVVXTxJ/T4Cvp6quk+WeZS/q7ss2PXa/JH/e3Z8YGW4/hMiQqjo0y3Egt1nXU6oA4EBzjMiQ7r60qs5Ocvj0LKyfqrpWkqckuWuSb82mA8u7+5iJuQCuakJk1n9O8utVdb/uPn96GNbK7ye5XZLTshwbYtMlsF9V9eFcwX8nuvvGB3icK8WumUFV9e4kN0pyWJJzk3xu4+PdfeuJuZhXVZ9N8m+7+39NzwKsv6r6+Q2fXi3JY5K8I8sJEUlypyxnZD6zu5+8xeNdLltEZr386z+FHeq8JGt1ZDuwvrr7mXs+rqoXJHlqd//qxudU1eOTfMcWj/Z12SICa6iqfirLnTNPWbdT7YD1ttqiemJ3f2jT8psmede6HWNmiwhro6oeluThWXZX3aq7z6qqX0pyVnf/yex0B95qV93G3wxulOS81UHNl2x8rt12wOX4XJKTk3xo0/KTk3x+85OnCZFBVXV4kickuU+SG2Q5VuQruvvQibkmVNWjk/xikqcm+fUND30sySOyXHNlu7OrDrgqPCvJc6tqd5Y77ybJd2e54uqTpobaH7tmBlXVU5P8VJJfy/IX55eT3DDJTyf5le5+3tx0W6uq3pfk57v7tVV1YZbrq5xVVd+R5E3dfe3hEWFUVZ2Y5O+6+7LVx/vV3e/aorFYU1X1k0keleSE1aL3Jnn2Om5dFiKDVqdb/Vx3v2715nvb7v7Hqvq5JHft7nsPj7hlquriJLfo7rM3hcjxWf7xPWp4xC1VVSclSXf/1T6Wd3e/aWQwxlTVZUmu193nrT7uLDfO3Kx30tZUDn52zcy6bpI9V1W9KMk1Vx+/Lssuip3krCQnJjl70/IfzlfX0U7yrCT7OsXumCybVvd1Z162txsl+dSGj+Hrqqpr5msviPgvQ+PskxCZ9dEsNzT7aJaDiu6e5J1Zzve+eHCuCc9I8pyqOirLb3l3qqr7Zzlu5MGjk824eZK/38fyM1ePscN099n7+hg2q6pvT/JfsxycuvHq3ZVlS9pabTETIrNemeUS3m9P8uwkf1RVD0lyXHbYrd67+/lVtSvJryY5KskfZrmi6CO7+6Wjw824OMmxST68aflx2ftuzexAjhHh63h+li3sP5uD4MrMjhFZI1V1xyR3TvKB7n7N9DxTVnePPKS7z5ueZUpVvTjLmVT37O4LVsuuleT0JOd2930m52PWfo4R+co/5o4R2dmq6qIk393dZ07PckUIkUFV9X1J3tbdX960fFeS79lJBySuzo45tLv/YdPyWyf58k67Q3FVHZvkTVlueLdnndw6yxVXT+ruj0/NxrzVpveNDstyb6InJHl8d/+PrZ+KdbG6JtEDu/ud07NcEUJkUFVdmuTYzb/5V9W1k5y3k36rqaq3Jnlud79k0/KfTvKI7r7LzGRzVsfL3DfJbVeL/jbJS7p77S5ItBWq6geS3DLLb/7v6e43Do+0dqrq3yV5YnffeXoW5qz+W/mlJA/bfHXVdSREBq02r163uz+1afnxSc5Yt8vwHkirU3Zvt49LEt8kyyWJrzEzGdOq6rgsx1PdPsv+7mQ5yPuMJD9u69BXVdXNspzufvT0LMxZ/Xt6RJaDUr+YZK+t7uv23uJg1QFV9arVh53kRVX1xQ0PH5rkVknetuWDzbo0yb5i45uz72slbGtVda/Le7y7X7FVs6yB38ry9+Om3f3hJKmqGyd50eqxHXO9nT1WxwvttSjLwc1PSvL+LR+IdfOI6QGuDFtEBlTV81cfnpLl0uUbT9X9UpKPJPnd7j5/i0cbU1WnZ3mz+YnuvnS1bFeSlyU5rLt/ZHK+rbbaWrYvneysgxFXN/A6efOZIKvLV//FTtxatuFg1b0WJzknyU9199u/9qtgPdkiMqC7H5QkVfWRJM/o7s/NTrQWfjHJW5J8qKreslp2lyRXS/J9Y1MN6e69LkC0irLbZTmt+wkjQ83a129MO/m3qO/f9PllWS529qHNB7+zM1XVdZPcP8lNstwy5PyqunOSj+/ZsrgubBEZVFWHJEl3X7b6/HpJfiTLgXg7bdfMnjNFHpG9D878HccAfFVVfU+S/9Ldt5meZatU1SuTfEuS+3T3OatlN0jy4iSf6u7L3Y0FO01V3T7JX2S5DtF3ZLl9xllV9aQkx3f3z0zOt5kQGVRV/yPJ67r72VV1tSTvS3J0lq0AP9vdLxwdkLVTVbdM8o7uvtr0LFulqr4tyauyHDu18WDVd2e5zsq5U7NNWZ36f4XspMsAsKiqN2a5WegTN927605J/ri7N5/+PcqumVm7s+ySSJJ7JflslntI3DfJY5PsuBCpqutnuZDXxssS77h/TPdx5cw9ByM+LsuWoh2ju89ZrY+7JbnFavF7u/vPB8ea9pf56q6pPQdzb/58z7IdczwRX3H7LFdV3eyfstzjbK0IkVlXS/Lp1cf/Lskru/uSqnpDkufOjbX1VgHykizHg+y5YuTGzXU77R/TM7Lvu6u+PTvw3ju9bLp9/eoPyy7cZyR5SpK/Xi27U5L/lOWXGwer7mwXZznjcLNbZLko4loRIrM+muTOVfXqLDe8+4nV8msl2WkXrfrNLGfN3DLJ3yT5wSzl/uQk/9fgXFM23131sizHQ3xhYpitVlWPyXJ80BdWH+9Xd//GFo21Tv5zkkd198YwO6uqzkvytO6+3dBcrIfTkzyxqva8p3RV3TDLXd3/+9RQ++MYkUFV9dAkz0lyUZKzk5zY3ZdV1SOT/Fh3/8DogFuoqj6Z5B7dfcbqdM3d3f2BqrpHliO+v3t4xC23Our9zlku8775Nt6/MzLUFqmqD2f5O/DPq4/3p7v7xls117qoqouz/Hvx3k3Lb5nknd39TTOTsQ6q6pgkf5rlthBHJ/lEll/s3pbkh9btTE0hMmx1dPMNkry+uy9aLbtHkk9391tHh9tCq/i4dXd/ZHVa8/26+y1VdaMk/7u7j5qdcGtV1f2S/F6WXTMXZO/dVN3d1x8ZjLVQVWck+VCSB3X3xatl35Tlrqs37e7dk/OxHlaXej8xyy8y71rX46rsmhlSVdfI8sb75iSbb0z06SQ76iZvWc4YukWWi7n9XZL/WFXnJHl4ko8NzjXlKUmeluTJO/m6EFV1WJbryzygu10x9Kt+LslrknysqvbcFPE7s+zevMfYVIzb+N7S3W9I8oYNj905y+UhLhgbcB9sERlSVVfPcgTz3Tdu+aiq2yR5R5LjdtiVVe+b5QqqL1idIfG6JNfJcp+EU7r7T0YH3GJVdUGS23f3WdOzTFsd93CX7v7A9CzrpKqOTvIzSU5YLXpvlpsirtVmd7bWwfjeIkQGVdWLk1zU3Q/dsOwZWS44c8+5yeat7jx7iyQfXbf/aLZCVT0nyfu7+7enZ5lWVU9Pku7+helZ1snqart3yL5Pd99xp/7zVQfbe4sQGVRVd0/yR0mu191fWl1p9dwst73fSTc1S5JU1U8luWv2fXDm2v3HcyBV1eFJ/r8s9x56d5JLNj7e3U+emGtCVf1OlmvrfDjLbsy9fuPv7kdOzDWpqm6R5NVZzq6qLLtkdmX5e/LFdbu7KlvrYHtvcYzIrNdnOd/7R5K8Isub8OFZ/oHZUVa/9T46yRuzXD1zpxfyQ7Ocwnx+kptm08GqWU5r3rZWVw592+r4mBOS7Lnh3eYzZHbq35PfzBJlt81yRsRts9y9+r8k+eXBuVgPB9V7iy0iw6rqqUlu3t0/VlUvTHJhdz98eq6ttjp99+Hd/fLpWdbB6riIX+vuZ03PMqGqLk1ybHefV1VnJfmu7v7n6bnWRVX9c5KTuvvMqvpMkjt09/ur6qQkv93dtx4ekWEH03uLLSLzXpjknaubeP14lnLdiQ7JcrYMi0Oz3F9lp7ogy26H85LcMJt21ZHKVy96+KkkxyV5f5bN7zedGoq1ctC8t9gisgZW1wS4OMl1uvuEr/f87aiqnpLkku5+0vQs62B1YNlnd9KxIBtV1fOSnJLl6P8bZHmDvXRfz92hFzR7U5Jndfcrq+olSa6d5FeTPCTLqZu2iHDQvLfYIrIeXphln+8TpgfZSlX1Wxs+PSTJfavq3yb5h3ztwZk77YDEo5L8h9VBZztxffzHLFuEbpbkN7JcqOvC0YnWy1OyXDEzWY4JeW2W46vOT/KTU0Otm6p6b5KbdfdOfa87KN5bdur/OevmRVluUPT86UG22Hdu+nzPrplbbFq+EzfbnZCv3mV3x62P1U3uXpt85foHz+xuIbLS3X+24eOzkpxQVddKckHbzL3Rc7NsLdqpDor3FrtmAIAxDgADAMYIEQBgjBBZE1V16vQM68T62Jv1sTfrY2/Wx96sj72t+/oQIutjrf+iDLA+9mZ97M362Jv1sTfrY29rvT6ECAAwZsefNXN4HdFHfuV0/DmX5Is5LEdMj7E2rI+9WR97sz72Zn3szfrY27qsjwtzwfnd/S2bl+/464gcmaNzx1rbK98CwLbw5/3ys/e13K4ZAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGDMtgiRqnpBVb1meg4A4MrZNT3AVeRRSSpJquovk5zZ3Y8YnQgA+Lq2RYh092emZwAArrxtESJV9YIk10lyfpKTkpxUVQ9fPXyj7v7I0GgAwOXYFiGywaOSHJ/kfUn+02rZp+bGAQAuz7YKke7+TFV9Kcnnu/sT+3teVZ2a5NQkOTJHbdV4AMAm2+KsmSuru0/r7t3dvfuwHDE9DgDsWDsyRACA9bAdQ+RLSQ6dHgIA+Pq2Y4h8JMkdquqGVXWdqtqOPyMAbAvb8U36GVm2irwnyxkzN5gdBwDYn21x1kx3P3DDxx9Icqe5aQCAK2o7bhEBAA4SQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGLNreoBp9U1H5pDjbzE9xtr42N2uNT3CWnnof3j19Ahr5VUP/v7pEdbKrnPOnx5hrVz26c9Mj7BW+kuXTI+wXr6078W2iAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBm24VIVX1fVb29qi6qqs9U1Tuq6lbTcwEAX2vX9ABXparaleT0JL+f5L5JDktyYpJLJ+cCAPZtW4VIkmOSXDPJq7v7H1fL3rf5SVV1apJTk+TIw47ZuukAgL1sq10z3f0vSV6Q5M+q6rVV9ZiqusE+nndad+/u7t2H7zp6y+cEABbbKkSSpLsflOSOSd6U5J5J3l9Vd5+dCgDYl20XIknS3X/f3U/t7pOT/GWSU2YnAgD2ZVuFSFXdqKp+vaq+p6q+vaq+P8mtk7xnejYA4Gttt4NVP5/k+CQvS3KdJJ9M8uIkT50cCgDYt20VIt39yST3mp4DALhittWuGQDg4CJEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxu6YHGNed+vJl01OsjW975cemR1grv3/y90yPsFYuvNfR0yOsleP+6ojpEdbK0e+8dHqEtXLZpz8zPcJBwRYRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxmybEKmqH6yqN1fVBVX1L1X1Z1V1wvRcAMD+bZsQSXJ0kt9McockJyf5TJJXV9Xhk0MBAPu3a3qAq0p3//eNn1fVg5J8NkuYvGXTY6cmOTVJjjzsmK0aEQDYZNtsEamqm1TVS6rqH6vqs0k+meXnu8Hm53b3ad29u7t3H37oUVs+KwCw2DZbRJK8Jsm5SR6a5GNJvpzkPUnsmgGANbUtQqSqrp3kFkke1t1vXC07Mdvk5wOA7Wq7vFFfkOT8JA+pqnOSHJfk6Vm2igAAa2pbHCPS3Zcl+akkt05yZpLnJvmVJF+cnAsAuHzbZYtIuvsNSW61afHVJmYBAK6YbbFFBAA4OAkRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGDMrukBxn3pkvQ5/zQ9xdq49MILp0dYK8c+7LjpEdbKBb98jekR1srZPzo9wXo55mY3mR5hrRz3Gu8te/ngvhfbIgIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjNm2IVJVZ1bVk6bnAAD2b9uGCACw/oQIADDmgIdIVR1dVS+sqouq6pNV9fiqek1VvWD1+DdX1R9U1QVVdXFV/XlVfcem17hXVb27qr5YVedU1ROqqjY8/q1Vdfrq68+uqgcf6J8LAPjX24otIs9MclKSH0/yA0luk+R7Nzz+giR3TPKjSe6Q5PNJXldV35QkVXX7JC9L8ook35nkl5I8PskjNr3GTZPcLcmPJXlAkhsemB8HALiq7DqQL15VV0vy4CQP6O7Xr5b9bJJzVx/fLMk9k5zU3W9aLbt/ko8muW+S30vymCR/1d1PXL3sB1Zf97gkv11Vxyf5oSR36e63rl7jlCRnXc5cpyY5NUmOrKOv0p8ZALjiDvQWkZskOSzJO/Ys6O7PJTlz9ekJSS5L8tcbHv9MkncnueWG57x10+u+JclxVXXMhtfY+D3OTvLx/Q3V3ad19+7u3n14HfmN/WQAwL/aOh+s2lfyOVfk+QDAGjnQIfKPSS5J8l17FlTVUUlutfr0vasZ7rTh8WOyHAvyng3PufOm171LknO7+8Ik71u9xh02vMYNklz/qvxBAICr3gENke6+KMl/S/LUqrprVd0yy3EfhywP9weTnJ7keVX1vVX1nUlelOSzSV6yeplnJjmpqp5UVcdX1X2T/HySp62+x/uTvG71GneqqttmOXj14gP5swEA/3pbsWvmsUnenORVSd6Y5B+SnJHkC6vHH5Tl+I5Xrf73qCQ/2N0XJ0l3vyvJTyT591mOLfn11Z/nbPgeD0zy4SRvSPLqLBHzkQP3IwEAV4UDetZM8pWtIvdf/UlVHZHk0Un+dPX4BUlO+Tqv8Yosp+/u7/FPZjn7ZqPf+8anBgC2wgEPkaq6XZYzW96R5OpZTru9epKXHujvDQCstwMeIiuPSXLzJF9O8ndJvq+7z92i7w0ArKmt2DXzt0l2H+jvAwAcfNb5OiIAwDYnRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABiza3qAaX3ZZbnswgunx2BNffncj02PsFZOeMLF0yOslfc+7cbTI6yVP3jUs6ZHWCuPe/tDpkdYLx/c92JbRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMVsWIlX1l1X1nK36fgDA+rNFBAAYs61DpKoOn54BANi/rQ6RXVX17Kq6YPXn6VV1SLJEQ1U9tarOrarPV9XfVNXdN35xVd2yql5bVRdW1XlV9UdVdb0Nj7+gql5TVY+rqnOTnLvFPx8AcCVsdYjcd/U975TkoUlOTfLo1WPPT3JSkp9Jcqskf5Dk1VV1mySpqmOTvCnJmUnukORuSa6W5PQ9MbNyUpJbJ/nBJHfd1xBVdWpVnVFVZ1ySL16lPyAAcMXt2uLv909JHtndneR9VXV8ksdU1elJ7pPkht390dVzn1NVd8sSLA9L8nNJ/r67H7fnxarqAUn+JcnuJO9YLf5Ckgd3934Lo7tPS3JakhxT1+qr8gcEAK64rd4i8vZVhOzx10mOS3KXJJXkPVV10Z4/Se6R5Car594+yfdtevyc1WM32fCaZ15ehAAA62Ort4hcnk7yXUku2bT84tX/HpLktUkeu4+v/eSGjz931Y8GABwIWx0id6yq2rBV5LuTfDzLlpFKcr3ufuN+vvZdSX4yydndvTlWAICD0Fbvmrl+kt+sqptX1b2T/EKSZ3X3B5K8OMkLqureVXXjqtpdVY+tqnutvva5Sa6R5KVVdcfVc+5WVadV1dW3+OcAAK4CW71F5MVJDk3yv7Lsivn9JM9aPfagJE9I8rQk/ybLQajvSPLGJOnuj1fVnZP8WpLXJTkyyUeT/M/EqS8AcDDashDp7pM3fPqIfTx+SZInrf7s7zU+mOTel/P4A7/R+QCArbetr6wKAKw3IQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjKnunp5h1DF1rb5j3XV6DICD3qEn3Gx6hLXyp3/xsukR1sqhx37ond29e/NyW0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDFCBAAYI0QAgDG7pgeYUFWnJjk1SY7MUcPTAMDOtSO3iHT3ad29u7t3H5YjpscBgB1rR4YIALAehAgAMEaIAABjtm2IVNUjqup903MAAPu3bUMkyXWS3Hx6CABg/7ZtiHT3k7q7pucAAPZv24YIALD+hAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMGbX9AAAbA/94XOmR1grt/21h02PsGYes8+ltogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECKGYZZoAAAX2SURBVAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGMOmhCpqsdW1Uem5wAArjoHTYgAANvPVRIiVXVMVV3zqnitK/E9v6WqjtzK7wkAXLW+4RCpqkOr6u5V9ZIkn0hym9Xya1TVaVV1XlVdWFV/VVW7N3zdA6vqoqq6a1WdWVWfq6o3VtWNNr3+L1bVJ1bPfWGSq20a4YeTfGL1ve78jf4cAMCcKx0iVfUdVfW0JOckeWmSzyX5wSRvqqpK8tokxyX5kSS3S/KmJG+oqmM3vMwRSR6f5MFJ7pTkmkn+64bv8ZNJ/t8kT0xyYpL3J3nMplFenORnklw9yeur6kNV9X9vDpr9/AynVtUZVXXGJfnilV0FAMBV5AqFSFVdu6oeWVXvTPK3SW6R5FFJrtfdD+nuN3V3J/n+JLdNcu/ufkd3f6i7fyXJWUnuv+EldyV5+Oo5/5DkGUlOXoVMkjw6yR909/O6+wPd/ZQk79g4U3d/ubv/tLvvk+R6SX519f0/WFV/WVUPrqrNW1H2fO1p3b27u3cfliOuyCoAAA6AK7pF5P9M8uwkX0hyfHffs7tf1t1f2PS82yc5KsmnVrtULqqqi5LcKslNNjzvi939/g2ffzzJ4Um+efX5CUn+etNrb/78K7r7s93937r7+5N8V5LrJvn9JPe+gj8fADBg1xV83mlJLknygCRnVtUrk/xhkr/o7ks3PO+QJJ9M8r37eI3Pbvj4y5se6w1ff6VV1RFZdgXdL8uxI/87y1aV07+R1wMAtsYVeuPv7o9391O6++ZJ7pbkoiR/nOTcqnpmVd129dR3Zdkacdlqt8zGP+ddibnem+S7Ny3b6/Na3KWqnpflYNnfTvKhJLfv7hO7+9ndfcGV+J4AwBa70lsguvvt3f1zSY7Nssvm+CR/U1Xfm+TPk7w1yelV9UNVdaOqulNV/T+rx6+oZyc5paoeUlU3q6rHJ7njpufcL8n/THJMkvsk+bbu/oXuPvPK/kwAwIwrumvma3T3F5O8PMnLq+pbk1za3V1VP5zljJffTfKtWXbVvDXJC6/Ea7+0qm6c5ClZjjl5VZLfSPLADU/7iywHy372a18BADgY1HKyy851TF2r71h3nR4D4KB3yJGuMbnRPz3kxOkR1sq7f+sx7+zu3ZuXu8Q7ADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY3ZNDwDA9nDZF74wPcJaue5vv216hLXy7v0st0UEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABiza3qACVV1apJTk+TIHDU8DQDsXDtyi0h3n9bdu7t792E5YnocANixdmSIAADrQYgAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOqu6dnGFVVn0py9vQcSa6T5PzpIdaI9bE362Nv1sferI+9WR97W5f18e3d/S2bF+74EFkXVXVGd++enmNdWB97sz72Zn3szfrYm/Wxt3VfH3bNAABjhAgAMEaIrI/TpgdYM9bH3qyPvVkfe7M+9mZ97G2t14djRACAMbaIAABjhAgAMEaIAABjhAgAMEaIAABj/n8ahgZGNGOeCAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zSx2iM36EZQZ",
        "outputId": "22dd46ba-58f5-4426-cc92-635d3f2e43ba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 678
        }
      },
      "source": [
        "translate(u'esta es mi vida.')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input: <start> esta es mi vida . <end>\n",
            "Predicted translation: this is my mother . <end> \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmcAAAJwCAYAAADMYcYyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5htB1nn+d9LEoIQLkK4ROSmiCLXCae5thCFEW/NMzK0ttwCcUg/TNvSQ6vI9ENL06KCoOJA0wSUcGsFMyoCgg0Cg3KRDmnkKgHlaggkApIQSELyzh97HymKc8KpyslZ7658Ps9Tz9m19q5db63nnLO/tdZea1V3BwCAGa6x9AAAAHyNOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScDVVV31VVb6yqOy09CwBw5IizuU5OclKSUxaeAwA4gsqFz+epqkrysSSvT/Ivknxbd1+26FAAwBFhy9lMJyW5bpKfTfLVJD+y6DQAwBEjzmY6OckZ3X1Rkt9ffw4AXA3YrTlMVV0nyaeT/Gh3/0VV3TXJ25Oc0N1fWHY6AOCqZsvZPP97kvO7+y+SpLvfneTDSf7VolMBwAapqutU1SOr6vpLz7JT4myeRyR56bZlL03yqCM/CgBsrJ9I8sKsXlc3it2ag1TVLZJ8NMntu/vDW5Z/e1ZHb35vd5+90HgAsDGq6k1Jbprkou7et/Q8OyHOAIA9papuneTsJHdP8o4kJ3b3B5acaSfs1hymqm65Ps/ZAe870vMAwAZ6RJK/WL9v+0+zYWc9EGfzfDTJjbcvrKobre8DAK7YI5O8ZH37ZUkedrANHxOJs3kqyYH2NR+X5CtHeBYA2ChVde8kJyQ5Y73oVUmuneQBiw21Q0cvPQArVfXb65ud5Fer6qItdx+V1X7zdx/xwQBgs5yc5JXdfWGSdPclVfWKrM568PolBztU4myOO63/rCS3T3LJlvsuSXJWkmcc6aEAYFNU1bFZnULjp7bd9dIkf1ZVx+2PtskcrTnIen/4K5Kc0t0XLD0PAGySqjo+q+tRv7S7L99238OTvKG7z11kuB0QZ4NU1VFZva/sLpt0yC8AcPg4IGCQ7r4syceTXHPpWQCAZdhyNkxVnZzVvvKHd/f5S88DANNV1Udz4DMdfIPu/o6reJwrzQEB8/xcktsk+fuq+lSSL229s7vvvMhUADDXs7fcPi7J45O8M8nb18vuldVZD555hOfaFXE2zxnf/CEAwH7d/U/RVVWnJ3lad//K1sdU1ROT3OEIj7YrdmsCAHtGVX0xq2tpfmTb8tsmOau7r7fMZIfOAQEAwF7ypSQnHWD5SUkuOsDycezWHKaqrpnkP2R1UMAtkxyz9f7uPmqJuQBgQ/xmkudU1b4k71gvu2dWVw548lJD7YQ4m+c/J/nJJL+a1V+wn09y6yT/KsmTlhsLAObr7qdX1ceSPC6rqwUkyQeTnNzdr1hssB3wnrNh1ocDP7a7X1dVFyS5a3f/bVU9Nsn9u/shC484UlU9Ol/b2vh154nbhMOmYa+rqm9N8sM58L/RpywyFAxly9k8N02y/+oAFya5wfr265I8bZGJhquqn0/yxCTPS3LfJP8lyW3Xt12PFBZWVfdM8pokFye5cZK/T3LC+vOPJRFnXCWq6gbZ9v767v7cQuMcMgcEzPOJJN+2vv2RJA9c375Xki8vMtF8j0lyanc/McmlSZ7d3Q/K6nw2t1p0MiBJfj3Jy5LcPKtL1P1AVlvQzoxfOjnMqupWVfXaqvpykn9Ict764/z1n+PZcjbPHyW5f1ZvYnxWkt+rqsdk9Z/ary852GDfntXJBpNVwO4/TPr31ssfs8RQwD+5c5Kf7u6uqsuSHNvdf1dVT0jy37IKNzhcXpjVXqefTnJODvHKAZOIs2HWW3/23z6jqj6Z5D5Jzu7uVy832WjnJjk+q62OH89qK+O7s9q1uXH/KGEPumTL7c9ktUX7g1m9dePbDvgVsHt3T3LP7n7f0oPsljgbpqrum+Rt3f3VJOnuv0ryV1V1dFXdt7vfsuyEI70xyYOSnJXkd5L8ZlX9RJITk2zEkTmwx52V5J8lOTvJm5P8clXdNMnDk7xnwbnYmz6a5Nilh7gyHK05zHqT/wnd/dlty2+U5LPOc/aNquoaSa6xP2ir6iez3tqY5HndfemS88HV3fp8U9ft7jdV1Y2TvDhf+zf66O5+76IDsqdU1Q8k+cUk/+f2qwRsCnE2TFVdnuSm3X3etuW3S3LmJlx24kirqlsm+WRv+8tcVZXkFt39iWUmA+BIW5+G6tgkR2V1RPBXt96/Ca+jdmsOUVV/sr7ZSV5aVRdvufuoJHdM8rYjPthm+GhWh+V/dtvyG67vs7UR4OrjZ5Ye4MoSZ3P8w/rPSvL5fP1pMy5J8pdJnn+kh9oQlQO/8f+4rA7bB46w9Qm1D2nXjBNFczh194uWnuHKEmdDdPejk2R9yYlndPeXlp1ovqr67fXNTvKrVbX1grZHZXXEzruP+GBAkjx7y+3jkjw+q1PbvH297F5Z/Rt95hGei6uB9QEnj0jynUme1N3nV9V9kpzT3R9ddrpvznvOhlm/uT3dffn685sl+bEkH+huuzW3qKo3rW/eL6v/8Lcern9JVmcef0Z3f/gIjwZsUVWnZ3U6oF/ZtvyJSe7Q3Q9fZDD2pKq6W5I/z+ptLXdI8j3r8+o9OcntuvuhS853KMTZMFX12iSv6+5nVdVxSf4myXWy+s3zp7v7xYsOOFBVvTDJ47r7i0vPAnyjqvpikhO3HzlXVbdNctYmvEGbzbH+xf0t3f1L64MD7rKOs3sl+f3uHn/lGLs159mX5BfWtx+c5ItJbpPkYUl+LqtD0Nli/y7h/arqW7I6TP/D3f3xZabaPNbbwVXVg5O8qrsvXd8+qO7+wyM01ib5UpKTsrok3VYnJblo+4PhSrpbVlcH2O7TWV2/ejxxNs9xSb6wvv2DSf5o/YLwxiTPWW6suda7TN7Z3f+lqq6Z1fta7pDkkqr68e5+7aIDDmW97cgZSW6W1RHBZ1zB4zqODj6Q30zynPX5zt6xXnbPJCcnefJSQ7FnfTnJtx5g+ffkG4/qH8mFz+f5RJL7VNV1srro+evXy28Yv2EezAPztf/wH5Tkulm9kD45/uO/ItbbIerua+w/MfT69sE+hNkBdPfTs3pz9p2S/Mb6405JTu5uFz7ncHtlkl+qqv1XCeiqunWSpyX5f5caaie852yYqvrXWR3ldGFW14k8sbsvr6qfTfK/dfcPLDrgQFX1lSS37e5PVdULkvxjd//79T/G93b3dRcdcCjrbffWR4LdJ8lN8vW/5HZ3P3eZqYAkqarrJfnTJHfO6j3b52a1O/NtSX54E86GYLfmMN39vKo6M8ktk7x+/1GbSf42yZOWm2y0c5Pcsao+ndXWoFPXy49L4tJNB2e97UJVPTzJC/K1cxJu/Q23k4gzWND64LB/vr6M04lZ/QJ1Vne/YdnJDp04G6Sqrp/kzt39F0nete3uLyT5wJGfaiP8bpKXJzknyWVZHUKdJPfI6mhXDsx6252nJnl6kqfsv54r32h9hOZ3rM8vdUGu4IS0jtbkcNn6Otrdb0zyxi333Ser01J9frEBD5E4m+XyJK+tqgd291v3L6yqu2T1F+zmi002WHc/parel+RWSV7R3fvPd/bVrN5jwAFYb7t2vSSnC7Nv6t8muWB9e+Mvp8PG2BOvow4IGKS7L8jqjYyP3HbXI5L8WXeff+Sn2hhfTvKAJK+vqlusl10zq/fucXDW2869LMmPLj3EdN39ou7ef43gH8/q79TvrZd/3ceCY7LH7JXXUXE2z4uT/Mv1qQ32XzHgoUlOX3KoyarqYUlekeTsrM4Jd8z6rmvka+eMYxvrbdcen+SHq+qPq+o/V9V/3Pqx9HBDXZTkRUk+U1UvqKr7LT0Qe9rGv46Ks3len9XWjB9bf37/rLZkvGqxieb7hSSP6e7/K6tdcvu9I8ldlxlpI1hvu/Ovk/xQkntntUXoX275eMiCc421vlzOTbPa1fltWW2p/XhV/VpV3XHZ6diDNv51VJwNsz4686X52ibZRyR5eXc7eu7gvitfu5jyVhdm9f4gDsx6250nJfn33X2T7r5jd99py8edlx5uqu7+Une/tLt/JKv3/fx6Vi+e7152MvaavfA66oCAmV6c5F1VdcusfjO//8LzTHdOkttldV64re6b1SlIODDrbXeOSvInSw+xqarqWkl+IKvTt9wuySeXnYg9aqNfR205G6i735/kfVm98fhT3f3OhUea7rQkv70+TDpJblFVJ2d1ugPnnDo46213XpjVtW45RLXyg1X1oiSfyerv1zlJ7t/dt1l2OvaiTX8dteVsrhcn+a0k/2HpQabr7qevz23z+iTXSvKmJBcneUZ3ux7pQVhvu3btJP9HVT0wyXuy7YS93f2zi0w126ez2lX+2iSPSvKaLaduYReq6oNJvqu7vY4f3Ma+jrp801BVdcOs3jz7vO4+d+l5NkFVXTvJ92a1RfgD3e10EIfAetuZqnrTFdzdLrH2jarqMUn+oLu/sPQse0VV/UySG3X3f1p6lqk2+XVUnAEADOI9ZwAAg4gzAIBBxNlwVXXq0jNsIutt56yz3bHedsd62znrbHc2cb2Js/k27i/VENbbzllnu2O97Y71tnPW2e5s3HoTZwAAgzhaM8k169i+Vq6z9BgHdGkuzjE5dukxNo71tnPW2e5Yb7tjve2cdbY7k9fbBfn8+d194+3LnbwuybVyndyjNurKDgDAhntDn7H98nlJ7NYEABhFnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwyNg4q6qTqqqr6vgr8xgAgE0yJs6q6s1V9ewdftnbkpyQ5B+ugpEAAI64o5ce4Mro7kuSnLv0HAAAh8uILWdVdXqS+yX5N+vdlJ3k1uu771JVf1VVF1XVmVV14pav+7rdmlV1/ap6SVV9tqq+UlV/V1X/7kj/PAAAuzUizpI8Lsnbk7wwq92UJyT55Pq+X03yi0lOzGr35cuqqg7yPL+c5E5JfizJdyc5JcnfX3VjAwAcXiN2a3b3P1bVJUku6u5zk6Sqvmd995O6+03rZU9J8pdJbp7kUwd4qlslOau737n+/OMH+55VdWqSU5PkWrn2Yfk5AACurClbzq7Ie7bcPmf9500O8tjnJvnJqvrrqnpGVd3vYE/a3ad1977u3ndMjj1cswIAXCmbEGeXbrnd6z8POHd3vzarrWfPSHJ8ktdU1Quv2vEAAA6fSXF2SZKjruyTdPf53f2S7n5Ukp9OcnJV2TQGAGyEEe85W/tYkrtX1a2TXJhdhOP6PWlnJXl/Vj/bg5P8XXdffNimBAC4Ck3acvaMrLaefSDJeUluuYvnuDjJU5P8dZK3Jrlukn9xuAYEALiqVXd/80ftcderG/Y96v5LjwEAXI28oc94V3fv27580pYzAICrPXEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBjl56gAnq6KNz1PE3WXqMjfOVO91i6RE2Tj/h/KVH2EgXv+CEpUfYSDf47x9aeoSNc/lFFy09wma67LKlJ9hMlx54sS1nAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDbHycVdXpVfXqpecAADgcjl56gMPgcUlq6SEAAA6HjY+z7v7HpWcAADhc9tRuzaq6b1W9o6ourKp/rKp3VtUdl54RAOBQbfyWs/2q6ugkr0zyO0keluSYJCcmuWzJuQAAdmLPxFmS6yW5QZJXdfffrpf9zcEeXFWnJjk1Sa51jeOu+ukAAA7Bxu/W3K+7P5fk9CR/VlWvqarHV9Utr+Dxp3X3vu7ed81rfMsRmxMA4IrsmThLku5+dJJ7JHlLkgcl+VBVPXDZqQAADt2eirMk6e6/7u6ndfdJSd6c5ORlJwIAOHR7Js6q6jZV9WtVde+qulVVfX+SOyf5wNKzAQAcqr10QMBFSW6X5A+SHJ/kM0leluRpSw4FALATGx9n3f2oLZ8+eKk5AAAOhz2zWxMAYC8QZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGOTopQeYoL/61Vz2mc8uPcbGOcY627Gj33fTpUfYSPd53VuXHmEjvfom37f0CBvn5n/yyaVH2EiXf+a8pUfYTJceeLEtZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADDIyDirqjdX1XOr6plV9bmqOq+qHldVx1bVc6rqC1X1iap6xPrxb6yqZ297jutV1UVV9eBlfgoAgJ0bGWdrD0tyQZJ7JPm1JL+V5I+TnJ1kX5IXJXlBVZ2Q5PlJHlpVx275+p9KcmGSVx3JoQEArozJcfb+7n5yd384yW8kOT/Jpd39rO7+SJKnJKkk90nyh0kuT/LjW77+lCQv7u5LD/TkVXVqVZ1ZVWdemouv0h8EAOBQTY6z9+y/0d2d5LNJ3rtl2aVJPp/kJt19cZKXZBVkqao7JLl7kt852JN392ndva+79x2TYw/2MACAI+ropQe4Atu3ePVBlu0PzBckeU9V3TKrSHt7d3/wqh0RAODwmrzlbEe6+/1J/irJY5I8PMnvLjsRAMDOTd5ythvPT/Jfs9rC9vKFZwEA2LE9s+Vs7eVJLknyiu6+YOlhAAB2auSWs+4+6QDL7niAZTfbtugGSb4lV3AgAADAZCPjbKeq6pgkN0ryK0n+Z3e/deGRAAB2Za/s1rxPkk8nuXdWBwQAAGykPbHlrLvfnNUJaQEANtpe2XIGALAniDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGCQo5ceAK5OvnruZ5YeYSOd9dDbLz3CRrrwlMuXHmHjnP3Yb196hI1069ccv/QIm+ktB15syxkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgkDFxVlW3rqquqn1LzwIAsJRF4qyqTq+qVy/xvQEAJhuz5eyqUlXXXHoGAIBD9U3jrKreXFXPrapnVtXnquq8qnpcVR1bVc+pqi9U1Seq6hFbvuZOVfWGqvry+mtOr6rrr+97cpKTk/zoejdmV9VJW77lrarq9VV1UVV9oKr+123zfG9VvaaqLqiqz1bV71XVzbbcf3pVvbqqnlBVn0ryqSu3igAAjpxD3XL2sCQXJLlHkl9L8ltJ/jjJ2Un2JXlRkhdU1QlVdZ0kf5bkwiR3T/LjSe6d5HfXz/WMJK9I8oYkJ6w/3rblez01yW8nuUuS/5Hk96vquCSpqhOSvCXJ+9bP/YAkxyV5ZVVt/Vnul+TOSX4oyf0P8WcEAFjc0Yf4uPd395OTpKp+I8kvJrm0u5+1XvaUJE9Icp8k35rkOkke0d0XrO8/Ncmbquq23f2Rqvpykou7+9z936Cq9t/8ze5+1XrZ/53kkUnumuQvkzw2yV939xO2fN0jk3wuq0h853rxV5Kc0t0XH+wHWs90apJcK9c+xNUAAHDVOtQtZ+/Zf6O7O8lnk7x3y7JLk3w+yU2S3D7Je/aH2drbklye5Ht38r2SnLP+8ybrP++W5L5VdeH+jySfXN/3nVu+7n1XFGbrmU/r7n3dve+YHHsIYwEAXPUOdcvZpds+74Ms+2ax1zv5Xt3d6y1q+5/3Gklek+TnDvB1n9ly+0uH8H0AAMY51DjbiQ8mOaWqrrtl69m9swqrD64/vyTJUbt47rOS/ESSj6+31gEA7ClXxak0XpbkoiQvXh+1ed8kz0vyh939kfVjPpbkjlX13VV1fFUdc4jP/Zwk10/y8qq6R1V9R1U9oKpOq6rrHu4fBADgSDvscdbdFyV5YJLrZfUG/VcmeXuSU7Y87PlZbUU7M8l5WR1IcCjPfc76sZcneV2S92cVbBevPwAANto33a3Z3ScdYNkdD7DsZltuvzdXcAqL7j4vyQ8e4K46wGNr2+cfTvKQK3juRx3sPgCA6fb8FQIAADaJOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCBHLz0AwDdz2QfOXnqEjXTbJ15z6RE2zus+/s6lR9hId//gY5ceYTO95cCLbTkDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAgRy89wFKq6tQkpybJtXLthacBAFi52m456+7Tuntfd+87JscuPQ4AQJKrcZwBAEwkzgAABtnTcVZVP1NVf7P0HAAAh2pPx1mS45N899JDAAAcqj0dZ9395O6upecAADhUezrOAAA2jTgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAgRy89AABXjb70kqVH2Dg/dJt7LD3CRjrvuZcuPcJmOv3Ai205AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAyyUXFWVT9XVR9beg4AgKvKRsUZAMBed9jirKquV1U3OFzPd4jf88ZVda0j+T0BAK5KVyrOquqoqnpgVf23JOcmuct6+fWr6rSq+mxVXVBV/19V7dvydY+qqgur6v5V9b6q+lJVvamqbrPt+X+hqs5dP/bFSY7bNsKPJDl3/b3uc2V+FgCACXYVZ1V1h6p6epJPJnl5ki8l+aEkb6mqSvKaJDdP8mNJ/pckb0nyxqo6YcvTHJvkiUlOSXKvJDdI8l+3fI+fSPLLSX4pyYlJPpTk8dtGeVmShya5bpLXV9VHquo/bo88AIBNcchxVlU3qqqfrap3JfmfSb4nyeOS3Ky7H9Pdb+nuTvL9Se6a5CHd/c7u/kh3PynJ3yV5xJanPDrJv1k/5j1JnpHkpHXcJcm/S/Ki7n5ed5/d3U9N8s6tM3X3V7v7T7v7p5LcLMmvrL//h6vqzVV1SlVt39q2/+c5tarOrKozL83Fh7oaAACuUjvZcvZvkzwryVeS3K67H9Tdf9DdX9n2uLsluXaS89a7Iy+sqguT3DHJd2553MXd/aEtn5+T5JpJvnX9+e2TvH3bc2///J909xe7+3e7+/uT/LMkN03yO0kecpDHn9bd+7p73zE59gp+bACAI+foHTz2tCSXJnlkkvdV1R8leUmSP+/uy7Y87hpJPpPk+w7wHF/ccvur2+7rLV+/Y1V1bFa7UR+e1XvR3p/V1rdX7ub5AACWcMgh1N3ndPdTu/u7kzwgyYVJfj/Jp6rqmVV11/VDz8pqq9Xl612aWz8+u4PZPpjkntuWfd3ntfLPq+p5WR2Q8P8k+UiSu3X3id39rO7+/A6+JwDAona1laq73x3PUKwAAAOqSURBVNHdj01yQla7O2+X5H9U1fcleUOStyZ5ZVX9cFXdpqruVVX/aX3/oXpWkpOr6jFV9V1V9cQk99j2mIcn+e9Jrpfkp5Lcort/vrvft5ufCwBgaTvZrfkNuvviJGckOaOqbpLksu7uqvqRrI60fH6Sm2S1m/OtSV68g+d+eVV9R5KnZvUetj9J8htJHrXlYX+e1QEJX/zGZwAA2Dy1OsDy6u16dcO+R91/6TEAWFgd6wCx3fjQc++09Agb6ROP/sV3dfe+7ctdvgkAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQcQZAMAg4gwAYBBxBgAwiDgDABhEnAEADCLOAAAGEWcAAIOIMwCAQY5eegAAmKIvvnjpETbS7U45c+kRNtInDrLcljMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAwizgAABhFnAACDiDMAgEHEGQDAIOIMAGAQcQYAMIg4AwAYRJwBAAxy9NIDLKWqTk1yapJcK9deeBoAgJWr7Zaz7j6tu/d1975jcuzS4wAAJLkaxxkAwETiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMIs4AAAYRZwAAg4gzAIBBxBkAwCDiDABgEHEGADCIOAMAGEScAQAMUt299AyLq6rzknx86TkO4vgk5y89xAay3nbOOtsd6213rLeds852Z/J6u1V333j7QnE2XFWd2d37lp5j01hvO2ed7Y71tjvW285ZZ7uzievNbk0AgEHEGQDAIOJsvtOWHmBDWW87Z53tjvW2O9bbzllnu7Nx6817zgAABrHlDABgEHEGADCIOAMAGEScAQAMIs4AAAb5/wG6lClmNoGZJwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A3LLCx3ZE0Ls",
        "outputId": "33d1f675-54e3-441b-a99d-58697cc08bc4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 678
        }
      },
      "source": [
        "translate(u'¿todavia estan en casa?')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input: <start> ¿ todavia estan en casa ? <end>\n",
            "Predicted translation: do you in the car ? <end> \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmYAAAJwCAYAAAAjo60MAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5ymd13v//cnnSQGCCWEDmIUCNJWqiASMehBzg9BkJoQJT+atAMoFkA4wJGmQUQSWuhVKRZKaFIMByGAQCJJgBhCKAGBFNLIfs4f171mZtglu2F3ru/sPJ+Pxzz2vq97ZvYz1yOZ+7VXre4OAADz22XuAQAAmAgzAIBBCDMAgEEIMwCAQQgzAIBBCDMAgEEIMwCAQQgzAIBBCDMAgEEIMwCAQQizAVTVz1XVB6vqZnPPAgDMR5iN4bAkd0lyxMxzAAAzKjcxn1dVVZLTkhyX5LeSXLO7L5l1KIZRVddIssfSZd19+kzjALCD2WI2v7sk+Zkkj0nyoyS/Oes0zK6qrlhVr66q85N8PclXV3wAsJMSZvM7LMnbuvuHSd60eM769vwkN0/y/yW5IMkDkjwpyRlJ7jfjXADsYHZlzqiq9knyjST/o7s/WlW3SHJ8kgO7+/vzTsdcquqMJPdf/DdxdpJbdfepVXX/JEd0991mHhGAHcQWs3ndO8l3uvujSdLdn01ySpLfnXUq5nalJP+5ePyDJFdZPD4+yR1mmQhgjauqfarqIVV1xbln+UmE2bwenOR1K5a9Lsnhqz8KA/lykhsuHp+U5HcXJ4n8dpL/mm0qgLXtvklelem9d1h2Zc6kqq6T6UDuG3f3KUuWXzvTWZo36e6TZxqPGVXV45Nc0t0vqqq7JvnHJLtn+ofUY7v7xbMOCLAGVdWHkhyQ5IfdvWHuebZEmMHgquq6STYkOaW7Pz/3PABrTVVdP8nJSW6T5BOZjt09cc6ZtsSuzBlV1XUXu6g2+9pqz8OYuvv07v57UQZwuT04yUcXx3L/cwa+AoItZjOqqksynYH57RXLr5Lk29296zyTsdqq6glJXtLdFyweb1F3v3CVxgLYKVTVKUme1d3HVtW9kxyV5Do9YAQJsxlV1cYkB3T3WSuWXy/Jid29zzyTsdqq6qtJNnT3dxePt6S7+4Y/4XUAlqiqOyR5X5JrdPe5VbVHkm8muV93HzfvdD9ut7kHWI+q6kWLh53kOVX1wyUv75ppH/hnV30wZtPdN9jcYwB+aocleWd3n5sk3X1RVb0l0xUQhBlJkpst/qwkN05y0ZLXLkpyQqarv7MOVdUtFsdBAPBTqKo9M10m4/4rXnpdkvdW1b6bgm0UdmXOZHHQ/1syXcn9nLnnYRyLXdwnJnltkjd099dmHglgTaqqq2a6B/XrunvjitcelOT93f3NWYbbAmE2k6raNdN9EG8+6im7zKOqDkrywEz/wrthko9lirS3dfcP5pxtLlW1V5LHJjkkydWz4ozy7v7FOeYC2N6E2Yyq6tQk97Hbii2pqttmirT7JtkvyT919+/MO9Xqq6pXJrlXkrcmOTPT8Zn/rbv/fI65ALY3YTajqjos01aRB3X3d+aeh3EtAu2lSX5xPV5Gpar+K8l9u/v9c88CjG9xdvtWBc5oZ7o7+H9eT0xygyRfr6ozkpy39EW7Z9a3qrpBpq1lD0xyoyQfSfL7sw41nx8mcawdsLWW3rpu3yRPSPLJJMcvlt0+0xUQXrDKc10mW8xmVFVP+0mv2z2zPlXVozLF2G2TfCHT2UNv6O6vzzrYjKrqMUlumuThI14QEhhXVR2b5OTufvaK5U9JctPuftAsg22BMIPBVNXpSd6Y6Swit2FKUlX/kOROSX6Q6YzVi5e+3t33nGMuYHxVdXame2OeumL5jZKc0N37zTPZ5tmVCeO5nq1CP+Y7Sd4+9xDAmnRekrskOXXF8rtkOkxiKMJsRovbQvxJphMArptk96Wvr8eDvJnuuZQkVXXNTP9d7LHi9Y/MMdecuvuhc8/AuPwu5TL8ZZK/qaoNST6xWHa7THcEePpcQ22JMJvXM5PcL8lzMv2H86Qk10/yu0n+bL6xmNMiyN6YadddZ7pDxNItaN5kYDm/S9mi7n5uVZ2W6VqI910sPinJYd39ltkG2wLHmM1ocTrvI7r7PVV1TpJbdPeXq+oRSQ7p7vvMPCIzWNzD7SpJHpXk35LcPckBSZ6R5PEj3nR3NVTVQ3PpFpGVWxGHOt2d1eV3KTuTXS77U9iBDsh0IHOSnJvkSovH70ny67NMxAh+Jckfdvd/ZNpSdlZ3/32SP8y0ZWDdqaonZTqt/dOZtoS8I9MZq/sneeV8kzEIv0vZKlV1paraf+nH3DOtJMzmdXqSay4en5rk0MXj2yc5f5aJGMEVMh3sniT/lekWRMn0xrNer233sCRHdvdTMp2R+eLFmZgvSHK9WSdjBH6XskVVdb2qendVnZ/ku0nOWnx8Z/HnUBxjNq+3Z7r33yeSHJXkjVX1sCTXSvK8OQdjVv+R5BeSnJbks0keXlVfy7Rrc71ey+zamS4OmUxvtJtOb3/jYvnD5hiKYfhdyk/yqkxbUX8vm7ml22gcYzaQxW137pjpQnj/OPc8zKOqHphk9+4+tqpulWl3zFWSXJjpYNW3zjrgDKrqK5nuK3tCVf1bkld2999W1d2TvL67rzLziAykqm6X5A7xu5QkVXVuktt19xfmnmVrCLMZVdWdk/xrd/9oxfLdktxhPV4WgR9XVXtn2oJ2+nq9p2pVvTzJGd399Kp6eKYz7z6R5FZJ3tLdtpgBm1VVn09yeHd/eu5ZtoYwm1FVXZLkwO7+9orlV0nybdfegUlV7ZJkl03/iKmq+2WxdTnJ0d198U/6enZuVXXfJN/v7vctnj81yZFJvpjpDfkbc87HvKrqrkn+KMkjV179f0TCbEZVtTHJAd191orlByX51Gi3iWDHqaqtPrOwu4/YkbOMqKqum+RrK++IUFWV5Drdffo8kzGCqjoxyeO6+32L3f//muSpmS41883ufsCsAzKrxSVU9sx0DcgLkyzbSzXae62D/2dQVe9aPOwkr6uqC5e8vGuSgzP9YmH9uNqK53dOsjHJpntlHpzpLOr1unv7q0kOTPLtFcv3X7xm6/L6dr0kX1o8vleSdywuKvq+JO+dbywG8ei5B9gWwmwe3138WUm+l+Wnc1+U5GNJXrbaQzGf7v6tTY+r6imZ/pt4aHeft1i2T5JX5NJQW29W3v1gk32TXLDKszCeC5L8zOLxIbn02nY/WLKcdaq7Xz33DNvCrswZVdXTkjx/05svJElVfSPT1cpPXLH8pkk+0N3XmGey1VdVL1o8fFSmU96X3nB41yS3SXJRd99xtWdjHFX1jkzX//tYplswXb+7z6yqQ5O8qLt/ftYBmV1VHZDkwUl+Nsmfdfd3quqOSc7s7q/OO91yLjA7r2dmydayqrpGVf1+Vd1hxpmY37659GKZSx2YZO9VnmVuN1t8VJIbL3l+syQ3SnJCksPnGo5hPDrT3ob7JHl4d5+5WP4bsStz3auqW2fa1f3ATNcy23RM2d2SPGuuubbEFrMZVdW7k7ynu4+qqn0zXVh0n0xvzL/X3a+ZdUBmUVXHZtod86RMl4RIktsl+YskH+ruw+eZbD5V9aokj+3us+eeZRSLy6jcItOdIZb9I3txCy8gSVV9KMlHuvtpixMBbt7dX6mq2yd5U3cPdfcQYTajqjoryV27+/NV9ZBMp/PePFPVP6G71+vtd9a1qrpCplsNHZFk98XiH2U6xuyJ3f3DLX3terFYR3dMckp3/+fc86y2qvq1THc92NyFddulduBSVXV2phvbf2VFmF0/yX90916zDriCXZnz2jfJ9xePfz3J2xfXY/pgpv3grEPdfX53PzLTm+4tFx/7d/cj12uUVdWxVfXIxeM9Mt2G6X1JvlRVvzHrcPM4Ksk/Jbl2d++y4mPdRVlV7VFVf15VJ1fVBVV1ydKPuedjducnufJmlv9CfvxM79kJs3mdnuSOizPuDk1y3GL5/ll+kDPr0yWZLplxyeJjPTs0l+7WvWemM+2ukeTpi4/15vpJnrnkWKr17plJDsu0pXljpsMA/ibTGfCPnHEuxvDOJE+rqj0Xz3uxtewvkvzdXENtiTCb1wuTvDbJGZluTr3pGlV3zvq9LMK6V1W7VdXzMl1K5XOZ/lv4XlU9t6p2/8lfvdO6ci79l+3dk/zd4o4Zb0pyk9mmms/HkzjT8FL3zXTQ/9GZ/hHzzu5+TJKnZTrAm/XtiZk2eJyV6QSqjyU5NdPlVP50xrk2y3XMZtTdR1fVp5JcN8lx3b1x8dKXM53yzfr03CT3T/LwTL9AkuROSZ6T6R9TT5xprjl9M8nBi0uJHJrpdjvJdDjAerwd00uTPL+qrpkp3Jetg+4+YZap5nNAkk2Xlzk3yZUWj9+TaasI69jipKFfXtya6VaZfo+e0N3vn3eyzRNmM6mqKyb5xe7+aJKVN1b9fi79JcP684AkR3T3Py9Z9uXFySIvz/oMs1cmeXOSMzNtEfnAYvltM53NvN68bfHnMZt5rbP+7oRweqZLzJyeaUvIoZl+r94+yy/gzTqz9L22uz+Y6RjuTa/dMcmJ3f292QbcDGE2n41J3l1Vh3b3xzctrKqbZ/oP51qzTcbcrphpq+lKX86lWwLWle5+RlV9IdOtd97S3RctXvpR1ucWkRvMPcBg3p7pEjOfyHRixBur6mGZfo8+b87BmN2ae691jNlMuvucTAckPmTFSw9O8t7u/s7qT8UgPpfkMZtZ/tgkn13lWUZyfpJfS3JcVV1nsWyPTLuu1pXFJUJukukA93cn2bhYdrdMF95dV7r7Kd39rMXjtyX55SR/neS3u/tPZh2OWa3F91phNq/XJPmdxen/qapdMu3GOnbOoZjdk5McVlVfqqpXLz6+lORBmc42W3eq6oFJ3pLk5ExbizadBLFLpvW1rixZH6dk+frYNetzfTyrqh6+6Xl3/9/ufmGSa1fVM2ccjTGsqfdaYTav4zJtBbjH4vkhmbYA/MNsEw2qqnatqkdV1XrYhXNakoMyHUe07+LjrZnOwjt9vrFm9eQkD+vux2fafbnJJzJd/X69sT6We3CSz2xm+afz41tKdmpVdY+qelxVrZt76m6FNfVeK8xmtDgL83W59BfHg5O8eXGRWZbo7kuSHJzkGXPPsgq+muRH3f0n3X3vxcefJrlw8dp69HNJjt/M8nNz6X3v1hPrY7mrZ7oUwkrfzXTG5rpQVX+U6Xi7JyX5XFXdbOaRhrDW3muF2fxek+TuVXXdJPdK8uqZ55lFVX2oql5VVVdePH5XVR224tOOTXLXGcZbbZXpzLqV9k1ywSrPMoozM21FXOnO2fyJEjs762O50zNdUmalO2e6TuR68chM91m+VqaTII6rql+vqusuro944OK9Zj1aM++1zsqcWXd/cXG22euTnNHdn5x7ppl8IdO1qi5ePP6ZJH9TVbdeXCgymf4hse9M8+1wVfWixcNO8pyqWnr3h12T3Cbr9+D/Y5K8qKp+f/H8OlV1p0zXfHv6bFPNx/pY7ugkf7k4hmjT5RAOyXTtv/V01u7+WVyovLufvTiW6t2L134p0/vMQVl/l1NZU++1wmwMr0nyV0nW7dlD3f0HS57+QZJU1V8neU9VXS/J3yd5dJKPzjDeatm026GS3DjJRUteuyjJCUmev9pDjaC7n7u4HtFxSfZK8qFMu3af391/M+twM7A+luvuF1TVVZO8KNOxQ8n0/8xR3f3c+SZbdSdnOlv3tCTp7v9dVa9IcmCSkzLtytt7tunmtybea6t7c3tMWE1VtX+mGDm6u7859zwjqaqDkrwsyYZMBzYf3t1fm3eqHauqXpXksYurVbNEVe2d6Y1nl0wXhlx3l8pYyvpYbnHf4U236Dppva2Pqnp0kl/t7nvPPcuI1sp7rTADABiEg/8BAAYhzAAABiHMBlFVR849w0isj+Wsj+Wsj+Wsj+Wsj+Wsj+VGXx/CbBxD/4cyA+tjOetjOetjOetjOetjOetjuaHXhzADABjEuj8rc4/as/fKPnOPkYtzYXbPnnOPMQzrYznrYznrYznrYznrYznrY7lR1sc5+d53uvtqK5ev+wvM7pV9cts6ZO4xAIB15P39tv/c3HK7MgEABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAaxZsOsqv6xqo6dew4AgO1lzYYZAMDORpgBAAxiTYRZVe1dVcdW1blV9a2q+uMVr1+5ql5dVd+rqvOr6v1VddO55gUAuDzWRJgleX6SuyW5d5JDktwyyZ2XvH5sktsm+Z9JbpPkh0neU1VXWN0xAQAuv93mHuCyVNW+SX4vyRHd/d7FsocmOWPx+OeS3DPJr3T3RxbLHpzk9CQPTPLyzXzPI5McmSR7Ze9V+CkAAC7bWthi9rNJ9khy/KYF3X1uks8vnt44ycYVr/9g8fpNNvcNu/uY7t7Q3Rt2z547am4AgG2yFsLsp9FzDwAAsLXWQph9OcnFSW63aUFV7ZPk4MXTkzL9HLdf8vp+SW6W5MTVGxMA4KczfJgtdlu+IslfVNXdFmdbvjLJrovXT0nyziRHV9WdqupmSV6X5Owkb5hpbACAbTb8wf8LT0yyT5K3Zzrj8q8Xzzd5aJK/SvKuJHsl+XiSu3f3+as8JwDA5bYmwqy7z0vykMXH5l7/XpLDVnUoAIDtbPhdmQAA64UwAwAYhDADABiEMAMAGIQwAwAYhDADABiEMAMAGIQwAwAYhDADABiEMAMAGIQwAwAYhDADABiEMAMAGIQwAwAYhDADABiEMAMAGIQwAwAYhDADABiEMAMAGIQwAwAYhDADABiEMAMAGIQwAwAYhDADABiEMAMAGIQwAwAYxG5zDzCEXXade4JhnP27vzT3CEPZ678umXuEoXzvoN3nHmEoV/jOxrlHGMr+x39j7hGGsvEb35p7hKH0Jf5/WeaizS+2xQwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgELOHWVU9pKq+W1V7rlj++qp61+Lx/19Vp1bVRYs/H7bic7uq7rNi2WlV9cQd/xMAAGwfs4dZkrdmmuN/blpQVVdMcq8kr6iqeyV5cZK/SnJwkqOSvKSqfmuGWQEAdpjd5h6gu8+vqtcnOSLJWxaLH5Dk7CT/lORfkry2u1+8eO3kqrp1kj9M8g+X5++sqiOTHJkke2Xvn2J6AIDtZ4QtZknysiR3q6prL54fkeTV3f2jJDdO8vEVn/+xJDe5vH9Zdx/T3Ru6e8Pu2fOyvwAAYBUMEWbd/bkkJyQ5vKoOTrIhySsv68tWPK4Vr+++/SYEANjxhgizhZclOTzJ7yf5eHd/abH8pCR3XPG5v5zkxCXPz0py4KYnVXXA0ucAAGvB7MeYLfHGJC9M8ogkD1+y/HlJ3lpVn07yviR3T/LAJL+95HM+mORRVfWvSS5J8uwkF6zG0AAA28swW8y6+5xMB/9fmEtPAkh3vyPJHyR5fKatZI9N8sjuXnrg//9K8pUkH07ytiQvT/LtVRkcAGA7GWmLWTLtfnxzd5+3dGF3vzTJS7f0Rd19ZpLfWLH477b/eAAAO84QYVZVV05ypyS/nuTmM48DADCLIcIsyWeS7J/kj7v7C3MPAwAwhyHCrLuvP/cMAABzG+bgfwCA9U6YAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMYre5BxjCxkvmnmAYVz7uy3OPMJQ+4CpzjzCWg/afe4KhfP8g/7ZdqjYeOPcIQ7nyx3409whD2fiDs+ceYSwXbX6x3yoAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAg1izYVZVx1bVP849BwDA9rLb3AP8FB6bpOYeAgBge1mzYdbdP5h7BgCA7WnNhllVHZvkqt19j6r6cJITk3w/yZFJNiZ5TZInd/fG2YYEANgGa/YYs814YJIfJblDkkcneVyS+806EQDANtiZwuzE7n5qd5/c3W9J8qEkh2zuE6vqyKr6VFV96uJcuLpTAgBswc4UZv++4vmZSa6+uU/s7mO6e0N3b9g9e+74yQAAtsLOFGYXr3je2bl+PgBgJydcAAAGIcwAAAYhzAAABrFmr2PW3YcveXyXn/Q6AMBaYIsZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCB2m3sAxnLJWWfNPcJYrI9lrnn6fnOPMJQzDzt47hGG8uRnvm7uEYZy9O/cc+4RhtLf8vt0a9hiBgAwCGEGADAIYQYAMAhhBgAwCGEGADAIYQYAMAhhBgAwCGEGADAIYQYAMAhhBgAwCGEGADAIYQYAMAhhBgAwCGEGADAIYQYAMAhhBgAwCGEGADAIYQYAMAhhBgAwCGEGADAIYQYAMAhhBgAwCGEGADAIYQYAMAhhBgAwCGEGADAIYQYAMAhhBgAwCGEGADCIocOsqu5SVV1VV517FgCAHW2oMKuqD1fVi+eeAwBgDkOFGQDAejZMmFXVsUl+JcmjFrsvO8n1Fy/fvKr+b1X9sKo+VVW3WvG1d6iqf1m8/vWq+tuq2m91fwIAgJ/OMGGW5LFJjk/yqiQHLj6+tnjtOUn+KMmtknw3yeurqpKkqm6W5H1J3pXk5kl+O8ktkrxyNYcHAPhp7Tb3AJt09w+q6qIkP+zubyZJVf3C4uU/6+4PLZY9I8nHklwryRlJnpTkzd39gk3fq6oekeQzVXX17v72yr+rqo5McmSS7JW9d+BPBQCw9YYJs8vw70sen7n48+qZwuzWSW5UVfdb8jm1+PNnk/xYmHX3MUmOSZL9av/e7tMCAFwOayXMLl7yeFNI7bLkz5cn+cvNfN3Xd+RQAADb02hhdlGSXbfxa05IctPuPnUHzAMAsGpGOvg/SU5Lcpuquv7iorJbM99fLL7mpVV1y6q6UVXdo6qO3qGTAgBsZ6OF2fMzbTU7MclZSa57WV/Q3f+e5M6ZLq3xL0k+l+kszm/tsCkBAHaAoXZldvfJSW6/YvGxKz7ntFx6cP+mZZ9KcvcdORsAwI422hYzAIB1S5gBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxit7kHANaOS845Z+4RhnKtt31l7hGG8pIT7jP3CEO5y2uOn3uEobzz+Xede4SxvPoNm11sixkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCB2+jCrqt2qquaeAwDgsgwZZjX5X1V1SlVdWFVnVNVzFq/9n6r6UlWdX1WnVdVzq2qvJV/79Kr6QlUdXlVfTnJhkn3m+lkAALbWbnMPsAXPTvKIJE9I8pEkV0tyy8Vr5yU5IsnXk9wkyUszxdefLfn6GyR5QJLfSXJRkguWfvOqOjLJkUmyV/beUT8DAMA2GS7MqmrfJI9P8rjufuVi8alJjk+S7n7mkk8/raqeneSJWR5meyR5cHd/a3N/R3cfk+SYJNmv9u/t+xMAAFw+w4VZpq1geyb5wOZerKr7JHlckhsl2TfJrouPpc7YUpQBAIxqyGPMtqSqbpfkTUnem+S3Mu3e/NMku6/41PNWeTQAgJ/aiFvMTsp0zNghSU5Z8dodk3x96e7MqrreKs4GALDDDBdm3X1OVR2V5DlVdWGmg/+vkuTWSU5Ocq2qemCmY84OTXL/2YYFANiOhguzhack+V6mA/qvneRbSV7T3X9bVc9L8ldJrpDkfUmemuQlcw0KALC9DBlm3b0xyf9ZfKx87SmZwm2pv13y+tOTPH0HjgcAsEOsqYP/AQB2ZsIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQwgwAYBDCDABgEMIMAGAQu809ALCGdM89wVA2/uDsuUcYyq6fPWfuEYaya22ce4ShnH3DmnuENcEWMwCAQQgzAIBBCDMAgEEIMwCAQQgzAIBBCDMAgEEIMwCAQQgzAIBBCDMAgEEIMwCAQQgzAIBBCDMAgEEIMwCAQQgzAIBBCDMAgEEIM/+4d4oAAAhvSURBVACAQQgzAIBBCDMAgEEIMwCAQQgzAIBBCDMAgEEIMwCAQQgzAIBBCDMAgEEIMwCAQQgzAIBBCDMAgEEIMwCAQQgzAIBBCDMAgEHsVGFWVY+uqs9U1XlV9bWqesrcMwEAbK3d5h5gOzskyVOTfDHJnZO8vKq+2N3vmncsAIDLtlOFWXffa8nTr1TVs5PcaK55AAC2xU4VZktV1R8n2T3Jmzbz2pFJjkySvbL3Kk8GALB5O9UxZptU1Z8meVySu3X3mStf7+5juntDd2/YPXuu/oAAAJux020xq6prJnlGkv/R3Z+dex4AgK21M24xOzBJJTlp7kEAALbFzhhmJyX5pSQ/tgsTAGBkO2OYHZzkdUmuNvcgAADbYmcMs72T/HymMzIBANaMne7g/+7+cKZjzAAA1pSdcYsZAMCaJMwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAax29wDAKxVG88/f+4RxlL+rb/UB3/vdnOPMJST3vGSuUcYyq5P2/xy/xcBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADEKYAQAMQpgBAAxCmAEADGLNhFlVPbGqTpt7DgCAHWXNhBkAwM5uu4RZVe1XVVfaHt9rG/7Oq1XVXqv5dwIA7EiXO8yqateqOrSq3pDkm0luvlh+xao6pqq+XVXnVNW/VNWGJV93eFWdW1WHVNUXquq8qvpQVd1gxfd/clV9c/G5r0my74oRfjPJNxd/1x0v788BADCKbQ6zqrppVT03ydeSvDnJeUnunuQjVVVJ/inJtZLcI8ktk3wkyQer6sAl32bPJE9JckSS2ye5UpKXLvk77pvkfyd5WpJbJflSkiesGOX1SR6Q5GeSHFdVp1bVU1cG3hZ+hiOr6lNV9amLc+G2rgIAgB1iq8Ksqq5SVY+pqk8n+UySX0jy2CTX6O6HdfdHuruT/GqSWyS5T3d/srtP7e4/S/KVJA9e8i13S/Koxef8e5LnJ7nLIuyS5HFJXt3dR3f3yd39rCSfXDpTd/+ou/+5u++f5BpJnr34+0+pqg9X1RFVtXIr26avPaa7N3T3ht2z59asAgCAHW5rt5j9QZKjklyQ5KDuvmd3v7W7L1jxebdOsneSsxa7IM+tqnOTHJzkZ5d83oXd/aUlz89MskeSKy+e3zjJ8Su+98rn/627z+7uV3b3ryb5pSQHJHlFkvts5c8HADC73bby845JcnGShyT5QlW9Pclrk3yguy9Z8nm7JPlWkjtt5nucveTxj1a81ku+fptV1Z6Zdp0+KNOxZ1/MtNXtnZfn+wEAzGGrQqi7z+zuZ3X3zyf5tSTnJnlTkjOq6gVVdYvFp56QaWvVxsVuzKUf396GuU5KcrsVy5Y9r8kvV9XRmU4++Oskpya5dXffqruP6u7vbcPfCQAwq23eQtXdn+juRyQ5MNMuzoOS/FtV3SnJ+5N8PMk7q+o3quoGVXX7qvrzxetb66gkh1XVw6rq56rqKUluu+JzHpTkfUn2S3L/JNfp7id19xe29WcCABjB1u7K/DHdfWGStyV5W1VdPckl3d1V9ZuZzqh8WZKrZ9q1+fEkr9mG7/3mqrphkmdlOmbtXUlemOTwJZ/2gUwnH5z9498BAGDtqelkyvVrv9q/b1uHzD0GsBb994nkJEnKzWSW2XCTuScYynvf8dq5RxjKrgee+unu3rByuf+LAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAYhzAAABiHMAAAGIcwAAAax29wDAKxZ3XNPMJa+ZO4JxvLJz889wVAOveYt5h5hMKdudqktZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIMQZgAAgxBmAACDEGYAAIPYbe4B5lBVRyY5Mkn2yt4zTwMAMFmXW8y6+5ju3tDdG3bPnnOPAwCQZJ2GGQDAiIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAghBkAwCCEGQDAIIQZAMAgqrvnnmFWVXVWkv+ce44kV03ynbmHGIj1sZz1sZz1sZz1sZz1sZz1sdwo6+N63X21lQvXfZiNoqo+1d0b5p5jFNbHctbHctbHctbHctbHctbHcqOvD7syAQAGIcwAAAYhzMZxzNwDDMb6WM76WM76WM76WM76WM76WG7o9eEYMwCAQdhiBgAwCGEGADAIYQYAMAhhBgAwCGEGADCI/wffDFGMBOqmnwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DUQVLVqUE1YW",
        "outputId": "edc73db8-f65c-49b0-e0fb-7cdf93b386a8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 702
        }
      },
      "source": [
        "# wrong translation\n",
        "translate(u'trata de averiguarlo.')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Input: <start> trata de averiguarlo . <end>\n",
            "Predicted translation: they ll be happy . <end> \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhkAAAKICAYAAADdIOhtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZRtd13n/c83uRmEMI8JGoYw2cxwmUQmERDBfhRpWhAk5GmiKC0sHrBFVEBABIKKjd3MCQgoiNiA0CBgGGUKM4JAwKAQMIQxATOQfJ8/9rmkUtybVN3cX+1z6r5ea91Vp87eVfWts5Kqd+2xujsAAPvaAXMPAABsTyIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACG2DH3AACrpqoOTXLdJJ3k89191swjwVKyJQNgg6pqR1U9M8k3k3wsySeSfLOqnlFVB807HSwfWzIANu4ZSR6Q5NeSvHvx3B2TPC3TH22PmWkuWErl3iUAG1NVX01yTHe/cd3z907ywu4+fJ7JYDnZXQKwcZdL8vndPP/5JJff4llg6YkMgI37WJLf3M3zj0zy0S2eBZae3SUAG1RVd0ryxiRfTvK+xdO3S3JEknt197v39LGwPxIZAJtQVUck+Y0kN1w89ekk/6u7T51vKlhOIgMAGMIprAAXoapuudF1u/vDI2eBVWNLBsBFqKrzM13Zsy5m1e7uA7dgJFgZtmQAXLRrzz0ArCpbMgA2YHHZ8Kcm+fPu/uLc88AqEBkAG1RVZya5cXefMvcssApcjAtg496c5KfmHgJWhWMyADbubUn+sKpumuRDSb67dmF3v2aWqWBJ2V0CsEGLM032xNklsI7IAACGcEwGADCEYzIANqGqrpDkXkmOTHLw2mXd/QezDAVLyu4SgA2qqtsleUOSs5NcJdPdWA9fvH9Kd990xvFg6dhdArBxz0zy8iTXSHJWptNZj0xyUpKnzzgXLCVbMgA2qKq+neTW3f3ZqvpWktt396er6tZJXtHd15t5RFgqtmQAbNw5ax7/e5JrLh6fmeSIrR8HlpsDPwE27sNJbp3ks0nenuQpVXW1JA9K8vEZ54KlZHcJwAZV1c4kl+nuE6vqKklemuQOmaLjod39iVkHhCUjMpZQVV0vyfOSPNIPLQBWlWMyltNDktwlyTEzzwEAe82WjCVTVZXklCRvSfJzSY7o7vNmHQpIklTVJ5Ls8Yem62TAhTnwc/ncJcllkvxmpqsK/myS1885EPADr173/kFJbp7puIw/3/pxYLnZkrFkquqEJOd097FV9awk1+zu+808FnARquqxmf5ffcTcs8AyERlLpKouneQrSe7d3e+qqpsneW+Sw7v7W/NOB+xJVR2V5KTuvsLcs8AyceDncvnFJKd397uSpLs/muRzSX5p1qmAi3OnJN+bewi2p6q6dFX9SlVdbu5ZNssxGcvlwUletu65lyU5Oslzt3wa4EKq6nXrn8p0g7RbJHnS1k/EfuL+SV6Y5JFJnjPzLJtid8mSqKofS/IvSX68uz+35vkfzXS2yX/q7s/ONB6QpKqOX/fU+Um+luQfuvvvZxiJ/UBVnZjkakm+1907555nM0QGACypqrpWpivK3ibJ+5Lcsrs/NedMm+GYjCVSVUcurpOx22VbPQ8As3twknctjtF7Y6aLNa4MWzKWSFWdl+lMktPWPX+lJKd194HzTAYkSVX9S3Z/Ma5OclaSk5O8qLvXH7sBe6WqPpfkqd19QlX9YpJnJ/mxXpFf3rZkLJfK7n+AHZbpBxgwr+OTXDHTWV8vW/z73OK51yU5L8lrquq/zjYh20ZV/USmA4t3XQTu9UkuleSnZxtqk5xdsgSq6s8WDzvJ06pq7alwB2baF/fRLR8MWO86Sf6ou/9o7ZNV9VuZDs6+b1X9TpLfTvLKOQZkW3lIktd295lJ0t3nVNWrMp1x+JY5B9sou0uWwOLI4SS5c6aLb52zZvE5mc4uOW7tWSfA1quq72Q68O7kdc9fN8mHu/uyVXWDJB/q7sNmGZJtoaoOSfLVJA/o7jetef4nk7w5ydV2xccysyVjCXT3XRcHfL4qyTHdfcbcMwG79b0kd8x07MVad8wFF+M6MMl/bOVQbEuXyXRdjAudGt3d766qX820G33pI8OWjCVRVQdmOu7iZqt0ehLsT6rqcUl+P8mLk3xw8fStM22+fnJ3/1FVPTrJvbr77vNMCctDZCyRqjo5yf0WpyoBS6iqfinTXZJvuHjqn5M8u7tfuVj+I0m6ux2szX5PZCyRqnpIkgckeVB3nz73PABsrYs4TfqHdPd1Bo9ziTkmY7k8Jsm1k3y5qr6U5LtrF3b3TWeZCoCtsvbeJIcleXSSD2Q6KSBJbp/pjMNnbfFce0VkLJdXX/wqwFZanFFyne4+varOyEX8ldndl926ydiOuvsH8VBVJyR5enf/4dp1FscG3WiLR9srdpew8qrqrpl2Mx2Z5OC1y7r7p2YZim1jsRvzr7r77MXjPerul2zRWOwHNnLK9DyTbZwtGay0qjo6yXOT/G2SuyR5bZLrZ9rt9LLZBmPb2BUOVbUj0x1X39/dX593KvYT3830c239KdN3yQWnTC81kbFEqurgJI/PBX+VH7R2uXuX7NZjkjyiu1+42JT9uO7+QlU9JytwDjmro7u/X1WvyXRWichgK/xJkj+vqp2Z7sCaJLfLdCXQJ8411Ga4d8lyeXKm/3ieleT8JI9N8ueZfqD9+oxzLbPrJHnr4vHZmQ6USqaDp46eYyC2tY8lue7cQ7B/6O5nZLoL602S/PHi302SPKS7nz7nbBtlS8ZyuX+SX+vuN1XVcZmuWf/5qvp0krsned684y2lr2e6Ml6SfDnJjZN8PMmVkvzIXEOxbT0xybOq6glJPpQfPgPsG3MMxfbV3a/KdDXolSQylsvVkuy62ueZSS6/ePymJCtRrTN4V5J7JPlEpv8R/6yq7p7kblmRGwixUt6wePuaXPgsk113ULZLkyGq6vJZt/dhFaJWZCyXf01yxOLtyUnumemvpdvHvRD25BFJDl08flqS7ye5Q6bgeMpcQ7Ft3XXuAdh/VNU1Mx3Yfpdc+My5lYlap7Aukap6WpIzu/upVXW/JH+Z5EtJrpHkmd39+FkHBGDLVNU/ZNqifVySU7PuGi3d/Y455toMkbHEquq2mf4q/2x3/93c8yyjqjovyeHdfdq656+U5DRn5LCvVdVNkvxqkqMy3TX5K1X180m+2N0fmXc6tpOqOjPJ7br7k3PPsrecXbJEqupOi3PxkyTd/f7u/uMkb6qqO8042jKrPTx/SJJztnIQtr+qukemu69eI8lP5YKDi49K8oS55mLb+pdMP8tWlmMylsuJSQ5Pctq65y+3WOav8oXF7bSTafPhry2Kf5cDk9wx090xYV96cpJHd/f/WlyXZZe3J/n/5hmJbeyRSZ5WVb++/qqfq0JkLJddB/Osd6WsO1WO/PfF20ry35Kct2bZOUlOSfJrWzwT29+Nk7xxN89/I8kVt3gWtr/XZtqS8ZmqOjvTge0/4LLibEhVvW7xsJO8bPEf0y4HZvrB9o9bPtgS6+5rJ0lVnZjkvt39zZlHYv/wjUy7Sk5Z9/wtMx2kDfvSI+Ye4JISGcth1yWKK8k3c+HTVc9J8u4kL9jqoVZBdzulkK30iiTPrKr7Z/qjYEdV3TnT0f/HzzoZ2852uOGes0uWyOIqgsd1t10jm1BV109yv+z+LqzHzDIU21JVHZTkhCS/lOmPgvMXb1+R5OjuPm/PHw2bV1VXy3Rp8aOS/F53n15Vd0hyanf/y7zTXTyRsUSq6oAk6e7zF+9fPcl9knyqu+0u2Y2quneSv0nykSS3ynTk/1GZ9mO+q7v/84zjsU1V1VFJbpHpDL2PdPfnZh6JbaiqbpXkbZnOMrlRkhsubgD5xCTX7+4HzjnfRjiFdbm8IYsDGqvqsCQnJXlmkndU1a/MOdgS+4MkT+ru22e6QdqDk1wr003T3j7fWMuvqm5SVc+pqv9bVYcvnvv5qrrF3LMtq8Xrc1B3f767X93drxIYDHRckmd39y0y/Xzb5c2ZrqG09ETGctmZ5B8Wj++b5DtJrprkYZluac4Pu0GSVy4en5vkUt19Vqb4eNRsUy0513vYa69I8tWqeu5ikzWMdKskuzsu4yuZ7nW19ETGcjksybcWj++R5G+7+9xM4XHUbFMttzNywb1LvpILbsO9I8kVZploNey63sMv5MIXLXt7ktvMMtFquFqm4D8q0xbGL1TVU6rqhjPPxfb0H9n9z7Eb5oevp7SURMZy+dckd6iqS2e6Odquu4heMcn3Zptqub0/yU8uHr8hF9yG+/gk751tquXneg97obvP6O7ju/vumQ40fk6Sn0nyT1X1wXmnYxt6bZInVNWuq352VV0r0125/2auoTZDZCyXP07yF5nOt/9ykncunr9TpluZ88MeneR9i8dPTPL3SX4x011s/9tMM62CXdd7WM/1Hjaou0/NFBlPS/LxTK8d7EuPyRT9X0tyqUyXMzg5ybeT/O6Mc22Ys0uWzOJo4iOTvKW7z1w8d+8k3+ru98w63JJZ3OflHkne391fv7j1uUBVPT3Tpdfvn+RTmY4HOjzT6ZnHd/cfzDfd8ququyb55UxBmySvSfKy7j5xvqnYrqrqpzJF7AFJPtzdb515pA0TGUuiqi6X5Kbd/a7dLLtDptNYXdVynao6K9NpXafMPcsq2cP1Hg5I8vK43sMeVdUzM71mV03ypiQvS/K67j77Ij8QNmm7/E4QGUuiqi6T6cDFe67dYlFVN0vygSTX6O7T55pvWVXV+5M8fpXKfplU1XVywV9IrvdwMarqPZnC4pXd/Y2552H72i6/E0TGEqmqlyc5s7t/dc1zx2W66IqLSu1GVd0ryR9lOu3yQ1l3Izm/CC5QVS/e6LqulLpni910t8nurzD70lmGYlvaDr8TRMYSqap7JvnLJFfv7nMWVwD9UpJHdPdr5p1uOVXV+WveXfsfcyXp7j5wi0daWlX1+nVP3SnTbpJdBxXfONMWjXeuyg+wrVZVN0jy+iTXyfTf2HmZTpc+N8nZq3BXTFbHdvid4AZpy+Utmc6Lvk+mA8nulukvpfW/HLjAQ5P8Wy58q/dk+mV55NaPs7y6++d2Pa6qx2X6b+2hu+6Vszh1+kVxJtNFeXaSD2e6pPhXk9w8yeWS/O+syNH+rJSV/51gS8aSWRz1f4Pu/vmqemmSM7r7N+aea1lV1XlJDu/u09Y9f6Ukp9mSsXtV9ZUkd+vuT617/kZJ3tbdV59nsuVWVV9Pcufu/mRVfTvJbbr7M4s7sf7P7r7pzCOyzaz67wRbMpbPS5N8qKqOTPILmcqVPatceDfJLoclOWuLZ1klhyU5ItPpq2sdnul8fHavcsGF8b6W6Vojn8m0Cfu6e/oguARW+neCyFgy3f1PVfXJTKcSfqm7PzD3TMuoqv5s8bCTPK2q1l4R9cBMB+Z9dMsHWx1/k+T4qnpsLriY2e0yXUlwJfb1zuSTSW6W5AuZjvD/H4utaQ/LdJEk2KdW/XeCyFhOL03yp0keP/cgS+wmi7eV5Mdz4ftvnJNpv/lxWz3UCnl4kmdlulbGQYvnvp/pmAw349uzpya59OLx72a6lP2JSU7PdGEzNqGqPp3ket3td9FFW9nfCY7JWEJVdcVMt3x/Xnd/de55lllVHZ/kkd39nblnWUWLgz133Xzv87sOAmXjFv+/frP9MN20qnpEkit195PmnmWZrfLvBJEBAAzhBmkAwBAiAwAYQmQssao6du4ZVpHXbfO8ZnvH67Z3vG6bt6qvmchYbiv5H9US8Lptntds73jd9o7XbfNW8jUTGQDAEPv92SUH1yF96A9Oe18u5+bsHJRD5h5j5XjdNs9rtne8bnvH67Z5y/yanZFvnt7dV9ndsv3+AiiH5tK5ba3UVVoBYGm8tV/9xT0ts7sEABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBCzRkZV3aWquqquPOccAMC+t6WRUVVvr6rnbOXXBADmYXcJADDElkVGVZ2Q5M5JfmOxi6STXGux+GZV9f6q+l5VnVRVt1z3sT9RVe9YLP9yVf3vqrrsYtmvVNXXq+qQdR/z8qp63fjvDADYna3ckvHIJO9NcnySwxf//m2x7GlJfjvJLZN8PcnLq6qSpKpukuTvk7wuyc2S3DfJzZO8ePGxf53p+/h/dn2hqrpckl9I8qKh3xEAsEc7tuoLdfe3q+qcJN/r7q8mSVXdcLH497r7xMVzf5Dk3UmukeRLSR6b5JXd/axdn6uqHp7kI1V11e4+rapenuSYJK9arPLAJN9J8obdzVJVxyY5NkkOzaX27TcKACRZnmMyPr7m8amLt1ddvL1VkgdV1Zm7/iV5z2LZUYu3L0hy96r60cX7xyR5SXd/f3dfrLuf3907u3vnQTlkd6sAAJfQlm3JuBjnrnnci7cHrHn7wiR/spuP+3KSdPfHqurDSY6uqv+TZGeSBw2aFQDYgK2OjHOSHLjJj/lwkht198kXs94LkvxWkisneU93f2Yv5gMA9pGt3l1ySpLbVNW1Fhfg2sjXf/riY55bVbeoqutW1X2q6nnr1vvLJFdP8vA44BMAZrfVkXFcpq0Zn0rytSRHXtwHdPfHk9wp0+mu70jysUxno/z7uvXOyHTg59m54ABQAGAmW7q7pLs/m+T2654+Yd06pySpdc+dlORnNvAlDs90Jsp3935KAGBfWJYDPy+RqrpCkjsmuUema2kAADPbFpGR5CNJrpjkd7r7k3MPAwBsk8jo7mvNPQMAcGHLcjEuAGCbERkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEPsmHuAudVBO7Ljylebe4yVcuovHjX3CCvpSY86Ye4RVtJTnvKQuUdYOVd58xfmHmElnf+tb889wmr6jz0vsiUDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYYttERlWdUFV/t/4xADCPbRMZAMByERkAwBAiAwAYQmQAAEPsmHuAOVTVsUmOTZJDDzxs5mkAYHvaL7dkdPfzu3tnd+88+IAfmXscANiW9svIAADGExkAwBAiAwAYQmQAAENsm7NLuvvo3T0GAOZhSwYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAyxY+4B5taHHJRzrnfE3GOslCP+5vNzj7CSnvPh/zL3CCvpa//vuXOPsHK+9hNHzj3CSrrm/5l7ghX1xj0vsiUDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOsRGRU1dur6jlzzwEAbNxKRAYAsHpEBgAwxCpFxo6qenZVfXPx75lVdUCSVNXBVfX0qvpSVX2vqj5YVfece2AA2J+tUmT8cqZ5b5/kV5Mcm+RRi2XHJ7lzkgcmuXGSlyR5fVXdbIY5AYAkO+YeYBO+kuQ3u7uT/HNVXT/Jo6vqtUkekORa3f2vi3WfU1U/nSlGfn39J6qqYzNFSg455HJbMjwA7G9WaUvG+xaBsct7k1wjyU8mqSSfqqozd/1Lcu8kR+3uE3X387t7Z3fvPPigSw8fHAD2R6u0JeOidJJbJzl33fP/McMsAEBWKzJuW1W1ZmvG7ZKcmmmLRiW5enefONt0AMCFrNLukiOS/GlV3aCq7pfksUn+pLs/m+TlSU6oqvtV1XWqamdVPaaq7jvrxACwH1ulLRkvT3Jgkvdn2j3yoiR/slj20CSPT/KMJD+a5BtJPpDElg0AmMlKREZ332XNu4/YzfJzkzxx8Q8AWAKrtLsEAFghIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwxI65B5hbffes7Pjgp+ceY6WcP/cAK2rHty839wgr6cr/eKm5R1g513zo5+YeYSV9/BvXm3uE1fTGPS+yJQMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABjiEkVGVb29qp6zr4YBALYPWzIAgCFEBgAwxL6IjAOq6g+r6vSqOq2qjquqA5Kkqh5UVR+sqjMWy/66qq6x6wOr6i5V1VV1n6r6aFWdVVUfqqpbrVnn6Ko6s6p+rqo+u1jnxKq6zmL5tarq/KrauXaoqnrYYqaD98H3CABs0r6IjF9O8v0kP5HkEUkeleS/LpYdnOQJSW6W5D5JrpzkL3fzOY5L8j+S7EzyhSR/V1WXWrP8kMXneWiS2yc5MMlrqqq6+5Qkb0lyzLrPeUySv+jucy7h9wcA7IV9ERmf6u7f7+7PdverkpyY5G5J0t0v7u43dvcXuvsDSR6e5I5V9aPrPseTu/vN3f3JTCHxI0keuGb5jiSP7O73dPdHkjw4yU12fZ0kL0jygKo6NEmq6seT3C7Ji/bB9wcA7IV9ERkfX/f+qUmumiRVdcuqem1VfbGqzkhy0mKdI9d9zHt3PejuM5N8Isl/WrP8/CQfWLPOFxdfZ9c6r01yTpL7Lt4/JskHFtHyQ6rq2Ko6qapOOidnb+y7BAA2ZV9Exrnr3u9Mx2lcOsmbk3wv05aHWyf5mcU6e3OcRO9xQfe5SV6a5Jiq2rH4envcitHdz+/und298+AcshejAAAXZ+TZJTfMdAzG73T3O7v7n7PYwrEbt9v1YBEnN07y6TXLD0hymzXrHJnkiHXrvDDJXZP8epLLJPmrffA9AAB7aWRk/GuSs5M8oqquU1X3TvLkPaz7u1V196q6UZIXZ9r18Yo1y7+f5E+r6vZVdfMkL0nyT0neumuF7v5MkncneWaSV3f3d/b5dwQAbNiwyOjuryV5SJKfT/KpTGeHPHoPq/92kmcl+XCS6yW5T3d/d83ys5M8NdMukfdnmvu+3b1+F8qLMu2KccAnAMxsxyX54O6+y26eO3rN41cmeeW6VWo3n+ofu/umF/O1XpvpAM+LcniSz3X3Oy9mPQBgsEsUGcuiqg5Lcs0kj8y0xQMAmNl2uaz4czLtanlPkufNPAsAkJkjo7vf3t3V3adfxDondPdhF/N5ju7uQ7r7v3T39/f9pADAZm2XLRkAwJIRGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGCI6u65Z5jVZeuKfdu629xjAOwzBxx66NwjrKT/+4X3zT3CSjrw8JM/1N07d7fMlgwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBA75h5gDlV1bJJjk+TQXGrmaQBge9ovt2R09/O7e2d37zwoh8w9DgBsS/tlZAAA44kMAGCIbRsZVfWIqvrnuecAgP3Vto2MJFdOcoO5hwCA/dW2jYzufmJ319xzAMD+attGBgAwL5EBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGGLH3AMAsG+df9ZZc4+wku55jVvMPcKKOnmPS2zJAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAxiJMZ0AAAYJSURBVAAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQKxMZVfWYqjpl7jkAgI1ZmcgAAFbLPomMqrpsVV1+X3yuTXzNq1TVoVv5NQGAjdvryKiqA6vqnlX1iiRfTXKzxfOXq6rnV9VpVXVGVb2jqnau+bijq+rMqrpbVX2yqr5bVSdW1bXXff7fqqqvLtZ9aZLD1o3ws0m+uvhad9jb7wMAGGPTkVFVN6qqZyT5tySvTPLdJD+T5J1VVUnekOQaSe6T5BZJ3pnkH6rq8DWf5pAkj0tyTJLbJ7l8kueu+Rr3T/KUJE9Icsskn0ny6HWjvDzJA5NcJslbqurkqvr99bECAMxjQ5FRVVeqqt+sqg8l+UiSGyZ5ZJKrd/fDuvud3d1J7prk5knu190f6O6Tu/v3knwhyYPXfModSX5jsc7HkxyX5C6LSEmSRyV5SXc/r7s/291PTfKBtTN19/e7+43d/YAkV0/yh4uv/7mqentVHVNV67d+7Pp+jq2qk6rqpHNz9kZeAgBgkza6JeO/J3l2krOSXL+7/3N3/3V3n7VuvVsluVSSry12c5xZVWcmuXGSo9asd3Z3f2bN+6cmOTjJFRbv/3iS96773Ovf/4Hu/k53v7i775rk1kmuluRFSe63h/Wf3907u3vnQTnkIr5tAGBv7djges9Pcm6SX0nyyar62yR/keRt3X3emvUOSPLvSe64m8/xnTWPv79uWa/5+E2rqkMy7Z55UKZjNf4p09aQ1+7N5wMALrkN/VLv7lO7+6ndfYMkP53kzCR/leRLVfWsqrr5YtUPZ9qKcP5iV8naf6dtYq5PJ7nduucu9H5NfrKqnpfpwNP/meTkJLfq7lt297O7+5ub+JoAwD606S0H3f2+7n54ksMz7Ua5fpIPVtUdk7w1yXuSvLaq7lVV166q21fVkxbLN+rZSR5SVQ+rqutV1eOS3HbdOg9K8vdJLpvkAUl+rLsf292f3Oz3BADsexvdXfJDuvvsJK9O8uqqumqS87q7q+pnM50Z8oIkV820++Q9SV66ic/9yqq6TpKnZjrG43VJ/jjJ0WtWe1umA0+/88OfAQCYW00nhey/LltX7NvW3eYeA4C5/eAERzbjref/9Ye6e+fulrmsOAAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADLFj7gEAYCl0zz3BtmNLBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgiB1zDzCHqjo2ybFJcmguNfM0ALA97ZdbMrr7+d29s7t3HpRD5h4HALal/TIyAIDxRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGEJkAABDiAwAYAiRAQAMITIAgCFEBgAwhMgAAIYQGQDAECIDABhCZAAAQ4gMAGAIkQEADCEyAIAhRAYAMITIAACGEBkAwBAiAwAYQmQAAEOIDABgCJEBAAwhMgCAIUQGADCEyAAAhhAZAMAQIgMAGKK6e+4ZZlVVX0vyxbnn2IMrJzl97iFWkNdt87xme8frtne8bpu3zK/ZNbv7KrtbsN9HxjKrqpO6e+fcc6war9vmec32jtdt73jdNm9VXzO7SwCAIUQGADCEyFhuz597gBXldds8r9ne8brtHa/b5q3ka+aYDABgCFsyAIAhRAYAMITIAACGEBkAwBAiAwAY4v8HVxS6MxHbzkEAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x720 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RTe5P5ioMJwN"
      },
      "source": [
        "## Next steps\n",
        "\n",
        "* [Download a different dataset](http://www.manythings.org/anki/) to experiment with translations, for example, English to German, or English to French.\n",
        "* Experiment with training on a larger dataset, or using more epochs\n"
      ]
    }
  ]
}