deakin_ai_challenge_training.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BgZ5rVkYMZNX",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
    "#  Copyright (c) 2023. Mohamed Reda Bouadjenek, Deakin University              +\n",
    "#           Email:  reda.bouadjenek@deakin.edu.au                              +\n",
    "#                                                                              +\n",
    "#  Licensed under the Apache License, Version 2.0 (the \"License\");             +\n",
    "#   you may not use this file except in compliance with the License.           +\n",
    "#    You may obtain a copy of the License at:                                  +\n",
    "#                                                                              +\n",
    "#                 http://www.apache.org/licenses/LICENSE-2.0                   +\n",
    "#                                                                              +\n",
    "#    Unless required by applicable law or agreed to in writing, software       +\n",
    "#    distributed under the License is distributed on an \"AS IS\" BASIS,         +\n",
    "#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  +\n",
    "#    See the License for the specific language governing permissions and       +\n",
    "#    limitations under the License.                                            +\n",
    "#                                                                              +\n",
    "# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**Notebook author:** [Mohamed Reda Bouadjenek](https://rbouadjenek.github.io/), Lecturer of Applied Artificial Intelligence,\n",
    "\n",
    "**Institution:** Deakin University, School of Information Technology, Faculty of Sci Eng & Built Env\n",
    "\n",
    "**Adress:** Locked Bag 20000, Geelong, VIC 3220\n",
    "\n",
    "**Phone:** +61 3 522 78380\n",
    "\n",
    "**Email:** reda.bouadjenek@deakin.edu.au\n",
    "\n",
    "<img style=\"float: left;\" src=\"images/deakin2.png\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "**Welcome to the Notebook for the Deakin Simpsons Challenge 2023!**\n",
    "\n",
    "![](images/Simpsons_cast.png)\n",
    "\n",
    "\n",
    "This Notebook allows you to build a classification model for The Deakin Simpsons challenge 2023. The **Deakin Simpsons challenge 2023** is a computer vision competition for which the goal is:\n",
    "\n",
    "\n",
    "> **Given an image of simpsons and a natural language question about the image, the task is to provide an accurate natural language answer using machine learning and deep learning.**\n",
    "\n",
    "The challenge is designed to provide students with the opportunity to work as team members, to compete with each other, and to enhance the student learning experience by improving their AI modeling, problem-solving, and team-working skills.\n",
    "\n",
    "\n",
    "\n",
    "As participants, your goal is to build a machine learning/deep learning model to answer a natural language **Yes**/**No** question given an image of Simpsons using machine learning and deep learning.\n",
    "\n",
    "\n",
    "Once you have built your model, you will have to submit it on the [CodaLab](https://codalab.lisn.upsaclay.fr/competitions/317?secret_key=1aab176a-2a97-46da-a214-711ff65b7e01) platform to be evaluated.\n",
    "We evaluate the performance of your model using the [Accuracy](https://visualqa.org/evaluation.html)  on a private test set that we have directly collected and labeled from TV show episodes.\n",
    "Once the evaluation completed, your entry will appear on the leaderboard to see your performance against other competitors.\n",
    "\n",
    "\n",
    "In the following, we will take you through  a 6-step process to build a simple model to perform this task as follows:\n",
    "\n",
    "1. `Setup the environment:` Thie first step consists of setting the environement and downloading the data.\n",
    "2. `Preprocessing:` The second step is a preprocessing step that consists of resizing, plitting, and piping the input data.\n",
    "3. `Exploring the data:` The third step consists of a simple data exploration step where you will see samples of the data and some statistics to help you in understanding the data.\n",
    "4. `Designing the model:` The forth step consists of designing an architecture for the task.\n",
    "5. `Traning:` The fifth step consists of starting the training process.\n",
    "6. `Monitoring:` The sixth step consists of monitoring the traning process to investigate possible overfitting.\n",
    "7. `Submitting:` The seventh and last step will take you through the submission process.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "We evaluate the performance of your model using the [Accuracy](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html)  on a private test set that we have directly collected and labeled from TV show episodes.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**References:**\n",
    "\n",
    "- [VQA: Visual Question Ansewering](https://visualqa.org/)\n",
    "- [VQA: Visual Question Answering (ICCV 2015)](https://arxiv.org/pdf/1505.00468.pdf)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Setup the environment\n",
    "\n",
    "First, you need now to load all the required packages for this Notebook.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "03iC8WrlKsG5"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "from tensorflow.python.keras.saving import hdf5_format\n",
    "import matplotlib.pyplot as plt\n",
    "import json\n",
    "import os\n",
    "import numpy as np\n",
    "from sklearn.utils import shuffle\n",
    "from tensorflow.keras.utils import plot_model\n",
    "import random\n",
    "import h5py\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "##################\n",
    "# Verifications:\n",
    "#################\n",
    "print('GPU is used.' if len(tf.config.list_physical_devices('GPU')) > 0 else 'GPU is NOT used.')\n",
    "print(\"Tensorflow version: \" + tf.__version__)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oeWhcxZhMkCB"
   },
   "source": [
    "# Downloading Dataset\n",
    "\n",
    "Now, please run the following cell to download the dataset that you will use to build your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lj-czauoLQEe"
   },
   "outputs": [],
   "source": [
    "# Traning Images\n",
    "data_url = \"http://206.12.93.90:8080/simpson2022_dataset/scene_img_abstract_v002_train2015.tar.gz\"\n",
    "zip_path = keras.utils.get_file(\"scene_img_abstract_v002_train2015.tar.gz\", data_url, extract=True)\n",
    "imgs_path_train = os.path.dirname(zip_path) + '/scene_img_abstract_v002_train2015/'\n",
    "\n",
    "#  Validation Images\n",
    "data_url = \"http://206.12.93.90:8080/simpson2022_dataset/simpsons_validation.tar.gz\"\n",
    "zip_path = keras.utils.get_file(\"simpsons_validation.tar.gz\", data_url, extract=True)\n",
    "imgs_path_val = os.path.dirname(zip_path) + '/simpsons_validation/'\n",
    "\n",
    "# Traning Questions\n",
    "data_url = \"http://206.12.93.90:8080/simpson2022_dataset/OpenEnded_abstract_v002_train2015_questions.zip\"\n",
    "zip_path = keras.utils.get_file(\"OpenEnded_abstract_v002_train2015_questions.zip\", data_url,\n",
    "                                cache_subdir='datasets/OpenEnded_abstract_v002_train2015_questions/', extract=True)\n",
    "q_train_file = os.path.dirname(zip_path) + '/OpenEnded_abstract_v002_train2015_questions.json'\n",
    "\n",
    "#  Validation Questions\n",
    "data_url = \"http://206.12.93.90:8080/simpson2022_dataset/questions_validation.zip\"\n",
    "zip_path = keras.utils.get_file(\"questions_validation.zip\", data_url,\n",
    "                                cache_subdir='datasets/questions_validation/', extract=True)\n",
    "q_val_file = os.path.dirname(zip_path) + '/questions_validation.json'\n",
    "\n",
    "# Traning Annotations\n",
    "data_url = \"http://206.12.93.90:8080/simpson2022_dataset/abstract_v002_train2015_annotations.zip\"\n",
    "zip_path = keras.utils.get_file(\"abstract_v002_train2015_annotations.zip\", data_url,\n",
    "                                cache_subdir='datasets/abstract_v002_train2015_annotations/', extract=True)\n",
    "anno_train_file = os.path.dirname(zip_path) + '/abstract_v002_train2015_annotations.json'\n",
    "\n",
    "# Validation Annotations\n",
    "data_url = \"http://206.12.93.90:8080/simpson2022_dataset/annotations_validation.zip\"\n",
    "zip_path = keras.utils.get_file(\"annotations_validation.zip\", data_url,\n",
    "                                cache_subdir='datasets/annotations_validation/', extract=True)\n",
    "anno_val_file = os.path.dirname(zip_path) + '/annotations_validation.json'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sbnt1nGcM579"
   },
   "source": [
    "# Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Define global variable that will be used for this challenge.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# We define the size of input images to 64x64 pixels.\n",
    "img_width = 32\n",
    "img_height = 32\n",
    "image_size = (img_height, img_width)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<span style=\"color:red;font-weight:bold;font: 24px\">Warning: Please do not change the cell below!</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answers = ['yes', 'no']\n",
    "num_answers = len(answers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "<span style=\"color:red;font-weight:bold;font: 24px\">End Warning!</span>\n",
    "\n",
    "<span style=\"color:red;font-weight:bold;font: 24px\">----------------------------------------------------------------</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EQN0Lj-VLU7p"
   },
   "outputs": [],
   "source": [
    "# Read the data files\n",
    "q_train = json.load(open(q_train_file))\n",
    "q_val = json.load(open(q_val_file))\n",
    "anno_train = json.load(open(anno_train_file))\n",
    "anno_val = json.load(open(anno_val_file))\n",
    "\n",
    "'''\n",
    "    Filter questions for which the answers are not in the set of possible answers.\n",
    "'''\n",
    "\n",
    "\n",
    "def filter_questions(questions, annotations, answers, imgs_path):\n",
    "    # Make sure the questions and annotations are alligned\n",
    "    questions['questions'] = sorted(questions['questions'], key=lambda x: x['question_id'])\n",
    "    annotations['annotations'] = sorted(annotations['annotations'], key=lambda x: x['question_id'])\n",
    "    q_out = []\n",
    "    anno_out = []\n",
    "    imgs_out = []\n",
    "    q_ids = []\n",
    "    question_ids_set = set()\n",
    "    # Filter annotations\n",
    "    for annotation in annotations['annotations']:\n",
    "        if annotation['multiple_choice_answer'] in answers:\n",
    "            question_ids_set.add(annotation['question_id'])\n",
    "            q_ids.append(annotation['question_id'])\n",
    "            anno_out.append(answers.index(annotation['multiple_choice_answer']))\n",
    "    # Filter images and questions\n",
    "    for q in questions['questions']:\n",
    "        if q['question_id'] in question_ids_set:\n",
    "            # Preprocessing the question\n",
    "            q_text = q['question'].lower()\n",
    "            q_text = q_text.replace('?', ' ? ')\n",
    "            q_text = q_text.replace('.', ' . ')\n",
    "            q_text = q_text.replace(',', ' . ')\n",
    "            q_text = q_text.replace('!', ' . ').strip()\n",
    "            q_out.append(q_text)\n",
    "            file_name = str(q['image_id'])\n",
    "            while len(file_name) != 12:\n",
    "                file_name = '0' + file_name\n",
    "            file_name = imgs_path + questions['data_type'] + '_' + questions['data_subtype'] + '_' + file_name + '.png'\n",
    "            imgs_out.append(file_name)\n",
    "    return imgs_out, q_out, anno_out, q_ids\n",
    "\n",
    "\n",
    "imgs_train, q_train, anno_train, q_ids_train = filter_questions(q_train, anno_train,\n",
    "                                                                answers, imgs_path_train)\n",
    "imgs_train, q_train, anno_train, q_ids_train = shuffle(imgs_train, q_train,\n",
    "                                                       anno_train, q_ids_train, random_state=0)\n",
    "\n",
    "imgs_val, q_val, anno_val, q_ids_val = filter_questions(q_val, anno_val,\n",
    "                                                        answers, imgs_path_val)\n",
    "imgs_val, q_val, anno_val, q_ids_val = shuffle(imgs_val, q_val,\n",
    "                                               anno_val, q_ids_val, random_state=0)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now prepare the vocabulary to be used.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the vocabulary\n",
    "vocab_size = 15000\n",
    "vocab = {}\n",
    "for q in q_train:\n",
    "    q = q.split(' ')\n",
    "    for token in q:\n",
    "        v = vocab.get(token, 0)\n",
    "        vocab[token] = v + 1\n",
    "vocab = list(dict(sorted(vocab.items(), key=lambda x: -x[1])[0:vocab_size]).keys())\n",
    "# Mapping tokens to integers\n",
    "token_to_num = keras.layers.StringLookup(vocabulary=vocab)\n",
    "# Mapping integers back to original tokens\n",
    "num_to_token = keras.layers.StringLookup(vocabulary=token_to_num.get_vocabulary(),\n",
    "                                         invert=True)\n",
    "vocab_size = token_to_num.vocabulary_size()\n",
    "print(f\"The size of the vocabulary ={token_to_num.vocabulary_size()}\")\n",
    "print(\"Top 20 tokens in the vocabulary: \", token_to_num.get_vocabulary()[:20])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "baZecOrUNKKa"
   },
   "source": [
    "## Defining Map Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "    This function is used to process and encode a single sample.\n",
    "'''\n",
    "\n",
    "\n",
    "def encode_single_sample(img_file, q, anno):\n",
    "    ###########################################\n",
    "    ##  Process the Image\n",
    "    ##########################################\n",
    "    # 1. Read image file  \n",
    "    img = tf.io.read_file(img_file)\n",
    "    # 2. Decode the image\n",
    "    img = tf.image.decode_jpeg(img, channels=3)\n",
    "    # 3. Convert to float32 in [0, 1] range\n",
    "    img = tf.image.convert_image_dtype(img, tf.float32)\n",
    "    # 4. Resize to the desired size\n",
    "    img = tf.image.resize(img, [img_height, img_width])\n",
    "    ###########################################\n",
    "    ##  Process the question\n",
    "    ##########################################\n",
    "    # 5. Split into list of tokens\n",
    "    word_splits = tf.strings.split(q, sep=\" \")\n",
    "    # 6. Map tokens to indices\n",
    "    q = token_to_num(word_splits)\n",
    "    # 7. Return an inputs to for the model\n",
    "    return (img, q), anno\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tngi5B4uNb61"
   },
   "source": [
    "## Creating `Dataset` objects\n",
    "\n",
    "We create our `tf.data.Dataset` object that returns a new dataset containing the transformed elements, in the same order as they appeared in the input. The function `encode_single_sample` is used to change both the values and the structure of a dataset's elements.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dicaNwv2LufX",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# We define the batch size\n",
    "batch_size = 128\n",
    "# Define the trainig dataset\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices(\n",
    "    (imgs_train, q_train, anno_train)\n",
    ")\n",
    "train_dataset = (\n",
    "    train_dataset.map(encode_single_sample, num_parallel_calls=tf.data.AUTOTUNE)\n",
    "        .padded_batch(batch_size)\n",
    "        .prefetch(buffer_size=tf.data.AUTOTUNE)\n",
    ")\n",
    "# Define the validation dataset\n",
    "val_dataset = tf.data.Dataset.from_tensor_slices(\n",
    "    (imgs_val, q_val, anno_val)\n",
    ")\n",
    "val_dataset = (\n",
    "    val_dataset.map(encode_single_sample, num_parallel_calls=tf.data.AUTOTUNE)\n",
    "        .padded_batch(batch_size)\n",
    "        .prefetch(buffer_size=tf.data.AUTOTUNE)\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualize the data\n",
    "\n",
    "Let's visualize an example in our dataset, including the image, the question, and the corresponding answer.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  Let's check results on some validation samples\n",
    "for batch in val_dataset.shuffle(random.randint(0, 200)).take(1):\n",
    "    imgs_batch = batch[0][0]\n",
    "    q_batch = batch[0][1]\n",
    "    ans_batch = batch[1]\n",
    "    answer_texts = [answers[i] for i in ans_batch]\n",
    "\n",
    "    q_orig = []\n",
    "    for q in q_batch:\n",
    "        q = num_to_token(q).numpy()\n",
    "        q = [t.decode(\"utf-8\") for t in q]\n",
    "        q = list(filter(('[UNK]').__ne__, q))\n",
    "        q.insert(int(len(q) / 2), '\\n')\n",
    "        q = ' '.join(q)\n",
    "        q_orig.append(q)\n",
    "\n",
    "    _, ax = plt.subplots(4, 4, figsize=(16, 16))\n",
    "    for i in range(16):\n",
    "        img = (imgs_batch[i, :, :, :] * 255).numpy().astype(np.uint8)\n",
    "        title = f\"Q: {q_orig[i]}\\n Ans: {answer_texts[i]}\"\n",
    "        ax[i // 4, i % 4].imshow(img)\n",
    "        ax[i // 4, i % 4].set_title(title)\n",
    "        ax[i // 4, i % 4].axis(\"off\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ew9ISxDuL7K-",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# This model is the deeper LSTM Q from Figure 8 in\n",
    "# https://arxiv.org/pdf/1505.00468.pdf\n",
    "def build_model(img_size, vocab_size, num_answers):\n",
    "    # Define the VGG19 conv_base for image input\n",
    "    img_input = keras.Input(shape=img_size + (3,), name=\"input_image\")\n",
    "    img = layers.Flatten()(img_input)\n",
    "    img = layers.Dense(64, activation='relu')(img)\n",
    "    #Define RNN for language input\n",
    "    q_input = keras.Input(shape=(None,), name=\"input_question\")\n",
    "    q = layers.Embedding(input_dim=vocab_size, output_dim=20)(q_input)\n",
    "    q = layers.SimpleRNN(64)(q)\n",
    "    # Combine CNN and RNN\n",
    "    mrg = layers.Multiply()([img, q])\n",
    "    # Output    \n",
    "    output = layers.Dense(num_answers, activation='softmax', name=\"output\")(mrg)\n",
    "    vqa_model = keras.Model(inputs=[img_input, q_input], outputs=output)\n",
    "    vqa_model.compile(keras.optimizers.Adam(learning_rate=0.001),\n",
    "                      loss='sparse_categorical_crossentropy',\n",
    "                      metrics=['accuracy'])\n",
    "    return vqa_model\n",
    "\n",
    "\n",
    "# Get the model\n",
    "model = build_model(image_size, vocab_size, num_answers)\n",
    "model.summary()\n",
    "plot_model(model, show_shapes=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training and Evaluating\n",
    "Let's now starting the training process.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Train the model\n",
    "history = model.fit(\n",
    "    train_dataset,\n",
    "    epochs=1,\n",
    "    validation_data=val_dataset\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Save the created model. It is important to note that we are saving with the model:\n",
    "1. The `answers`: which is important for submission as we will submit answers and not indexes.\n",
    "1. The `token_to_num`: which is important for submission as il allows to map tokens to indexes.\n",
    "2. The `image_size`: which is important to resize the test images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "model.save('model.h5', overwrite=True, save_format='h5')\n",
    "with h5py.File('model.h5', mode='a') as f:\n",
    "    f.create_dataset('answers', data=answers)\n",
    "    f.create_dataset('vocab', data=vocab)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Monitoring and analysis of the model\n",
    "\n",
    "The next step consists of monitoring the traning process to investigate possible overfitting.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def learning_plots(history):\n",
    "    plt.figure(figsize=(15, 4))\n",
    "    ax1 = plt.subplot(1, 2, 1)\n",
    "    for l in history.history:\n",
    "        if l == 'loss' or l == 'val_loss':\n",
    "            loss = history.history[l]\n",
    "            plt.plot(range(1, len(loss) + 1), loss, label=l)\n",
    "\n",
    "    plt.title('Training and validation loss')\n",
    "    plt.xlabel('Epochs')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.legend()\n",
    "\n",
    "    ax2 = plt.subplot(1, 2, 2)\n",
    "    for k in history.history:\n",
    "        if 'accuracy' in k:\n",
    "            loss = history.history[k]\n",
    "            plt.plot(range(1, len(loss) + 1), loss, label=k)\n",
    "    plt.title('Training and validation accuracy')\n",
    "    plt.xlabel('Epochs')\n",
    "    plt.ylabel('Accuracy')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "learning_plots(history)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Run the `classification_report` function below to build a text report showing the main classification metrics for your model:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_prob = model.predict(val_dataset)\n",
    "y_pred = y_prob.argmax(axis=-1)\n",
    "print(classification_report(anno_val, y_pred, target_names=answers))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference and visualization\n",
    "\n",
    "Let's check results on some validation samples\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for batch in val_dataset.shuffle(20).take(1):\n",
    "    imgs_batch = batch[0][0]\n",
    "    q_batch = batch[0][1]\n",
    "    ans_batch = batch[1]    \n",
    "    answer_texts = [answers[i] for i in ans_batch]\n",
    "    preds = model.predict(batch[0])\n",
    "    pred_texts = [answers[i] for i in np.argmax(preds, axis=1)]\n",
    "    q_orig = []\n",
    "    for q in q_batch:\n",
    "        q = num_to_token(q).numpy()\n",
    "        q = [t.decode(\"utf-8\") for t in q]\n",
    "        q = list(filter(('[UNK]').__ne__, q))\n",
    "        q.insert(int(len(q) / 2), '\\n')\n",
    "        q = ' '.join(q)\n",
    "        q_orig.append(q)\n",
    "    _, ax = plt.subplots(2, 4, figsize=(16, 16))\n",
    "    for i in range(8):\n",
    "        example_input = (np.array([imgs_batch[i]]), np.array([q_batch[i]]))\n",
    "        title = f\"Q: {q_orig[i]}\\n Pred Ans: {pred_texts[i]} \\n True Ans: {answer_texts[i]}\"\n",
    "        ax[i // 4, i % 4].imshow(imgs_batch[i])\n",
    "        ax[i // 4, i % 4].set_title(title)\n",
    "        ax[i // 4, i % 4].axis(\"off\")\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Observe where are the errors!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to submit the model to CodaLab?\n",
    "\n",
    "Congratulation, you have sucessfully created your model and you need now to submit it to CodaLab!\n",
    "\n",
    "CodaLab is an open-source web-based platform that enables researchers, developers, and data scientists to organize and participate to data science ana machine learning competitions.\n",
    "\n",
    "Submitting your model to CodaLab is very simple. You need to follow the following steps:\n",
    "\n",
    "1. Mount your Google Drive by runing this code:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "2. Copy the model to your Google Drive by runing the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!cp model.h5 'drive/My Drive/'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "3. Now, go to your Google Drive and you will find a file called `model.h5`.\n",
    "\n",
    "\n",
    "\n",
    "4. Copy the model file `model.h5` you have created to the directory `deakin_ai_challenge_submission`.\n",
    "\n",
    "\n",
    "\n",
    "5. Zip `deakin_ai_challenge_submission/` to generate `deakin_ai_challenge_submission.zip`.\n",
    "\n",
    "\n",
    "6. Submit `deakin_ai_challenge_submission.zip` to CodaLab  following these steps.\n",
    "\n",
    "\n",
    "6. Your submission will appear here. Just wait until it is processed (it may take time)!\n",
    "\n",
    "\n",
    "***Watch the video below or click [here](https://www.youtube.com/watch?v=UVP4yATavFM):***\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "\n",
    "YouTubeVideo('UVP4yATavFM', width=1000, height=600)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "**Congratulation on building your model and submitting to CodaLab! We hope that your model will achieve a high accuracy on the testset.**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Acknowledgment\n",
    "\n",
    "\n",
    "**Author:** [Mohamed Reda Bouadjenek](https://rbouadjenek.github.io/), Lecturer of Applied Artificial Intelligence,\n",
    "\n",
    "**Institution:** Deakin University, School of Information Technology, Faculty of Sci Eng & Built Env\n",
    "\n",
    "**Adress:** Locked Bag 20000, Geelong, VIC 3220\n",
    "\n",
    "**Phone:** +61 3 522 78380\n",
    "\n",
    "**Email:** reda.bouadjenek@deakin.edu.au\n",
    "\n",
    "**www.deakin.edu.au**\n",
    "\n",
    "<div>\n",
    "<img style=\"float: left;\" src=\"images/deakin2.png\" width=\"200\" >\n",
    "</div>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "<div>  <a href=\"https://twitter.com/DeakinAI2021\" > <img style=\"float: left;\" src=\"https://irisconnect.com/uk/wp-content/uploads/sites/3/2020/12/twitter-Follow-us-button.png\" width=\"200\" > </a>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "deakin_ai_challenge_training.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}