acse-xm421_coursework_II.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/acse-xm421/ACSE-19-20/blob/master/acse-xm421_coursework_II.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://drive.google.com/uc?id=1dFgNX9iQUfmBOdmUN2-H8rPxL3SLXmxn\" width=\"400\"/>"
      ],
      "metadata": {
        "id": "X0Zu1667pzmz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### ***Name***: Xuefei Mi\n",
        "### ***CID***: 02084183"
      ],
      "metadata": {
        "id": "zpGtjUPip2FW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Instructions:\n",
        "\n",
        "Follow the instructions below to complete the coursework and submit it:\n",
        "\n",
        "<br>\n",
        "\n",
        "1. Complete your coursework using this provided Jupyter Notebook template (use Google Colab or your local machine. Your copy of the notebook should be named: `yourusername_coursework_II.ipynb`. And don't forget to fill in the two fields at the top of this notebook with your name and CID.\n",
        "\n",
        "<br>\n",
        "\n",
        "2. Once you have completed your answers, upload your final notebook to the repo you got from the github classroom link. Make sure to have all the answers in there:\n",
        "\n",
        "   - **All the cells in your final Jupyter Notebook should be executed before saving and uploading to github in order to have the output of the cells available in the uploaded version** (images you plot, outputs of calculations, etc). We will not rerun code blocks in the notebooks, it is your responsibility to run them before uploading the notebook.\n",
        "\n",
        "   - Add comments in the code to explain what you are doing at every step in the coding cells.\n",
        "\n",
        "   - All answers requiring written answers (ie, not code) should be in markdown blocks in the Jupyter Notebook. This provided Jupyter Notebook template has allocated blocks for the questions, but **you can add any coding or markdown blocks you need**.\n",
        "\n",
        "<br>\n",
        "\n",
        "3. The coursework is released on **Friday 16 December at 14:00h UK time**, and the answers have to be submitted by **Friday 16 December, 17:00h UK time**. We will not accept late submissions.\n",
        "\n",
        "4. If you have questions during the coursework, come to me directly and I will address them. Usually, they will be relevant to the rest of the students and if this is the case, I will answer them for everybody. **DO NOT ask questions in the chat as this may give answers away to other students, if you prefer to, send the questions via chat to me privately, but do not use any of the general chats please.**\n",
        "\n",
        "<br>\n",
        "\n",
        "<br>\n",
        "\n",
        "### The coursework consists of 8 questions you have to complete. You will find them below. \n",
        "\n",
        "- This is an open-book assessment, so feel free to browse the materials we have seen in the module to assist you with your answers.\n",
        "\n",
        "<br>\n",
        "\n",
        "---\n",
        "\n",
        "<br>"
      ],
      "metadata": {
        "id": "odUp8tEbqChU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## provided imports\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from torchsummary import summary\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "from torchvision.datasets import CelebA\n",
        "from torchvision.transforms import Compose, Resize, ToTensor\n",
        "from torchvision.utils import make_grid\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "device='cpu'"
      ],
      "metadata": {
        "id": "4tRBb6hZLId5"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Question 1** [10 points]\n",
        "\n",
        "Suppose we have a feed-forward network (FFN) designed to predict which class an image belongs to. Given the following architecture and input characteristics:\n",
        "\n",
        "- Images are single channel and have a size of 100x105.\n",
        "- The FFN has 4 hidden layers with 1001, 2002, 1345, and 1002 output nodes (output size of each layer).\n",
        "- The FFN output size is 10 nodes.\n",
        "- There are bias terms everywhere.\n",
        "- The activation function in all layers except the output layer are ReLU.\n",
        "\n",
        "Answer the following questions:\n",
        "#### **1.1** Implement this network as an nn.Module."
      ],
      "metadata": {
        "id": "_unMIndvslgY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class FFNq1(nn.Module):\n",
        "    def __init__(self):\n",
        "      super(FFNq1, self).__init__()\n",
        "      self.fc1 = nn.Linear(100*105, 1001)\n",
        "      self.fc2 = nn.Linear(1001, 2002)\n",
        "      self.fc3 = nn.Linear(2002, 1345)\n",
        "      self.fc4 = nn.Linear(1345, 1002)\n",
        "      self.output = nn.Linear(1002, 10)\n",
        "      self.act = nn.ReLU()\n",
        "      self.actout = nn.Softmax()\n",
        "        ### your code here\n",
        "\n",
        "    def forward(self, x):\n",
        "      x = torch.flatten(x, start_dim = 1)\n",
        "\n",
        "      x = self.act(self.fc1(x))\n",
        "      x = self.act(self.fc2(x))\n",
        "      x = self.act(self.fc3(x))\n",
        "      x = self.act(self.fc4(x))\n",
        "      x = self.actout(self.output(x))\n",
        "\n",
        "      return x\n",
        "      \n",
        "x= torch.rand(1, 1, 100, 105)\n",
        "model = FFNq1()\n",
        "output = model(x)\n",
        "\n",
        "\n",
        "        ### your code here"
      ],
      "metadata": {
        "id": "-AY2N62tzzUN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b2ef7938-0526-46ca-8e7d-83bd594c7046"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-8-17a4f347213b>:20: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
            "  x = self.actout(self.output(x))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "your answers here"
      ],
      "metadata": {
        "id": "djNJex3Mz30T"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **1.2** How many trainable parameters does the network have? Use a code block to calculate it and explain the process (where you get all the numbers you multiply and add to find the answer) in comments or in a markdown block.\n",
        "\n",
        "Calculate the solution explicitly. If you want to use `torchsummary`, that's ok, **but only use it to validate that your answer is correct**. You will be assessed on the explicit calculation only."
      ],
      "metadata": {
        "id": "pnz_TzIf9AWl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sum(p.numel() for p in model.parameters())"
      ],
      "metadata": {
        "id": "V-fK39O69CJU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d389284a-c9f5-486e-f4f4-83065e3e4b5a"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "16570262"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# set_seed(42)\n",
        "\n",
        "# Check number of parameters\n",
        "nparam_layer1 = ((1*100*105+1) * (1001)) \n",
        "nparam_layer2 = ((1001+1) * 2002)\n",
        "nparam_layer3 = ((2002+1) * 1345)\n",
        "nparam_layer4 = ((1345+1) * 1002)\n",
        "nparam_layer5 = ((1002+1) * 10)\n",
        "\n",
        "print(nparam_layer1 + nparam_layer2 + nparam_layer3+nparam_layer4+nparam_layer5)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aVzbeuBBTmij",
        "outputId": "a3e7bf8e-2d8f-4881-a2cc-265525aaa48a"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "16570262\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "your answers here"
      ],
      "metadata": {
        "id": "kQf4acI19FW2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **1.3** How would you compute the loss? Get an instance of the loss you would use (call it `criterion`) and explain what are the operations that this loss performs on the outputs.\n",
        "\n"
      ],
      "metadata": {
        "id": "FKJFKQ9u0II_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "criterion = nn.CrossEntropy()\n"
      ],
      "metadata": {
        "id": "4q4qUvpA0DvA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The cross-entropy is a loss function that can be used to quantify the difference between two probability distributions.[1] The output is the probability(p) of different classes, and we compare it with our true y, eg(1,0,0,0,0,0,0,0,0,0)\n",
        "\n",
        "Loss_i = -y_i*log(p_i)\n",
        "Loss = (1/N) * sum(Loss_i)"
      ],
      "metadata": {
        "id": "QtzFWZl00DvB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **1.4** How do you make predictions once the network is trained?\n",
        "\n",
        "To clarify the question: how do you decide the class of a new image based on the output of the network?"
      ],
      "metadata": {
        "id": "vL_e3VG70KXB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Pick the class with the biggest probability."
      ],
      "metadata": {
        "id": "ihnb63rW0D-C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\\\\\n",
        "\n",
        "---\n",
        "\n",
        "\\\\"
      ],
      "metadata": {
        "id": "CpeGLPZ7zxpq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Question 2** [10 points]\n",
        "\n",
        "Suppose we have a convolutional neural network (CNN). Given the following architecture and input characteristics:\n",
        "\n",
        "- Images have three channels, each with size 48x48\n",
        "- The `nn.Module` of the network is:"
      ],
      "metadata": {
        "id": "4ChJgUdv0aMm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class CNN(nn.Module):\n",
        "  def __init__(self):\n",
        "    super(CNN, self).__init__()\n",
        "    self.c1 = nn.Conv2d(3, 12, kernel_size=3, stride=1, padding=1) \n",
        "    self.s2 = nn.MaxPool2d(kernel_size=2, stride=2)               \n",
        "    self.c3 = nn.Conv2d(12, 24, kernel_size=3, stride=1, padding=1)           \n",
        "    self.s4 = nn.MaxPool2d(kernel_size=2, stride=2)               \n",
        "    self.c5 = nn.Conv2d(24, 1, kernel_size=3, stride=1, padding=1)                           \n",
        "    self.act = nn.ReLU()                                          \n",
        "    \n",
        "  def forward(self, x):\n",
        "    x = self.act(self.c1(x))                                      \n",
        "    x = self.act(self.s2(x))                                      \n",
        "    x = self.act(self.c3(x))                                     \n",
        "    x = self.act(self.s4(x))                                     \n",
        "    x = self.c5(x)\n",
        "    return x             \n",
        "\n",
        "\n",
        "                         "
      ],
      "metadata": {
        "id": "6qqifoRO1MRg"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Answer the following questions:\n",
        "\n",
        "#### **2.1** How many trainable parameters does this network have?\n",
        "\n",
        "Calculate the solution explicitly. If you want to use `torchsummary`, that's ok, **but only use it to validate that your answer is correct**. You will be assessed on the explicit calculation only."
      ],
      "metadata": {
        "id": "AYYbqfJ34zfz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "x= torch.rand(1, 3, 48, 48)\n",
        "model = CNN()\n",
        "output = model(x)\n",
        "\n",
        "sum(p.numel() for p in model.parameters())\n"
      ],
      "metadata": {
        "id": "J7yr6Uzb4-Xv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "49577fcc-ef3f-4641-8702-060e47cabd98"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3169"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "nparam_layer1 = (3*3*3+1)*12\n",
        "nparam_layer2 = (3*3*12+1)*24\n",
        "nparam_layer3 = (3*3*24+1)*1\n",
        "\n",
        "print(nparam_layer1 + nparam_layer2 + nparam_layer3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZY8draDoXS4F",
        "outputId": "1deb28d8-7478-4f56-ccde-cee40ad5563d"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "3169\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from torchsummary import summary\n",
        "summ = summary(model, torch.Size((3, 48,48)))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R6IWzkr5ZdcU",
        "outputId": "dd23de81-f984-41ce-b95e-02ba10ec3138"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "----------------------------------------------------------------\n",
            "        Layer (type)               Output Shape         Param #\n",
            "================================================================\n",
            "            Conv2d-1           [-1, 12, 48, 48]             336\n",
            "              ReLU-2           [-1, 12, 48, 48]               0\n",
            "         MaxPool2d-3           [-1, 12, 24, 24]               0\n",
            "              ReLU-4           [-1, 12, 24, 24]               0\n",
            "            Conv2d-5           [-1, 24, 24, 24]           2,616\n",
            "              ReLU-6           [-1, 24, 24, 24]               0\n",
            "         MaxPool2d-7           [-1, 24, 12, 12]               0\n",
            "              ReLU-8           [-1, 24, 12, 12]               0\n",
            "            Conv2d-9            [-1, 1, 12, 12]             217\n",
            "================================================================\n",
            "Total params: 3,169\n",
            "Trainable params: 3,169\n",
            "Non-trainable params: 0\n",
            "----------------------------------------------------------------\n",
            "Input size (MB): 0.03\n",
            "Forward/backward pass size (MB): 0.79\n",
            "Params size (MB): 0.01\n",
            "Estimated Total Size (MB): 0.83\n",
            "----------------------------------------------------------------\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "your answers here"
      ],
      "metadata": {
        "id": "tId4sSky6EsZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **2.2** Does the number of trainable parameters you have calculated depends on the size of the input?\n",
        "\n",
        "Provide a short explanation of your answer (one or two sentences)"
      ],
      "metadata": {
        "id": "ORbVS8wD55TB"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "RRvZ9_706HRf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "No. Because the filter size only depends on the number of filters used in the convolutional layers and the size of the filters. Plus there is no FC layer, so it has nothing to do with input size."
      ],
      "metadata": {
        "id": "Bkah2neO6HRg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\\\\\n",
        "\n",
        "---\n",
        "\n",
        "\\\\"
      ],
      "metadata": {
        "id": "C2det5QI6ZCD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Question 3** [10 points]\n",
        "\n",
        "Given a dataset with 10000 samples, answer the following questions:\n",
        "\n",
        "#### **3.1** The dataset is split in three: training dataset (80%), validation dataset (10%), and test dataset (10%). Which one of these statements is incorrect? Explain in one sentence why it is incorrect (don't need to explain the two that are correct).\n",
        "\n",
        "- a) The training dataset is used to update the model parameters, but not the hyperparameters.\n",
        "- b) The validation dataset is used to optimise hyperparameters, and after this is done, we can combine it with the training set for a final training with hyperparameters fixed.\n",
        "- c) The test dataset that will be used to assess generalisation can also be used to optimise hyperparameters, but not for training weights and biases.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "DNXortmm6sV1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Answers (select one option in each case and explain the one that is False):\n",
        "- a) True\n",
        "- b) True\n",
        "- c) False. The testset can only be used for assessing generalisation, or, it will happen data leak, resulting in inaccurate test score."
      ],
      "metadata": {
        "id": "_4_gGUS7M7St"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **3.2** How many iterations (model updates) will I have done in total if I train the network using the training dataset for 200 epochs?\n",
        "\n",
        "- i) Full-batch gradient descent\n",
        "- ii) Mini-batch gradient descent with a batch size of 99\n",
        "- iii) Pure stochastic gradient descent (batch size of 1)"
      ],
      "metadata": {
        "id": "QbRpBJ4kMyxe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### Answers\n",
        "total_num_iterations_i = 200\n",
        "total_num_iterations_ii = (int(8000/99)+1)*200 = 16200\n",
        "total_num_iterations_iii = 8000*200 = 1600000"
      ],
      "metadata": {
        "id": "MzUHiEpGNTc7"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **3.3** If the dataset is composed of images with three channels and 50x50 dimensions per channel and they are used as the input of a CNN where the first layer produces 25 feature maps of size 50x50 each. What would be the effect on the feature maps after applying a `nn.Dropout(0.2)` layer? And if we use `nn.Dropout2d(0.2)` layer instead?\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "slMHbhxHAftM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### your code (if you need it) here"
      ],
      "metadata": {
        "id": "IuHp1BHTSvJt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "For nn.Dropout(0.2), 20% of elements in the feature map will be dropout.\n",
        "\n",
        "For nn.Dropout2d, it will drop out entire channels or spatial regions of the feature maps instead of each elements. So 5 feature map will be dropout."
      ],
      "metadata": {
        "id": "W5FC35MCwwnj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\\\\\n",
        "\n",
        "---\n",
        "\n",
        "\\\\"
      ],
      "metadata": {
        "id": "2rmNi_BPSxJU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Question 4** [15 points]\n",
        "\n",
        "#### Given the network we used in class to illustrate how backpropagation works:\n",
        "\n",
        "<br>\n",
        "\n",
        "<center><img src=\"https://drive.google.com/uc?id=1YT8y2gqkDsrwbNyi0Lmo3YSHh3kcrf6H\" width=\"600\"/></center>\n",
        "\n",
        "[link to the figure](https://drive.google.com/file/d/1YT8y2gqkDsrwbNyi0Lmo3YSHh3kcrf6H/view?usp=share_link) in case you cannot see it in the notebook.\n",
        "\n",
        "<br>\n",
        "\n",
        "#### Continue the derivation we saw in class to calculate the expression for the gradient of $w_1$:\n",
        "\n",
        "<br>\n",
        "\n",
        "$$\\frac{\\partial C}{\\partial w_1}$$\n",
        " \n",
        "#### where\n",
        "\n",
        "$$C = \\frac{1}{2}(a_4-y)^2$$\n",
        "\n",
        "<br>\n",
        "\n",
        "#### and assume that the activation function $g$ is now a sigmoid function $\\sigma$. Choose what is the correct expression for the gradient of $w_1$ from the list below:\n",
        "\n",
        "<br>\n",
        "\n",
        "- a) $(a_4 -y)\\;\\sigma(z_4)\\;(1-\\sigma(z_4))\\;w_3\\; \\sigma(z_3)\\;(1-\\sigma(z_3))\\; w_2 \\;\\sigma(z_2)\\;(1-\\sigma(z_2))\\; a_1$\n",
        "\n",
        "- b) $(a_4 -y)\\;\\sigma(z_4)\\;(1-\\sigma(z_4))\\;w_3\\; \\sigma(z_3)\\;(1-\\sigma(z_3))\\; a_2 \\;\\sigma(z_2)\\;(1-\\sigma(z_2))\\; a_1$\n",
        "\n",
        "- c) $(a_4 -y)\\;\\sigma(z_4)\\;w_3\\; \\sigma(z_3)\\; w_2 \\;\\sigma(z_2)\\; a_1$\n",
        "\n",
        "- d) $(a_4 -y)\\;\\sigma(z_4)\\;w_3\\; \\sigma(z_3)\\; w_2 \\;\\sigma(z_2)\\; w_1$\n",
        "\n",
        "- e) *none of the above*"
      ],
      "metadata": {
        "id": "2zXwL_gGUIvS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "your answer here (pick an option and explain why it is the correct one). You don't have to add any mathematical derivation."
      ],
      "metadata": {
        "id": "IORLIxwIYeQn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "a. Assume there no z1. because $a2 = w1*a1+b1, z2 = \\sigma(a2), a3 = w2*z2+b2$, and so on. So when using chain rule, we need to pass through several a and z. eg, The derivative of $$\\frac{\\partial a_3}{\\partial z_2} = w_2$$, which is not associated with a2.\n",
        "And also, we nned to calculate $$\\frac{\\partial z_2}{\\partial a_2} = \\sigma(a2)*(1-\\sigma(a2))$$, which is missing in c and d options. \n",
        "So I choose a."
      ],
      "metadata": {
        "id": "u77kYZVaeaSC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\\\\\n",
        "\n",
        "---\n",
        "\n",
        "\\\\"
      ],
      "metadata": {
        "id": "KcT48iUmYytK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Question 5** [10 points]\n",
        "\n",
        "Identify and briefly explain the errors in the following code snippets:\n",
        "\n",
        "#### **5.1** The following network implementation is designed to take 1-channel two-dimensional inputs (1-chanel images) and classify them into 6 different classes. The code below has 4 errors. Find them and correct them so that you can run your network with a dummy input (code provided at the end of the cell). Indicate with comments in the code where you have made your changes:\n",
        "\n",
        "***\\[ you don't need to add any additional blocks in this question (5.1), only use the code block to make your changes and add comments on it\\]***"
      ],
      "metadata": {
        "id": "O688qxS2Y1T-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class NetworkQ5(nn.Module):\n",
        "  def __init__(self):\n",
        "    super(NetworkQ5, self).__init__()\n",
        "\n",
        "    self.c1 = nn.Conv2d(1, 12, kernel_size=5, stride=1, padding=2) #p4: it is a image, so it should be 2d. the channel cannot be taken into dimension.\n",
        "    self.s2 = nn.MaxPool2d(kernel_size=2, stride=2)               \n",
        "    self.s2_dr = nn.Dropout2d(p=0.2)\n",
        "    self.c3 = nn.Conv2d(12, 32, kernel_size=5, stride=1)           \n",
        "    self.c3_bn = nn.BatchNorm2d(32) # p1:the batchnorm is following c3, so its num_features should be 32.\n",
        "    self.s4 = nn.MaxPool2d(kernel_size=2, stride=2)               \n",
        "    self.c5 = nn.Linear(800, 120)   # p5: the flattened output of the last layer should be [1, 800], I have corrected it.                          \n",
        "    self.f6 = nn.Linear(120, 84)  \n",
        "    self.output = nn.Linear(84, 5)  # p2:we need to classify them into 6 classes, so the out_features in the final layer should be 6.\n",
        "    self.act = nn.ReLU()                         \n",
        "    self.actout = nn.Softmax()  # p3:when do classification, we need to use softmax() to convert our output to the probability of different classes.               \n",
        "    \n",
        "  def forward(self, x):\n",
        "    x = self.act(self.c1(x))                                      \n",
        "    x = self.act(self.s2_dr(self.s2(x)))                     \n",
        "    x = self.act(self.c3(x))\n",
        "    x = self.c3_bn(x)\n",
        "    x = self.act(self.s4(x))                      \n",
        "    print('before flatten',x.shape)               \n",
        "    x = x.view(-1, x.size(1)*x.size(2)*x.size(3))               \n",
        "    print('after',x.shape) \n",
        "    x = self.act(self.c5(x))                                    \n",
        "    x = self.act(self.f6(x))                                    \n",
        "    return self.actout(self.output(x))   #  the same as p3                                  \n",
        "  \n",
        "x = torch.randn((1, 1, 28, 28)).to(device)\n",
        "model = NetworkQ5().to(device)\n",
        "y = model(x)\n",
        "print(y)"
      ],
      "metadata": {
        "id": "GArYKVEFZAJ-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2a98734d-c34b-41a2-fa55-e2cec8190b17"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "before flatten torch.Size([1, 32, 5, 5])\n",
            "after torch.Size([1, 800])\n",
            "tensor([[0.2313, 0.2009, 0.1854, 0.2302, 0.1523]], grad_fn=<SoftmaxBackward0>)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-26-c9432a451062>:28: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.\n",
            "  return self.actout(self.output(x))   #  the same as p3\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **5.2** The following training loop has 3 errors. Identify and fix them.\n",
        "\n",
        "**This code snippet is not designed to be executed, it will not run**. The question is: what are the three conceptual errors in this training loop. Identify them in the code and add a comment next to their corresponding lines, and explain in the markdown block below why you think they are errors. You can assume that the following objects exist and you can use them: `data_loader`, `model`, `device`, `criterion`, and `optimizer`. Once you have identified them, fix them."
      ],
      "metadata": {
        "id": "sM_Gv9vO0sft"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for epoch in range(10):  \n",
        "\n",
        "    running_loss = 0.0\n",
        "    model.train() # e1: in the training loop we use model.train() instead of model.eval, so the parameters can be updated using backpropogation.\n",
        "    for X, y in data_loader:\n",
        "        # with torch.no_grad(): #e2: the same as e1, in the training loop, we want the gradient to update the parameters, so we cannot set as no_grad.\n",
        "        X, y = X.to(device), y.to(device)\n",
        "        optimizer.zero_grad() # e3: the optimizer should be rest at the start of each batch, instead of in the end. Or we cannot update the parameters.\n",
        "        output = model(X)\n",
        "        loss = criterion(output, y)\n",
        "        loss.backward()\n",
        "        optimizer.step()"
      ],
      "metadata": {
        "id": "Er-iTSFWyyXz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The specific answers are in the comment above."
      ],
      "metadata": {
        "id": "JD4M2k2TCv5Z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>\n",
        "\n",
        "---\n",
        "\n",
        "<br>"
      ],
      "metadata": {
        "id": "OsHWORWkDALq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Question 6** [15 points]\n",
        "\n",
        "Given an RNN, with a single layer, that has been trained and is used to generate new outputs, like the one in the figure below:\n",
        "\n",
        "<center><img src=\"https://drive.google.com/uc?id=1GmEyTghDKdkBx0gbFrw_KOn6zuKbNaHc\" width=\"500\"/></center> \n",
        "\n",
        "[link to the figure](https://drive.google.com/file/d/1GmEyTghDKdkBx0gbFrw_KOn6zuKbNaHc/view?usp=share_link) in case you cannot see it in the notebook.\n",
        "\n",
        "<br>\n",
        "\n",
        "If the dimension of my hidden vector is 20, and the dimension of my input and output is 35.\n",
        "\n",
        "#### **Q6.1** What are the sizes of the matrices $W_{hh}$, $W_{xh}$ and $W_{hy}$? Use the next text block to write your answers:\n"
      ],
      "metadata": {
        "id": "GFQa2f5i08tM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "$W_{hh}$:\n",
        "- ... rows=20\n",
        "- ... columns=20\n",
        "\n",
        "$W_{xh}$:\n",
        "- ... rows=20\n",
        "- ... columns=35\n",
        "\n",
        "$W_{hy}$:\n",
        "- ... rows=35\n",
        "- ... columns=20"
      ],
      "metadata": {
        "id": "6ZyyKTsDDECy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Q6.2** Given that this RNN network is trained and that we want to use it to generate new samples (words or characters), what operations do you perform on the outputs $y$ to generate a new sample in the sequence?"
      ],
      "metadata": {
        "id": "3T7WxNoCDSL5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Input that output as the new input of RNN, repeat until I have generated the desired number of samples."
      ],
      "metadata": {
        "id": "Ec4Rmg5aDvcA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Q6.3** Indicate which of the following statements is False, and explain why:\n",
        "\n",
        "- a) LSTMs have two vectors that are passed: the cell state and the hidden state. The cell state is responsible for keeping 'longer term' memory in the system.\n",
        "\n",
        "- b) The activations functions of the various gates in an LSTM are `tanh`, so that they can act as 'continuous switches' to pass or not pass information.\n",
        "\n",
        "- c) LSTMs cells can be stacked to form deeper LSTMs architectures, in the same way that we can stack cells in RNNs."
      ],
      "metadata": {
        "id": "unxjg_qDH3nu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "B.  The activations functions of the various gates in an LSTM is sigmoid. Because its output is (0,1), it can act as the gate closed or open.\n",
        "\n",
        "tanh in LSTM is just used to make gradient more stable in case of gradient vanishing or explosion. "
      ],
      "metadata": {
        "id": "mhD1EcAiTFHf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>\n",
        "\n",
        "---\n",
        "\n",
        "<br>"
      ],
      "metadata": {
        "id": "IFmOYqf2Dx7z"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Question 7** [15 points]\n",
        "\n",
        "Transformers are based on the self-attention mechanism. Answer the following questions about self-attention:\n",
        "\n",
        "#### **Q7.1** The inputs of the network are embedded and positionally encoded before entering the first self-attention layer of the encoder. What are the three vectors that we create, and how are they used to produce the output of this self-attention layer? You can assume that we only have one head.\n",
        "\n",
        "***\\[ answer the question in your own words, but do not use more than ~100 words \\]***"
      ],
      "metadata": {
        "id": "_mNjcWNhD2RO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The three vectors are K, V, Q. Lets assume X=(x1, x2). We use WK,WQ,WV to get corresponding K=(k1, k2), V=(v1, v2), Q=(q1, q2). Take x1 as an example. scores=(q1*k_i)/sqrt(d_k), so for different x_i, we have different score, which correspond to their different importance to x1. Then softmax, and multiply them to their cooresponding v, and sum up, we can get the final z1. It consists  of all v_i and their corresponding importance to x1."
      ],
      "metadata": {
        "id": "lTBsEmP8KoWc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Q7.2** Explain how the encoder and the decoder interact in the Transformer we saw in class.\n",
        "\n",
        "***\\[ answer the question in your own words, but do not use more than ~100 words \\]***"
      ],
      "metadata": {
        "id": "YVk6P3A1J-of"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In transformer, the encoder captures the meaning of the input sequence and passes this information to the decoder, which uses it to generate the output sequence.[4]\n",
        "\n",
        "Let's assume the output of encoder zx. The decoder uses zx as the key and value input in the multi-attention block. The decoder takes the output of the masked multi-attention output as the query input of the latter multi-attention block. And then use multi-attention block to process it. In class, the transformer has 6 encoders and 6 decoders, we take the output of last encoder as zx, and in every decoder, especially in its multi-attention block, take zx as the key and value input of it."
      ],
      "metadata": {
        "id": "Rhy4O1k6KqUJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<br>\n",
        "\n",
        "---\n",
        "\n",
        "<br>"
      ],
      "metadata": {
        "id": "W8wJW0ikavNX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Question 8** [15 points]\n",
        "\n",
        "#### **Q8.1** Write custom dataset class that:\n",
        "\n",
        "- Gets a sample from CelebA available from `torchvision`. Here is the link to the documentation: https://pytorch.org/vision/stable/generated/torchvision.datasets.CelebA.html\n",
        "\n",
        "- Removes a squared block of the sample at a random location. *\\[You want to generate images like the ones in figure-1 b) in this publication https://arxiv.org/pdf/1801.07939 (just given as a reference to what you are trying to achieve, no need to read anything from the paper here).\\]*\n",
        "\n",
        "- Returns the corrupted image as the input and the original image as the target.\n",
        "\n",
        "Download the data with `torchvision`, and resize the images to have dimension 224 x 224.  Use one of the two templates$^1$ below as a starting point, and add any methods, code, or arguments you deem necessary. \n",
        "\n",
        "$^1$ your custom class can either inherit from the ``CelebA`` class or from the ``Dataset`` class, you can choose which one you prefer to use.\n",
        "\n"
      ],
      "metadata": {
        "id": "kS--as3LESuq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### you can use this provided code in this cell or the one\n",
        "### in the cell below (only use one of the two).\n",
        "### Modify it as you see fit to produce what the question asks\n",
        "\n",
        "class CorruptedCelebA(CelebA):\n",
        "    def __init__(self, crop_size=80, **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        self.crop_size = crop_size\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        img, _ = super().__getitem__(idx)\n",
        "\n",
        "\n",
        "        square = torch.zeros(3, self.crop_size, self.crop_size)\n",
        "        convex = torch.randint(0,224-self.cropsize,1)\n",
        "        img = img\n",
        "        return corrupted_img, img"
      ],
      "metadata": {
        "id": "qnMbyxWEEmt1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "try:\n",
        "  from google.colab import drive\n",
        "  drive.mount('/content/drive')\n",
        "except:\n",
        "  pass"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xVojU2uA2sh-",
        "outputId": "c5536777-9944-47ba-b0d6-0d28e0d4c7b2"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "### you can use this provided code in this cell or the one\n",
        "### in the cell above (only use one of the two). \n",
        "### Modify it as you see fit to produce what the question asks\n",
        "\n",
        "class CorruptedCelebA(Dataset):\n",
        "    def __init__(self, data, crop_size=80, **kwargs):\n",
        "        super().__init__(**kwargs)\n",
        "        \n",
        "        self.crop_size = crop_size\n",
        "        self.data = data\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        img, _ = self.data[idx]\n",
        "        corrupted_img = img\n",
        "\n",
        "\n",
        "        h, w = corrupted_img.shape[1:]\n",
        "        x = torch.randint(0, w - self.crop_size, (1,)).item()\n",
        "        y = torch.randint(0, h - self.crop_size, (1,)).item()\n",
        "\n",
        "        corrupted_img[:, y:y+self.crop_size, x:x+self.crop_size] = 0\n",
        "\n",
        "        ### your code here\n",
        "        return corrupted_img, img\n",
        "    \n",
        "    def __len__(self):\n",
        "        return len(self.data)"
      ],
      "metadata": {
        "id": "eZEpQ97NIBiz"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from torchvision.transforms import Compose, ToTensor \n",
        "import torchvision.datasets\n",
        "transform=Compose([\n",
        "            Resize((224, 224)),\n",
        "            ToTensor(),\n",
        "        ])\n",
        "train_dataset = torchvision.datasets.CelebA(root=\"./\", download=True, transform=transform, target_transform=None)"
      ],
      "metadata": {
        "id": "7tA31kTq0Vep"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Q8.2** Plot 32 of these samples and their corresponding targets."
      ],
      "metadata": {
        "id": "XG3wX71GEtFF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "idxs = torch.randint(low=0, high=32, size=(6,))\n",
        "fig, axs = plt.subplots(4, 8, figsize=(15, 5))\n",
        "for i, idx in enumerate((idxs)):\n",
        "    img, target = train_dataset[idx]\n",
        "    axs[i].imshow(img, cmap=\"gray\")\n",
        "    axs[i].set_title(str(target))"
      ],
      "metadata": {
        "id": "dNNQpORN3UyP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Q8.3** From the list below, choose two transforms that could help improve the generalisation of a model, and two that would damage its performance. Briefly discuss each of these transforms and how you think it would influence the model.\n",
        "\n",
        "- `torchvision.transforms.CenterCrop()`\n",
        "- `torchvision.transforms.Grayscale()`\n",
        "- `torchvision.transforms.RandomAffine()`\n",
        "- `torchvision.transforms.RandomHorizontalFlip()`\n",
        "- `torchvision.transforms.RandomRotation()`\n",
        "- `torchvision.transforms.RandomVerticalFlip()`\n",
        "- `torchvision.transforms.GaussianBlur()`"
      ],
      "metadata": {
        "id": "DwkxPP7aElhh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "improve: torchvision.transforms.RandomHorizontalFlip(), torchvision.transforms.RandomRotation()\n",
        "\n",
        "bacasue it doesnot loss any information but do xome augmentaion\n",
        "\n",
        "damege: torchvision.transforms.GaussianBlur(), torchvision.transforms.CenterCrop()\n",
        "\n",
        "bacause its blur and loss imformation"
      ],
      "metadata": {
        "id": "W6j0Unok3ic1"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "daSzlkw94JH8"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}