9_stanford_cars_EfficientNet_Ranger_Mish_Trial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stanford Cars - NB9: Mish EfficientNet + Ranger - 5 run avg trial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TL;DR\n",
    "- Achieved **93.8%** 5-run, 40epoch, mean test set accuracy on Stanford Cars using Mish EfficientNet-b3 + Ranger\n",
    "- Beat the EfficientNet paper EfficientNet-b3 result by **0.2%**\n",
    "- EfficientNet author's best result using b3 was 93.6%, best EfficientNet result was 94.8% (current SOTA) with EfficientNet-b7\n",
    "- Used MEfficientNet-b3, created by swapping the Squish activation function for the **Mish** activation function\n",
    "- Used the **Ranger** optimisation function (a combination of RAdam and Lookahead) and trained with **FlatCosAnnealScheduler**\n",
    "- EfficientNet-b3 with Ranger but without Mish was giving test set accuracy around 93.4% (-0.4%) and was much more stable to train than my efforts to train the model with RMSProp, which was used in the paper\n",
    "\n",
    "\n",
    "## Credits: \n",
    "- Ranger - @lessw2020\n",
    "    - Lookahead paper: [Lookahead Optimizer: k steps forward, 1 step back](https://arxiv.org/abs/1907.08610)\n",
    "    - RAdam paper: [On the Variance of the Adaptive Learning Rate and Beyond, RAdam](https://arxiv.org/abs/1908.03265)\n",
    "    - @lessw2020 Ranger implementation https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer/blob/master/ranger.py\n",
    "    - version 9.3.19 used\n",
    " \n",
    "- Mish @digantamisra98\n",
    "    - Paper: [Mish: A Self Regularized Non-Monotonic Neural Activation Function](https://arxiv.org/abs/1908.08681v1)\n",
    "    - Mish Repo: https://github.com/digantamisra98/Mish\n",
    "    - Mish blog: https://medium.com/@lessw/meet-mish-new-state-of-the-art-ai-activation-function-the-successor-to-relu-846a6d93471f\n",
    "    - Mish code implementation - @lessw2020 - https://github.com/lessw2020/mish/blob/master/mish.py\n",
    "    \n",
    "- EfficientNet - @lukemelas\n",
    "    - Efficient Pytorch implementation that I swapped in Mish for: https://github.com/lukemelas/EfficientNet-PyTorch\n",
    "\n",
    "- FlatCosAnnealScheduler - @muellerzr\n",
    "    - Code taken from fastai thread below, being added to the fastai rep atm\n",
    "\n",
    "- [Inspirational fastai thread, credit to all the contributors here](https://forums.fast.ai/t/meet-mish-new-activation-function-possible-successor-to-relu/53299/280)\n",
    "\n",
    "\n",
    "### Training Params used:\n",
    "   - 40 epoch\n",
    "   - lr = 15e-4\n",
    "   - start_pct = 0.10\n",
    "   - wd = 1e-3\n",
    "   - bn_wd=False\n",
    "   - true_wd=True\n",
    "\n",
    "*Default Ranger params were used*: \n",
    "   - alpha=0.5\n",
    "   - k=6\n",
    "   - N_sma_threshhold=5\n",
    "   - betas=(.95,0.999)\n",
    "   - eps=1e-5\n",
    "\n",
    "### Augmentations used:\n",
    "- Image size : 299 x 299\n",
    "- Standard Fastai transforms from **get_transforms()**: \n",
    "     - do_flip = True, max_rotate = 10.0, max_zoom = 1.1, max_lighting = 0.2, max_warp = 0.2, p_affine: float = 0.75, p_lighting = 0.75\n",
    "- **ResizeMethod.SQUISH**, which I found worked quite well from testing with ResNet152\n",
    "\n",
    "### Training Notes\n",
    "- Unlike testing done on the fastiai forums with XResNet and the Imagewoof dataset, this setup performed better with a shorter amount of time with a flat lr, followed by a longer cosine anneal.\n",
    "- I used the full test set as the validation set, similar to the Imagewoof thread in the fastai thread linked above\n",
    "- I manually restarted the gpu kernel and changed the run count as weights seemed to be being saved between runs. This persisted even when using learn.purge() and learn.destroy(). There had been a mention on the forums that the lookahead element of the Ranger implementation might have been responsible, but the problem persisted even after using version 9.3.19 which was supposed to address the issue.\n",
    "- Ran on a Paperspace P4000 machine\n",
    "\n",
    "### Thanks\n",
    "Thanks as always to the amazing team at fast.ai and the fastai community! This and the following notebooks are all thanks to fastai's AMAZING MOOC and deep learning library, checkout https://fast.ai for the course and library, you won't regret it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "import json\n",
    "from PIL import ImageDraw, ImageFont\n",
    "from matplotlib import patches, patheffects\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import scipy.io as sio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.vision import *\n",
    "from fastai.utils.mem import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @lukemelas EfficientNet implementation: https://github.com/lukemelas/EfficientNet-PyTorch\n",
    "from efficientnet_pytorch import EfficientNet\n",
    "\n",
    "# @lessw2020 implementation : https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer/blob/master/ranger.py\n",
    "# version 9.3.19 used\n",
    "from ranger import Ranger\n",
    "\n",
    "from helper_functions import compare_most_confused, compare_top_losses, show_img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = 'data/stanford-cars/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>bbox_x1</th>\n",
       "      <th>bbox_y1</th>\n",
       "      <th>bbox_x2</th>\n",
       "      <th>bbox_y2</th>\n",
       "      <th>class_id</th>\n",
       "      <th>class_name</th>\n",
       "      <th>is_test</th>\n",
       "      <th>filename_cropped</th>\n",
       "      <th>bbox_h</th>\n",
       "      <th>bbox_w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>00001.jpg</td>\n",
       "      <td>39</td>\n",
       "      <td>116</td>\n",
       "      <td>569</td>\n",
       "      <td>375</td>\n",
       "      <td>14</td>\n",
       "      <td>Audi TTS Coupe 2012</td>\n",
       "      <td>0</td>\n",
       "      <td>cropped_00001.jpg</td>\n",
       "      <td>260</td>\n",
       "      <td>531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>00002.jpg</td>\n",
       "      <td>36</td>\n",
       "      <td>116</td>\n",
       "      <td>868</td>\n",
       "      <td>587</td>\n",
       "      <td>3</td>\n",
       "      <td>Acura TL Sedan 2012</td>\n",
       "      <td>0</td>\n",
       "      <td>cropped_00002.jpg</td>\n",
       "      <td>472</td>\n",
       "      <td>833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>00003.jpg</td>\n",
       "      <td>85</td>\n",
       "      <td>109</td>\n",
       "      <td>601</td>\n",
       "      <td>381</td>\n",
       "      <td>91</td>\n",
       "      <td>Dodge Dakota Club Cab 2007</td>\n",
       "      <td>0</td>\n",
       "      <td>cropped_00003.jpg</td>\n",
       "      <td>273</td>\n",
       "      <td>517</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    filename  bbox_x1  bbox_y1  bbox_x2  bbox_y2  class_id  \\\n",
       "0  00001.jpg       39      116      569      375        14   \n",
       "1  00002.jpg       36      116      868      587         3   \n",
       "2  00003.jpg       85      109      601      381        91   \n",
       "\n",
       "                   class_name  is_test   filename_cropped  bbox_h  bbox_w  \n",
       "0         Audi TTS Coupe 2012        0  cropped_00001.jpg     260     531  \n",
       "1         Acura TL Sedan 2012        0  cropped_00002.jpg     472     833  \n",
       "2  Dodge Dakota Club Cab 2007        0  cropped_00003.jpg     273     517  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels_df = pd.read_csv('labels_df.csv')\n",
    "labels_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look closer at the data, how many class_ids do we have? Does it match the number of class names?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loading\n",
    "Used the standard fastai image transforms and held out 20% of the training data for validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data(SZ:int=299, do_cutout:bool=False, p_cutout:float=0.75):\n",
    "    SEED = 42\n",
    "    LABEL = 'class_name'\n",
    "    \n",
    "    if do_cutout == True:\n",
    "        cutout_tfm = cutout(n_holes=(1,2), length=(100, 100), p=p_cutout)\n",
    "        car_tfms = get_transforms(xtra_tfms=[cutout_tfm])\n",
    "    else: car_tfms = get_transforms()\n",
    "\n",
    "    #tfms = get_transforms()\n",
    "\n",
    "    trn_labels_df = labels_df.loc[labels_df['is_test']==0, ['filename', 'class_name', 'class_id']].copy()\n",
    "\n",
    "    src = (ImageList.from_df(trn_labels_df, path, folder='train', cols='filename')\n",
    "                        .split_by_rand_pct(valid_pct=0.2, seed=SEED)\n",
    "                        .label_from_df(cols=LABEL))\n",
    "\n",
    "    data = (src.transform(car_tfms, \n",
    "                          size=SZ,  \n",
    "                          resize_method=ResizeMethod.SQUISH, \n",
    "                          padding_mode='reflection')\n",
    "                .databunch()\n",
    "                .normalize(imagenet_stats))\n",
    "    \n",
    "    # Get test data\n",
    "    TEST_SZ = 299\n",
    "    src_test = (ImageList.from_df(labels_df, path, folder='merged', cols='filename')\n",
    "           # the 'is_test' column has values of 1 for the test set\n",
    "           .split_from_df(col='is_test')\n",
    "           .label_from_df(cols=LABEL))\n",
    "\n",
    "    data_test = (src_test.transform(car_tfms, \n",
    "                                  size=SZ,  \n",
    "                                  resize_method=ResizeMethod.SQUISH, \n",
    "                                  padding_mode='reflection')\n",
    "                .databunch()\n",
    "                .normalize(imagenet_stats))\n",
    "    \n",
    "    return data, data_test, src, src_test, car_tfms\n",
    "\n",
    "data, data_test, src, src_test, car_tfms = get_data(do_cutout=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Flat and cosine annealer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# By @muellerzr on the fastai forums:\n",
    "# https://forums.fast.ai/t/meet-mish-new-activation-function-possible-successor-to-relu/53299/133       \n",
    "\n",
    "from fastai.callbacks import *\n",
    "\n",
    "def FlatCosAnnealScheduler(learn, lr:float=4e-3, tot_epochs:int=1, moms:Floats=(0.95,0.999),\n",
    "                          start_pct:float=0.72, curve='cosine'):\n",
    "    \"Manage FCFit trainnig as found in the ImageNette experiments\"\n",
    "    n = len(learn.data.train_dl)\n",
    "    anneal_start = int(n * tot_epochs * start_pct)\n",
    "    batch_finish = ((n * tot_epochs) - anneal_start)\n",
    "    if curve==\"cosine\":\n",
    "        curve_type=annealing_cos\n",
    "    elif curve==\"linear\":\n",
    "        curve_type=annealing_linear\n",
    "    elif curve==\"exponential\":\n",
    "        curve_type=annealing_exp\n",
    "    else:\n",
    "        raiseValueError(f\"annealing type not supported {curve}\")\n",
    "\n",
    "    phase0 = TrainingPhase(anneal_start).schedule_hp('lr', lr).schedule_hp('mom', moms[0])\n",
    "    phase1 = TrainingPhase(batch_finish).schedule_hp('lr', lr, anneal=curve_type).schedule_hp('mom', moms[1])\n",
    "    phases = [phase0, phase1]\n",
    "    return GeneralScheduler(learn, phases)\n",
    "                \n",
    "def fit_fc(learn:Learner, tot_epochs:int=None, lr:float=defaults.lr,  moms:Tuple[float,float]=(0.95,0.85), start_pct:float=0.72,\n",
    "                  wd:float=None, callbacks:Optional[CallbackList]=None, show_curve:bool=False)->None:\n",
    "    \"Fit a model with Flat Cosine Annealing\"\n",
    "    max_lr = learn.lr_range(lr)\n",
    "    callbacks = listify(callbacks)\n",
    "    callbacks.append(FlatCosAnnealScheduler(learn, lr, moms=moms, start_pct=start_pct, tot_epochs=tot_epochs))\n",
    "    learn.fit(tot_epochs, max_lr, wd=wd, callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_metrics_to_csv(exp_name, run_count, learn, metrics):\n",
    "    for m in metrics:\n",
    "        name = f'{m}_{exp_name}_run{str(run_count)}_2019-09_04'\n",
    "\n",
    "        ls = []\n",
    "        if m == 'val_loss_and_acc':\n",
    "            acc = []\n",
    "            for l in learn.recorder.metrics:\n",
    "                 acc.append(l[0].item())\n",
    "            ls = learn.recorder.val_losses \n",
    "\n",
    "            d = {name: ls, 'acc': acc}\n",
    "            df = pd.DataFrame(d)\n",
    "            #df.columns = [name, 'acc']\n",
    "        elif m == 'trn_loss':\n",
    "            for l in learn.recorder.losses:\n",
    "                ls.append(l.item())\n",
    "            df = pd.DataFrame(ls)\n",
    "            df.columns = [name]\n",
    "\n",
    "        df.to_csv(f'{name}_{m}.csv')\n",
    "        print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MEfficientNet + Ranger Trial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modified version of @lukemelas' EfficientNet implementation with Mish instead of Swish activation\n",
    "from MEfficientNet_PyTorch.efficientnet_pytorch import EfficientNet as MEfficientNet\n",
    "\n",
    "effnet_b3 = 'efficientnet-b3'\n",
    "def getModel(data, model_name):\n",
    "    model = MEfficientNet.from_pretrained(model_name)\n",
    "    model._fc = nn.Linear(1536, data.c)\n",
    "    return model\n",
    "\n",
    "mish_model = getModel(data, effnet_b3) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.728042</td>\n",
       "      <td>2.873347</td>\n",
       "      <td>0.458774</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.944554</td>\n",
       "      <td>1.743006</td>\n",
       "      <td>0.745430</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.516537</td>\n",
       "      <td>1.591039</td>\n",
       "      <td>0.792439</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.334032</td>\n",
       "      <td>1.482762</td>\n",
       "      <td>0.830991</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.264926</td>\n",
       "      <td>1.473987</td>\n",
       "      <td>0.829126</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.249027</td>\n",
       "      <td>1.411017</td>\n",
       "      <td>0.861957</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.190200</td>\n",
       "      <td>1.417674</td>\n",
       "      <td>0.851884</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.150759</td>\n",
       "      <td>1.384996</td>\n",
       "      <td>0.859470</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.098737</td>\n",
       "      <td>1.330367</td>\n",
       "      <td>0.875762</td>\n",
       "      <td>04:37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.102312</td>\n",
       "      <td>1.352557</td>\n",
       "      <td>0.877254</td>\n",
       "      <td>04:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.093992</td>\n",
       "      <td>1.446026</td>\n",
       "      <td>0.844174</td>\n",
       "      <td>04:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.063572</td>\n",
       "      <td>1.281574</td>\n",
       "      <td>0.888820</td>\n",
       "      <td>04:43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.029319</td>\n",
       "      <td>1.330305</td>\n",
       "      <td>0.885711</td>\n",
       "      <td>04:42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.043103</td>\n",
       "      <td>1.289455</td>\n",
       "      <td>0.890810</td>\n",
       "      <td>04:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.029761</td>\n",
       "      <td>1.333754</td>\n",
       "      <td>0.877378</td>\n",
       "      <td>04:46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>1.019359</td>\n",
       "      <td>1.282478</td>\n",
       "      <td>0.894665</td>\n",
       "      <td>04:43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>0.983806</td>\n",
       "      <td>1.248258</td>\n",
       "      <td>0.902127</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>0.960469</td>\n",
       "      <td>1.220568</td>\n",
       "      <td>0.911329</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>0.948175</td>\n",
       "      <td>1.232524</td>\n",
       "      <td>0.907350</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>0.943492</td>\n",
       "      <td>1.180493</td>\n",
       "      <td>0.920781</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>0.939930</td>\n",
       "      <td>1.216507</td>\n",
       "      <td>0.915060</td>\n",
       "      <td>04:40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>0.922105</td>\n",
       "      <td>1.176421</td>\n",
       "      <td>0.921652</td>\n",
       "      <td>04:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>0.929112</td>\n",
       "      <td>1.173882</td>\n",
       "      <td>0.921776</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>0.913459</td>\n",
       "      <td>1.165142</td>\n",
       "      <td>0.924263</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>0.911010</td>\n",
       "      <td>1.168120</td>\n",
       "      <td>0.925134</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>0.901649</td>\n",
       "      <td>1.145828</td>\n",
       "      <td>0.928989</td>\n",
       "      <td>04:37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>0.900681</td>\n",
       "      <td>1.144459</td>\n",
       "      <td>0.929238</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>0.894669</td>\n",
       "      <td>1.125707</td>\n",
       "      <td>0.933839</td>\n",
       "      <td>04:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>0.889733</td>\n",
       "      <td>1.120837</td>\n",
       "      <td>0.935456</td>\n",
       "      <td>04:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>0.890326</td>\n",
       "      <td>1.128216</td>\n",
       "      <td>0.931725</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>0.883061</td>\n",
       "      <td>1.114050</td>\n",
       "      <td>0.937446</td>\n",
       "      <td>04:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31</td>\n",
       "      <td>0.884941</td>\n",
       "      <td>1.112452</td>\n",
       "      <td>0.936078</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32</td>\n",
       "      <td>0.885446</td>\n",
       "      <td>1.108336</td>\n",
       "      <td>0.937819</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33</td>\n",
       "      <td>0.879232</td>\n",
       "      <td>1.105868</td>\n",
       "      <td>0.936699</td>\n",
       "      <td>04:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>34</td>\n",
       "      <td>0.877137</td>\n",
       "      <td>1.104897</td>\n",
       "      <td>0.937570</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>35</td>\n",
       "      <td>0.876879</td>\n",
       "      <td>1.102897</td>\n",
       "      <td>0.937073</td>\n",
       "      <td>04:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>36</td>\n",
       "      <td>0.876367</td>\n",
       "      <td>1.102296</td>\n",
       "      <td>0.937570</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>37</td>\n",
       "      <td>0.874715</td>\n",
       "      <td>1.101775</td>\n",
       "      <td>0.937694</td>\n",
       "      <td>04:37</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>38</td>\n",
       "      <td>0.875881</td>\n",
       "      <td>1.100920</td>\n",
       "      <td>0.938316</td>\n",
       "      <td>04:38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>39</td>\n",
       "      <td>0.874303</td>\n",
       "      <td>1.101267</td>\n",
       "      <td>0.937694</td>\n",
       "      <td>04:37</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   trn_loss_mefficient_b3_ranger_40e_15e4_wd1e-3_10pct_start_run5_2019-09_04\n",
      "0                                           5.269136                        \n",
      "1                                           5.274109                        \n",
      "2                                           5.290759                        \n",
      "3                                           5.294561                        \n",
      "4                                           5.287923                        \n",
      "   val_loss_and_acc_mefficient_b3_ranger_40e_15e4_wd1e-3_10pct_start_run5_2019-09_04  \\\n",
      "0                                           2.873347                                   \n",
      "1                                           1.743006                                   \n",
      "2                                           1.591039                                   \n",
      "3                                           1.482762                                   \n",
      "4                                           1.473987                                   \n",
      "\n",
      "        acc  \n",
      "0  0.458774  \n",
      "1  0.745430  \n",
      "2  0.792439  \n",
      "3  0.830991  \n",
      "4  0.829126  \n"
     ]
    }
   ],
   "source": [
    "exp_name = 'mefficient_b3_ranger_40e_15e4_wd1e-3_10pct_start'\n",
    "metrics = ['trn_loss', 'val_loss_and_acc']\n",
    "\n",
    "#Adding Mish activation to EfficientNet-b3 meant reducing bs from 32 -> 24\n",
    "data_test.batch_size = 24\n",
    "\n",
    "# Manually restarted the gpu kernel and changed the run count as weights seemed to be being saved between runs\n",
    "run_count = 5\n",
    "\n",
    "learn = Learner(data_test, \n",
    "                model=mish_model,\n",
    "                wd = 1e-3,\n",
    "                opt_func=Ranger,\n",
    "                bn_wd=False,\n",
    "                true_wd=True,\n",
    "                metrics=[accuracy],\n",
    "                loss_func=LabelSmoothingCrossEntropy()\n",
    "               ).to_fp16()\n",
    "\n",
    "fit_fc(learn, tot_epochs=40, lr=15e-4, start_pct=0.10, wd=1e-3, show_curve=False)\n",
    "\n",
    "learn.save(f'9_{exp_name}_run{run_count}')\n",
    "\n",
    "# SAVE METRICS\n",
    "save_metrics_to_csv(exp_name, run_count, learn, metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mean Test Set Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "aa = pd.read_csv('val_loss_and_acc_mefficient_ranger_40e_15e4_wd1e-3_10pct_start_run1_2019-09_04_val_loss_and_acc.csv')\n",
    "a = pd.read_csv('val_loss_and_acc_mefficient_b3_ranger_40e_15e4_wd1e-3_10pct_start_run2_2019-09_04_val_loss_and_acc.csv')\n",
    "b = pd.read_csv('val_loss_and_acc_mefficient_b3_ranger_40e_15e4_wd1e-3_10pct_start_run3_2019-09_04_val_loss_and_acc.csv')\n",
    "c = pd.read_csv('val_loss_and_acc_mefficient_b3_ranger_40e_15e4_wd1e-3_10pct_start_run4_2019-09_04_val_loss_and_acc.csv')\n",
    "d = pd.read_csv('val_loss_and_acc_mefficient_b3_ranger_40e_15e4_wd1e-3_10pct_start_run5_2019-09_04_val_loss_and_acc.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9380425333976745"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(aa['acc'][39:].values[0] \n",
    " + a['acc'][39:].values[0] \n",
    " + b['acc'][39:].values[0]\n",
    " + c['acc'][39:].values[0]\n",
    " + d['acc'][39:].values[0]) / 5"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}