notebooks/fabric_for_deep_learning_adversarial_samples_fashion_mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate Adversarial Samples for Deep Learning Models with the Adversarial Robustness Toolbox (ART)\n",
    "\n",
    "This notebook shows how to use adversarial attack techniques from the [Adversarial Robustness Toolbox (ART)](https://developer.ibm.com/code/open/projects/adversarial-robustness-toolbox/) on Deep Learning models trained with *FfDL*. The *ART* library supports crafting and analyzing various attack and defense methods for deep learning models. \n",
    "\n",
    "In this notebook, you will learn how to incorporate one of the attack methods supported by *ART*, the *Fast Gradient Method* (*FGM*), into your training pipeline to generate adversarial samples for the purposes of evaluating the robustness of the trained model. The model is a Convolutional Neural Network (CNN) trained on the *[MNIST handwritten digit data](http://yann.lecun.com/exdb/mnist/)* using [Keras](https://keras.io/) with a [TensorFlow](https://www.tensorflow.org/) backend.\n",
    "\n",
    "The *ART* Github repository can be found here - https://github.com/Trusted-AI/adversarial-robustness-toolbox\n",
    "\n",
    "This notebook uses Python 3.\n",
    "\n",
    "\n",
    "## Contents\n",
    "\n",
    "1.\t[Set up the environment](#setup)\n",
    "2.\t[Create a Keras model](#model)\n",
    "3.  [Train the model](#train)\n",
    "4.\t[Generate adversarial samples for a robustness check](#art)\n",
    "5.\t[Summary and next steps](#summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## 1. Setup\n",
    "\n",
    "It is recommended that you run this notebook inside a Python 3 virtual environment. Make sure you have all required libraries installed.\n",
    "\n",
    "To store model and training data, this notebook requires access to a Cloud Object Storage (COS) instance. [BlueMix Cloud Object Storage](https://console.bluemix.net/catalog/services/cloud-object-storage) offers a free *lite plan*. Follow [these instructions](https://dataplatform.ibm.com/docs/content/analyze-data/ml_dlaas_object_store.html) to create your COS instance and generate [service credentials](https://console.bluemix.net/docs/services/cloud-object-storage/iam/service-credentials.html#service-credentials) with [HMAC keys](https://console.bluemix.net/docs/services/cloud-object-storage/hmac/credentials.html#using-hmac-credentials).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Enter your cluster and object storage information:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "user_data = {\n",
    "    \"ffdl_dir\"                   : os.environ.get(\"FFDL_DIR\"),\n",
    "    \"ffdl_cluster_name\"          : os.environ.get(\"CLUSTER_NAME\"),\n",
    "    \"vm_type\"                    : os.environ.get(\"VM_TYPE\"),\n",
    "    \"cos_hmac_access_key_id\"     : os.environ.get(\"AWS_ACCESS_KEY_ID\"),\n",
    "    \"cos_hmac_secret_access_key\" : os.environ.get(\"AWS_SECRET_ACCESS_KEY\"),\n",
    "    \"cos_region_name\"            : os.environ.get(\"AWS_DEFAULT_REGION\"),\n",
    "    \"cos_service_endpoint\"       : os.environ.get(\"AWS_ENDPOINT_URL\") \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "unset_vars = [key for (key, value) in user_data.items() if not value]\n",
    "\n",
    "for var in unset_vars:\n",
    "    print(\"Dictionary 'user_data' is missing '%s'\" % var)\n",
    "    \n",
    "assert not unset_vars, \"Enter 'user_data' to run this notebook!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Verify or Install Required Python Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All required libraries are installed.\n",
      "keras>=2.1.6\r\n",
      "tensorflow>=1.8\r\n",
      "ipython>=5.0.0\r\n",
      "jupyter>=1.0.0\r\n",
      "requests>=2.12.0,<=2.18.4\r\n",
      "wget\r\n",
      "boto3\r\n",
      "git+git://github.com/Trusted-AI/adversarial-robustness-toolbox@master\r\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "\n",
    "def is_venv():\n",
    "    return (hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix))\n",
    "\n",
    "try:\n",
    "    import keras, tensorflow, requests, wget, boto3, art\n",
    "    print(\"All required libraries are installed.\")\n",
    "    !cat requirements.txt\n",
    "except ModuleNotFoundError:\n",
    "    if is_venv:\n",
    "        print(\"Installing required libraries into virtual environment.\")\n",
    "        !python -m pip install -r requirements.txt\n",
    "    else:\n",
    "        print(\"Please install the required libraries.\")\n",
    "        !cat requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Connect to Cloud Object Storage  (COS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a `boto3.resource` to interact with the COS instance. The `boto3` library allows Python developers to manage Cloud Object Storage (COS)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "cos = boto3.resource(\"s3\", \n",
    "                     aws_access_key_id     = user_data[\"cos_hmac_access_key_id\"],\n",
    "                     aws_secret_access_key = user_data[\"cos_hmac_secret_access_key\"],\n",
    "                     endpoint_url          = user_data[\"cos_service_endpoint\"],\n",
    "                     region_name           = user_data[\"cos_region_name\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for bucket in cos.buckets.all():\n",
    "#     print(bucket.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create two buckets, which you will use to store training data and training results.\n",
    "\n",
    "**Note:** The bucket names must be unique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating bucket \"training-data-c279ded3-c921-4360-9e05-aa086271a009\" ...\n",
      "Creating bucket \"training-results-c279ded3-c921-4360-9e05-aa086271a009\" ...\n"
     ]
    }
   ],
   "source": [
    "from uuid import uuid4\n",
    "\n",
    "bucket_uid             = str(uuid4())\n",
    "training_data_bucket   = 'training-data-' + bucket_uid\n",
    "training_result_bucket = 'training-results-' + bucket_uid\n",
    "\n",
    "def create_buckets(bucket_names):\n",
    "    for bucket in bucket_names:\n",
    "        print('Creating bucket \"{}\" ...'.format(bucket))\n",
    "        try:\n",
    "            cos.create_bucket(Bucket=bucket)\n",
    "        except boto3.exceptions.botocore.client.ClientError as e:\n",
    "            print('Error: {}.'.format(e.response['Error']['Message']))\n",
    "\n",
    "buckets = [training_data_bucket, training_result_bucket]\n",
    "\n",
    "create_buckets(buckets)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you should have 2 buckets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. Download MNIST Training Data and Upload it to the COS Buckets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select a data set (https://keras.io/datasets/):\n",
    "- `mnist.npz`\n",
    "- `fashion_mnist.npz`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets = [\"mnist.npz\", \"fashion_mnist.npz\"]\n",
    "\n",
    "dataset_filename = datasets[1] # 'fashion_mnist.npz'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the training data and upload it to the `training-data` bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading files to training-data-c279ded3-c921-4360-9e05-aa086271a009:\n",
      "- fashion_mnist.npz was uploaded\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist, fashion_mnist\n",
    "import numpy as np\n",
    "\n",
    "if \"fashion\" in dataset_filename:\n",
    "    (x_train, y_train), (x_test, y_test) = fashion_mnist.load_data() \n",
    "else:\n",
    "    (x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "    \n",
    "np.savez_compressed(dataset_filename, x_train=x_train , y_train=y_train, x_test=x_test, y_test=y_test)\n",
    "\n",
    "bucket_obj = cos.Bucket(training_data_bucket)\n",
    "print(\"Uploading files to {}:\".format(training_data_bucket))\n",
    "\n",
    "bucket_obj.upload_file(dataset_filename, dataset_filename)\n",
    "print('- {} was uploaded'.format(dataset_filename)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have a look at the list of the created buckets and their contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training-data-c279ded3-c921-4360-9e05-aa086271a009\n",
      "  File: fashion_mnist.npz, 30146.33kB\n",
      "training-results-c279ded3-c921-4360-9e05-aa086271a009\n"
     ]
    }
   ],
   "source": [
    "def print_bucket_contents(buckets):\n",
    "    for bucket_name in buckets:\n",
    "        print(bucket_name)\n",
    "        bucket_obj = cos.Bucket(bucket_name)\n",
    "        for obj in bucket_obj.objects.all():\n",
    "            print(\"  File: {}, {:4.2f}kB\".format(obj.key, obj.size/1024))\n",
    "\n",
    "print_bucket_contents(buckets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are done with COS, and you are ready to train your model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"model\"></a>\n",
    "## 2. Create the Keras model\n",
    "\n",
    "In this section we:\n",
    "\n",
    "- [2.1 Package the model definition](#zip)\n",
    "- [2.2 Prepare the training definition metadata](#manifest)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Create the Model Zip File <a id=\"zip\"></a>\n",
    "\n",
    "Let's create the model [`convolutional_keras.py`](../edit/convolutional_keras.py) and add it to a zip file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "script_filename  = \"convolutional_keras.py\"\n",
    "archive_filename = 'model.zip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting convolutional_keras.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $script_filename\n",
    "\n",
    "from __future__ import print_function\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "\n",
    "import keras\n",
    "import numpy as np\n",
    "import sys\n",
    "import os\n",
    "\n",
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 1\n",
    "\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "\n",
    "def main(argv):\n",
    "    if len(argv) < 2:\n",
    "        sys.exit(\"Not enough arguments provided.\")\n",
    "    global image_path\n",
    "    i = 1\n",
    "    while i <= 2:\n",
    "        arg = str(argv[i])\n",
    "        if arg == \"--data\":\n",
    "            image_path = os.path.join(os.environ[\"DATA_DIR\"], str(argv[i+1]))\n",
    "        i += 2\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main(sys.argv)\n",
    "\n",
    "\n",
    "# load training and test data from npz file\n",
    "f = np.load(image_path)\n",
    "x_train = f['x_train']\n",
    "y_train = f['y_train']\n",
    "x_test  = f['x_test']\n",
    "y_test  = f['y_test']\n",
    "f.close()\n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "# model\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, validation_split=0.1)\n",
    "\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n",
    "\n",
    "model_wt_path = os.environ[\"RESULT_DIR\"] + \"/keras_original_model.hdf5\"\n",
    "model.save(model_wt_path)\n",
    "print(\"Model saved to file: %s\" % model_wt_path)\n",
    "\n",
    "model_def_path = os.environ[\"RESULT_DIR\"] + \"/keras_original_model.json\"\n",
    "model_json = model.to_json()\n",
    "with open(model_def_path, \"w\") as json_file:\n",
    "    json_file.write(model_json)\n",
    "print(\"Model definition saved to file: %s\" % model_def_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "\n",
    "zipfile.ZipFile(archive_filename, mode='w').write(script_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Prepare the Training Definition Metadata <a id=\"manifest\"></a>\n",
    "- *FfDL* does not have a *Keras* community image so we need to `pip`-install *Keras* prior to running the `training_command` \n",
    "- Your COS credentials are referenced in the `data_stores` > `connection` data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "training_command = \"pip3 install keras; python3 %s --data ${DATA_DIR}/%s\" % (script_filename, dataset_filename)\n",
    "\n",
    "manifest = {\n",
    "  \"name\": \"keras_digit_recognition\",\n",
    "  \"description\": \"Hand-written Digit Recognition Training\",\n",
    "  \"version\": \"1.0\",\n",
    "  \"gpus\": 0,\n",
    "  \"cpus\": 2,\n",
    "  \"memory\": \"2Gb\",\n",
    "  \"data_stores\": [\n",
    "    {\n",
    "      \"id\": \"sl-internal-os\",\n",
    "      \"type\": \"s3_datastore\",\n",
    "      \"training_data\": {\n",
    "        \"container\": training_data_bucket\n",
    "      },\n",
    "      \"training_results\": {\n",
    "        \"container\": training_result_bucket\n",
    "      },\n",
    "      \"connection\": {\n",
    "        \"type\": \"s3_datastore\",\n",
    "        \"auth_url\":  user_data[\"cos_service_endpoint\"],\n",
    "        \"user_name\": user_data[\"cos_hmac_access_key_id\"],\n",
    "        \"password\":  user_data[\"cos_hmac_secret_access_key\"]\n",
    "      }\n",
    "    }\n",
    "  ],\n",
    "  \"framework\": {\n",
    "    \"name\": \"tensorflow\",\n",
    "    \"version\": \"1.5.0-py3\",\n",
    "    \"command\": training_command\n",
    "  },\n",
    "  \"evaluation_metrics\": {\n",
    "    \"type\": \"tensorboard\",\n",
    "    \"in\": \"$JOB_STATE_DIR/logs/tb\"\n",
    "  }\n",
    "}\n",
    "\n",
    "yaml.dump(manifest, open(\"manifest.yml\", \"w\"), default_flow_style=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Train the Model<a id=\"train\"></a>\n",
    "\n",
    "In this section, learn how to:\n",
    "- [3.1 Setup the command line environment](#cmd_setup)\n",
    "- [3.2 Train the model in the background](#backg)\n",
    "- [3.3 Monitor the training log](#log)\n",
    "- [3.4 Cancel the training](#cancel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Setup the Command Line Environment <a id=\"cmd_setup\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the Kubernetes cluster configuration using the [BlueMix CLI](https://console.bluemix.net/docs/cli/index.html#overview). Make sure your machine is logged in with `bx login`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: VM_TYPE=ibmcloud\n",
      "env: CLUSTER_NAME=my-ffdl-cluster\n",
      "env: KUBECONFIG=~/.bluemix/plugins/container-service/clusters/my-ffdl-cluster/kube-config-dal12-my-ffdl-cluster.yml\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %env VM_TYPE      {user_data[\"vm_type\"]}\n",
    "    %env CLUSTER_NAME {user_data[\"ffdl_cluster_name\"]}\n",
    "    cluster_config  = !bx cs cluster-config {user_data[\"ffdl_cluster_name\"]} | grep \"export KUBECONFIG=\"\n",
    "    %env KUBECONFIG   {cluster_config[-1].split(\"=\")[-1]}\n",
    "except IndexError:\n",
    "    print(\"The cluster %s could not be found.\" % {user_data[\"ffdl_cluster_name\"]})\n",
    "    print(\"Run 'bx cs clusters' to list all clusters you have access to.\")\n",
    "    #!bx cs clusters\n",
    "    raise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup the DLaaS URL, username and password"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: DLAAS_URL=http://169.48.201.210:30020\n",
      "env: DLAAS_USERNAME=test-user\n",
      "env: DLAAS_PASSWORD=test\n"
     ]
    }
   ],
   "source": [
    "node_ip        = !(cd {user_data[\"ffdl_dir\"]} && make --no-print-directory kubernetes-ip)\n",
    "restapi_port   = !kubectl get service ffdl-restapi -o jsonpath='{.spec.ports[0].nodePort}'\n",
    "dlaas_url      = \"http://%s:%s\" % (node_ip[0], restapi_port[0])\n",
    "\n",
    "%env DLAAS_URL        $dlaas_url\n",
    "%env DLAAS_USERNAME = test-user\n",
    "%env DLAAS_PASSWORD = test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obtain the correct FfDL CLI for your machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import platform\n",
    "\n",
    "ffdl = \"%s/cli/bin/ffdl-%s\" % (user_data[\"ffdl_dir\"], \"osx\" if platform.system() == \"Darwin\" else \"linux\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Start the Training Job<a id=\"backg\"></a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Deploying model with manifest 'manifest.yml' and model file 'model.zip'...\",\n",
       " 'Model ID: training-MyCDwcHmg',\n",
       " 'OK']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = !{ffdl} train \"manifest.yml\" \"model.zip\"\n",
    "out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.  Monitor the Training Logs<a id=\"log\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting model training logs for '\u001b[1;36mtraining-MyCDwcHmg\u001b[0m'...\n",
      "Status: PENDING\n",
      "Status: Not Started\n",
      "Training with training/test data at:\n",
      "  DATA_DIR: /job/training-data-c279ded3-c921-4360-9e05-aa086271a009\n",
      "  MODEL_DIR: /job/model-code\n",
      "  TRAINING_JOB: \n",
      "  TRAINING_COMMAND: pip3 install keras; python3 convolutional_keras.py --data ${DATA_DIR}/fashion_mnist.npz\n",
      "Storing trained model at:\n",
      "  RESULT_DIR: /job/training-results-c279ded3-c921-4360-9e05-aa086271a009\n",
      "Contents of $MODEL_DIR\n",
      "total 12\n",
      "drwxrwxrwx 2 6342627 root 4096 Jun 18 18:04 .\n",
      "drwxrwxrwx 6 root    root 4096 Jun 18 18:04 ..\n",
      "-rwxrwxrwx 1 6342627 root 2673 Jun 18 11:02 convolutional_keras.py\n",
      "Contents of $DATA_DIR\n",
      "total 30156\n",
      "drwxr-xr-x 2 6342627 root     4096 Jun 18 18:04 .\n",
      "drwxrwxrwx 6 root    root     4096 Jun 18 18:04 ..\n",
      "-rw-r--r-- 1 6342627 root 30869845 Jun 18 18:02 fashion_mnist.npz\n",
      "DATA_DIR=/job/training-data-c279ded3-c921-4360-9e05-aa086271a009\n",
      "ELASTICSEARCH_PORT=tcp://172.21.40.112:9200\n",
      "ELASTICSEARCH_PORT_9200_TCP=tcp://172.21.40.112:9200\n",
      "ELASTICSEARCH_PORT_9200_TCP_ADDR=172.21.40.112\n",
      "ELASTICSEARCH_PORT_9200_TCP_PORT=9200\n",
      "ELASTICSEARCH_PORT_9200_TCP_PROTO=tcp\n",
      "ELASTICSEARCH_SERVICE_HOST=172.21.40.112\n",
      "ELASTICSEARCH_SERVICE_PORT=9200\n",
      "ELASTICSEARCH_SERVICE_PORT_HTTP=9200\n",
      "FFDL_LCM_PORT=tcp://172.21.112.20:80\n",
      "FFDL_LCM_PORT_80_TCP=tcp://172.21.112.20:80\n",
      "FFDL_LCM_PORT_80_TCP_ADDR=172.21.112.20\n",
      "FFDL_LCM_PORT_80_TCP_PORT=80\n",
      "FFDL_LCM_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_LCM_SERVICE_HOST=172.21.112.20\n",
      "FFDL_LCM_SERVICE_PORT=80\n",
      "FFDL_LCM_SERVICE_PORT_GRPC=80\n",
      "FFDL_RESTAPI_PORT=tcp://172.21.130.217:80\n",
      "FFDL_RESTAPI_PORT_80_TCP=tcp://172.21.130.217:80\n",
      "FFDL_RESTAPI_PORT_80_TCP_ADDR=172.21.130.217\n",
      "FFDL_RESTAPI_PORT_80_TCP_PORT=80\n",
      "FFDL_RESTAPI_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_RESTAPI_SERVICE_HOST=172.21.130.217\n",
      "FFDL_RESTAPI_SERVICE_PORT=80\n",
      "FFDL_RESTAPI_SERVICE_PORT_FFDL=80\n",
      "FFDL_TRAINER_PORT=tcp://172.21.226.67:80\n",
      "FFDL_TRAINER_PORT_80_TCP=tcp://172.21.226.67:80\n",
      "FFDL_TRAINER_PORT_80_TCP_ADDR=172.21.226.67\n",
      "FFDL_TRAINER_PORT_80_TCP_PORT=80\n",
      "FFDL_TRAINER_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_TRAINER_SERVICE_HOST=172.21.226.67\n",
      "FFDL_TRAINER_SERVICE_PORT=80\n",
      "FFDL_TRAINER_SERVICE_PORT_GRPC=80\n",
      "FFDL_TRAININGDATA_PORT=tcp://172.21.106.158:80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP=tcp://172.21.106.158:80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_ADDR=172.21.106.158\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_PORT=80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_TRAININGDATA_SERVICE_HOST=172.21.106.158\n",
      "FFDL_TRAININGDATA_SERVICE_PORT=80\n",
      "FFDL_TRAININGDATA_SERVICE_PORT_GRPC=80\n",
      "FFDL_UI_PORT=tcp://172.21.201.22:80\n",
      "FFDL_UI_PORT_80_TCP=tcp://172.21.201.22:80\n",
      "FFDL_UI_PORT_80_TCP_ADDR=172.21.201.22\n",
      "FFDL_UI_PORT_80_TCP_PORT=80\n",
      "FFDL_UI_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_UI_SERVICE_HOST=172.21.201.22\n",
      "FFDL_UI_SERVICE_PORT=80\n",
      "FFDL_UI_SERVICE_PORT_HTTP=80\n",
      "GPU_COUNT=0.000000\n",
      "HOME=/root\n",
      "JOB_STATE_DIR=/job\n",
      "LEARNER_ID=1\n",
      "LOG_DIR=/job/logs\n",
      "MODEL_DIR=/job/model-code\n",
      "OLDPWD=/notebooks\n",
      "PATH=/usr/local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n",
      "PROMETHEUS_PORT=tcp://172.21.53.216:9090\n",
      "PROMETHEUS_PORT_9090_TCP=tcp://172.21.53.216:9090\n",
      "PROMETHEUS_PORT_9090_TCP_ADDR=172.21.53.216\n",
      "PROMETHEUS_PORT_9090_TCP_PORT=9090\n",
      "PROMETHEUS_PORT_9090_TCP_PROTO=tcp\n",
      "PROMETHEUS_SERVICE_HOST=172.21.53.216\n",
      "PROMETHEUS_SERVICE_PORT=9090\n",
      "PROMETHEUS_SERVICE_PORT_PROMETHEUS=9090\n",
      "PWD=/job/model-code\n",
      "PYTHONPATH=:/job/model-code\n",
      "RESULT_DIR=/job/training-results-c279ded3-c921-4360-9e05-aa086271a009\n",
      "S3_PORT=tcp://172.21.95.18:80\n",
      "S3_PORT_80_TCP=tcp://172.21.95.18:80\n",
      "S3_PORT_80_TCP_ADDR=172.21.95.18\n",
      "S3_PORT_80_TCP_PORT=80\n",
      "S3_PORT_80_TCP_PROTO=tcp\n",
      "S3_SERVICE_HOST=172.21.95.18\n",
      "S3_SERVICE_PORT=80\n",
      "SHLVL=3\n",
      "TRAINING_COMMAND=pip3 install keras; python3 convolutional_keras.py --data ${DATA_DIR}/fashion_mnist.npz\n",
      "TRAINING_ID=training-MyCDwcHmg\n",
      "_=/usr/bin/env\n",
      "Mon Jun 18 18:04:22 UTC 2018: Running training job\n",
      "Collecting keras\n",
      "  Downloading https://files.pythonhosted.org/packages/68/12/4cabc5c01451eb3b413d19ea151f36e33026fc0efb932bf51bcaf54acbf5/Keras-2.2.0-py2.py3-none-any.whl (300kB)\n",
      "Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Collecting keras-preprocessing==1.0.1 (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/f8/33/275506afe1d96b221f66f95adba94d1b73f6b6087cfb6132a5655b6fe338/Keras_Preprocessing-1.0.1-py2.py3-none-any.whl\n",
      "Collecting keras-applications==1.0.2 (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/e2/60/c557075e586e968d7a9c314aa38c236b37cb3ee6b37e8d57152b1a5e0b47/Keras_Applications-1.0.2-py2.py3-none-any.whl (43kB)\n",
      "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Collecting pyyaml (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz (253kB)\n",
      "Building wheels for collected packages: pyyaml\n",
      "  Running setup.py bdist_wheel for pyyaml: started\n",
      "  Running setup.py bdist_wheel for pyyaml: finished with status 'done'\n",
      "  Stored in directory: /root/.cache/pip/wheels/03/05/65/bdc14f2c6e09e82ae3e0f13d021e1b6b2481437ea2f207df3f\n",
      "Successfully built pyyaml\n",
      "Installing collected packages: keras-preprocessing, keras-applications, pyyaml, keras\n",
      "Successfully installed keras-2.2.0 keras-applications-1.0.2 keras-preprocessing-1.0.1 pyyaml-3.12\n",
      "You are using pip version 9.0.1, however version 10.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "2018-06-18 18:04:27.811560: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n",
      "Train on 54000 samples, validate on 6000 samples\n",
      "Epoch 1/1\n",
      "\n",
      "  128/54000 [..............................] - ETA: 4:54 - loss: 2.3122 - acc: 0.1094\n",
      "  256/54000 [..............................] - ETA: 4:26 - loss: 2.2559 - acc: 0.1484\n",
      "  384/54000 [..............................] - ETA: 4:09 - loss: 2.2135 - acc: 0.1849\n",
      "  512/54000 [..............................] - ETA: 4:05 - loss: 2.1616 - acc: 0.2031\n",
      "  640/54000 [..............................] - ETA: 4:00 - loss: 2.1043 - acc: 0.2297\n",
      "  768/54000 [..............................] - ETA: 3:55 - loss: 2.0374 - acc: 0.2578\n",
      "  896/54000 [..............................] - ETA: 3:54 - loss: 1.9933 - acc: 0.2757\n",
      " 1024/54000 [..............................] - ETA: 3:52 - loss: 1.9585 - acc: 0.2930\n",
      " 1152/54000 [..............................] - ETA: 3:51 - loss: 1.9320 - acc: 0.3047\n",
      " 1280/54000 [..............................] - ETA: 3:49 - loss: 1.8973 - acc: 0.3211\n",
      " 1408/54000 [..............................] - ETA: 3:47 - loss: 1.8475 - acc: 0.3388\n",
      " 1536/54000 [..............................] - ETA: 3:47 - loss: 1.8413 - acc: 0.3405\n",
      " 1664/54000 [..............................] - ETA: 3:46 - loss: 1.8464 - acc: 0.3504\n",
      " 1792/54000 [..............................] - ETA: 3:46 - loss: 1.8266 - acc: 0.3666\n",
      " 1920/54000 [>.............................] - ETA: 3:44 - loss: 1.7821 - acc: 0.3812\n",
      " 2048/54000 [>.............................] - ETA: 3:44 - loss: 1.7542 - acc: 0.3892\n",
      " 2176/54000 [>.............................] - ETA: 3:43 - loss: 1.7296 - acc: 0.3966\n",
      " 2304/54000 [>.............................] - ETA: 3:42 - loss: 1.6966 - acc: 0.4102\n",
      " 2432/54000 [>.............................] - ETA: 3:41 - loss: 1.6631 - acc: 0.4219\n",
      " 2560/54000 [>.............................] - ETA: 3:40 - loss: 1.6452 - acc: 0.4273\n",
      " 2688/54000 [>.............................] - ETA: 3:39 - loss: 1.6376 - acc: 0.4301\n",
      " 2816/54000 [>.............................] - ETA: 3:38 - loss: 1.6211 - acc: 0.4332\n",
      " 2944/54000 [>.............................] - ETA: 3:37 - loss: 1.6019 - acc: 0.4406\n",
      " 3072/54000 [>.............................] - ETA: 3:37 - loss: 1.5782 - acc: 0.4489\n",
      " 3200/54000 [>.............................] - ETA: 3:36 - loss: 1.5601 - acc: 0.4541\n",
      " 3328/54000 [>.............................] - ETA: 3:36 - loss: 1.5428 - acc: 0.4600\n",
      " 3456/54000 [>.............................] - ETA: 3:36 - loss: 1.5245 - acc: 0.4682\n",
      " 3584/54000 [>............\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                          ... ===========>.] - ETA: 4s - loss: 0.6157 - acc: 0.7794\n",
      "53120/54000 [============================>.] - ETA: 3s - loss: 0.6154 - acc: 0.7796\n",
      "53248/54000 [============================>.] - ETA: 3s - loss: 0.6148 - acc: 0.7798\n",
      "53376/54000 [============================>.] - ETA: 2s - loss: 0.6144 - acc: 0.7800\n",
      "53504/54000 [============================>.] - ETA: 2s - loss: 0.6137 - acc: 0.7802\n",
      "53632/54000 [============================>.] - ETA: 1s - loss: 0.6132 - acc: 0.7804\n",
      "53760/54000 [============================>.] - ETA: 1s - loss: 0.6128 - acc: 0.7805\n",
      "53888/54000 [============================>.] - ETA: 0s - loss: 0.6127 - acc: 0.7805\n",
      "54000/54000 [==============================] - 234s 4ms/step - loss: 0.6124 - acc: 0.7806 - val_loss: 0.3721 - val_acc: 0.8633\n",
      "/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n",
      "Test loss: 0.38816263160705566\n",
      "Test accuracy: 0.8606\n",
      "Model saved to file: /job/training-results-c279ded3-c921-4360-9e05-aa086271a009/keras_original_model.hdf5\n",
      "Model definition saved to file: /job/training-results-c279ded3-c921-4360-9e05-aa086271a009/keras_original_model.json\n",
      "Exception ignored in: <bound method BaseSession.__del__ of <tensorflow.python.client.session.Session object at 0x7f4651d91780>>\n",
      "Traceback (most recent call last):\n",
      "  File \"/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py\", line 702, in __del__\n",
      "TypeError: 'NoneType' object is not callable\n",
      "Training process finished. Exit code: 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if \"Model ID\" in out[1]:\n",
    "    model_id = out.fields()[1][-1]\n",
    "    !{ffdl} logs --follow {model_id}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Generate Adversarial Samples <a id=\"art\"></a>\n",
    "\n",
    "In this section, we learn how to:\n",
    "- [4.1 Generate adversarial samples with ART (synchronously in notebook)](#artLocal)\n",
    "- [4.2 Generate adversarial samples with ART (asynchronously using FfDL)](#artWithFfDL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1. Generate Adversarial Samples Locally <a id=\"artLocal\"></a>\n",
    "\n",
    "This section shows how to use the ART Fast Gradient Method (FGM) to generate adversarial samples for the model previously trained synchronously in this notebook. \n",
    "\n",
    "A trained model should have been created in the `training_result_bucket`. Now ART can be used to check the robustness of the trained model. \n",
    "\n",
    "The original dataset used to train the model as well as the trained model serve as inputs to the `robustness_check.py` script. We can download both from the `training_data_bucket` and the `training_result_bucket` respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, download the original data set and the trained model from Cloud Object Store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights_filename = \"keras_original_model.hdf5\"\n",
    "network_definition_filename = \"keras_original_model.json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print contents of COS buckets used in the previous training run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training-data-c279ded3-c921-4360-9e05-aa086271a009\n",
      "  File: fashion_mnist.npz, 30146.33kB\n",
      "training-results-c279ded3-c921-4360-9e05-aa086271a009\n",
      "  File: training-MyCDwcHmg/keras_original_model.hdf5, 14092.55kB\n",
      "  File: training-MyCDwcHmg/keras_original_model.json, 2.75kB\n",
      "  File: training-MyCDwcHmg/learner-1/load-data.log, 8.32kB\n",
      "  File: training-MyCDwcHmg/learner-1/load-model.log, 0.42kB\n",
      "  File: training-MyCDwcHmg/learner-1/training-log.txt, 41.91kB\n"
     ]
    }
   ],
   "source": [
    "print_bucket_contents([training_data_bucket, training_result_bucket])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded keras_original_model.hdf5\n",
      "Downloaded keras_original_model.json\n"
     ]
    }
   ],
   "source": [
    "# download network definition and weights to current working directory\n",
    "\n",
    "weights_file_in_cos_bucket = os.path.join(model_id, weights_filename)\n",
    "network_definition_file_in_cos_bucket = os.path.join(model_id, network_definition_filename)\n",
    "\n",
    "bucket_obj = cos.Bucket(training_result_bucket)\n",
    "\n",
    "bucket_obj.download_file(weights_file_in_cos_bucket, weights_filename)\n",
    "print('Downloaded', weights_filename)\n",
    "\n",
    "bucket_obj.download_file(network_definition_file_in_cos_bucket, network_definition_filename)\n",
    "print('Downloaded', network_definition_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load & compile the model that we created using `convolutional_keras.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Definition: keras_original_model.json\n",
      "Weights:            keras_original_model.hdf5\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from keras import backend as K\n",
    "from keras.models import model_from_json\n",
    "\n",
    "print('Network Definition:', network_definition_filename)\n",
    "print('Weights:           ', weights_filename)\n",
    "\n",
    "# load model\n",
    "json_file = open(network_definition_filename, 'r')\n",
    "model_json = json_file.read()\n",
    "json_file.close()\n",
    "\n",
    "model = model_from_json(model_json)\n",
    "model.load_weights(weights_filename)\n",
    "comp_params = {'loss': 'categorical_crossentropy',\n",
    "                       'optimizer': 'adam',\n",
    "                       'metrics': ['accuracy']}\n",
    "model.compile(**comp_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After loading & compiling the model, the next step is to create a KerasClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create ART classifier object\n",
    "from art.estimators.classification import KerasClassifier\n",
    "\n",
    "classifier = KerasClassifier(clip_values=(0, 1), model=model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the test data and labels from `.npz` file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import np_utils\n",
    "\n",
    "f = np.load(dataset_filename)\n",
    "x_original = f['x_test']\n",
    "y = f['y_test']\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the original (non-adversarial) sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJIAAACPCAYAAAARM4LLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAC15JREFUeJztnVtoVtkVx//LeL/H2xgv6ETjJQgqDFKxSMUqVhBftEweah8G8mCLFfrQGfsgiII+2LcKBqujUKYWWhjRUbFiLIOXRGW0jpKLxXEyqDEq3q/j7kM+073+yfed77K/y9H1A/H8z/m+c3bCyt7rrL322uKcg2HkSq9iN8B4NzBDMoJghmQEwQzJCIIZkhEEMyQjCGZIRhDMkIwg5GRIIrJMRJpEpFVEPg3VKCN+SLaRbREpA9AMYAmANgCNAGqcc1dSfKdkw+i9eum/qUGDBin96NGjrO89cOBApX/88UelX7x4kfW9C0CHc2501Id65/CAeQBanXP/BQAR+RuAlQCSGlIpw4Yzb948pY8fP571vWfMmKH048ePlW5ubs763gXgu3Q+lMvQNh7A955uS5xTiEitiJwTkXM5PMsocXLpkdLCOVcHoA4o7aHNyI1cDOkHABM9PSFxriTo37+/0uvXr1e6pqZG6fLycqVHj9ZuwdOnT5UeMWJE2m15/vy50s+ePVOafaaTJ08qvWvXLqWPHDmS9rMLRS5DWyOAKhH5UET6AvgYwIEwzTLiRtY9knPutYj8FsBRAGUAdjvnvg3WMiNW5OQjOee+AvBVoLYYMSbrOFJWD8ujs71t2zala2trlR4yZIjS7KewfvXqldIDBgxQuk+fPl3HZWVl6trLly+VZv+KY1b9+vVL+Sy+/+nTp5VeuHAh8sh559xHUR+yKRIjCGZIRhDMkIwgxNZHYh9o586dSt+6dUvp169fZ3T/vn37Ks2xHh/+Hb5580Zp359K5/vcVn72hAkTlD58+LDSK1asSPm8DDEfySgcZkhGEMyQjCDE1ke6ffu20jy3xqkaHLsZO3Zsyvvfv39fac4Z8v0YTkHhtty9e1dpjguxD8RxJRFRmuNUgwcPVnrKlCldxx0dHcgR85GMwmGGZAQh7/lI+WLYsGFK89ATNZTt2LFD6bq6OqXPnz+v9M2bN5X2X8E5DffGjRtKjxkzRmkemioqKpRua2tTmn+2oUOHKs1TKpWVlV3HAYa2tLAeyQiCGZIRBDMkIwix9ZH4FZnTWfmVmdmwYYPSDx48UJpf0XlJUX19fdfxokWLUj7ryhW9sGbmzJlKs8+zbt06pTdv3qz0nTt3lGZ/cMGCBV3HDQ0NKdsWCuuRjCCYIRlBMEMyghCrKRI/tYNjKzylwT7S8OHDlT5wQC94WblypdJRvxf//ps2bVLXHj58qPSxY8eU5qVM7e3tSvPP1tLSojRPuXAa8f79+7uO16xZ063tGWJTJEbhMEMygmCGZAQhVnGkcePGJb3G6a08/8SMH9+t3kVKVq9enfTavn37lOaYFsekLl68qDTPtXEKTKZUVVXl9P1ssB7JCIIZkhEEMyQjCLHykUaNGpX2Z3kJEC/BZh+J56sYLjXjc/ToUaX9fCCge9xn+fLlSp84cUJp9qGi0oZ5+VJUGnE+sB7JCEKkIYnIbhFpF5HL3rkRInJMRFoS/5enuofx7pNOj/Q5gGV07lMAx51zVQCOJ7TxHhPpIznn/i0ik+n0SgA/SxzvBVAP4A8B29UjvFTZJyr/iEvLsB/BcSi+3/Tp05XeunVr17G//Kcnrl69qjRXuZ00aZLSa9euVXr+/PlK37t3T2nOAc80RhaCbH2kD5xzb7PhbwH4IFB7jJiS81ubc86lmtUXkVoAtcmuG+8G2fZIt0WkAgAS/7cn+6Bzrs4591E6qQhGfMm2RzoA4NcAtib+/zJYi1LAJYt92Mfh+S3WHJvZsmWL0hyHWrp0qdKzZ8/uOp41a5a6xvlB7BP5/hWg84cAYM6cOUgF/yyZltHJB+m8/n8B4DSA6SLSJiKfoNOAlohIC4CfJ7TxHpPOW1tNkkuLA7fFiDEW2TaCEKu5Ns7b8WE/geej2G/gdWy8zo3hz/tldaqrq1N+l8sQsq/H+UsM549H+UipPpuqhGEuWI9kBMEMyQiCGZIRhFj5SKniSAzPP/EOkLztAtckYl+CyyX37v3/X13UNqXsn7HPxKUC+X7sn3GcifOdfCZPnqz0tWvXUrY1W6xHMoJghmQEIVZDGy+79uHKrjxU7d27V2lOd+U0E4bDCX6aiT/M9QS/vvNQxyV6OHV2z549SkdNofhwerINbUZJY4ZkBMEMyQhCrHwkvxwM+x1cmo/L43HZG4bDBezH5FL+J2qKg69zqOHs2bMZ3d/fDTMqBTkU1iMZQTBDMoJghmQEIVY+kh9H4vJ4UbsjcUliJmpKhMnEZ2I/hb/LmuNlmZQhBHTMK5NppVywHskIghmSEQQzJCMIsfKR/PhLlN/Q1NSkdNSyar4fz63x9UziM1FxJPb3eAsxLp/M8P38tmVSCigXrEcygmCGZATBDMkIQqx8JD/vJ2pZTXNzs9KcWpvq3j3BPpGvc4nzAN3zjxjOrWI9cuTIpN/l5eP5wnokIwhmSEYQzJCMIMTKR/LzbKJ8JF7GzKVluFxyVHnkTIiKQXHbon6WqVOnKs3LmbiMoZ9bxXla+cJ6JCMI6dRHmigiJ0Tkioh8KyK/S5y3EslGF+n0SK8B/N45Vw3gJwB+IyLVsBLJhkc6hbZuAriZOH4kIlcBjEcRSiT7vgTPLzEcF+JYC69ji7pfKjLN544qU8jwNqnXr19Xeu7cuUnvX15emIEiIx8pUW97LoCzsBLJhkfab20iMhjAPwCsd849pMhu0hLJVh75/SCtHklE+qDTiP7qnPtn4nRaJZKtPPL7QWSPJJ1dz18AXHXO/cm7VPASyb6PxDnaDOdocw425wCxTxW1pUSqa1E52kyUj8SlaS5duqT0qlWrkn63UKWS0xnaFgD4FYD/iMg3iXMb0GlAf0+US/4OwC/z00QjDqTz1vY1gGR/jlYi2QBgkW0jELGaa/PnkKJypjl+wtu381r/VCWGo65HrVOLyvdmzaX+eJstzrVK1Z6obetDYT2SEQQzJCMIZkhGEGLlI/k5RH5uEtC9huT27duVXrxYv2Cy75Dp1gq+H5LpmreobR2GDh2qdH19vdIHDx5UeuPGjUnvF1XDIBTWIxlBMEMyghCroc1PG+XhgFNnuUvv6OhQuqqqSmkuG5xJ6m3UUMbXOZTAy5H8EodA9yXb/LMw/u+Gd/DOF9YjGUEwQzKCYIZkBCFWPtKpU6e6jnnagHdh5GmEadOm5a9hBaayslJp3k3J35KisbGxIG2yHskIghmSEQQzJCMIsfKRGhoauo55KXKmaSFxJmqbLj+GxmWi84X1SEYQzJCMIJghGUGIlY/kl7y7cOGCusZxpCdPnqS8Fy8/4rm7Qm1P1RP8bG5ba2ur0ocOHVLaL6985syZwK3rGeuRjCCYIRlBMEMygiC5bLGZ8cNE7qBzVe4oAKmTaopHqbatWO2a5JyL3KuroIbU9VCRc6VaVKJU21aq7XqLDW1GEMyQjCAUy5DqivTcdCjVtpVquwAUyUcy3j1saDOCUFBDEpFlItIkIq0iUtRyyiKyW0TaReSyd64kaofHsbZ5wQxJRMoA/BnALwBUA6hJ1OsuFp8DWEbnSqV2ePxqmzvnCvIPwHwARz39GYDPCvX8JG2aDOCyp5sAVCSOKwA0FbN9Xru+BLCkVNvnnCvo0DYewPeebkucKyVKrnZ4XGqbm7OdBNf5Z1/UV1qube5fK4X2+RTSkH4AMNHTExLnSom0aocXglxqmxeDQhpSI4AqEflQRPoC+BidtbpLibe1w4EC1Q7viTRqmwNFbF+PFNhpXA6gGcA1AH8ssgP7BTo363mFTn/tEwAj0fk21ALgXwBGFKltP0XnsHUJwDeJf8tLpX09/bPIthEEc7aNIJghGUEwQzKCYIZkBMEMyQiCGZIRBDMkIwhmSEYQ/ge/lt3UHfc3WgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 144x144 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "plt.figure(figsize=(2, 2))\n",
    "plt.imshow(x_original[1], cmap='gray')\n",
    "print(y[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standardize the Numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preprocess\n",
    "x_original = np.expand_dims(x_original, axis=3)\n",
    "x_original = x_original.astype('float32') / 255\n",
    "y = np_utils.to_categorical(y, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the model and calculated test accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model test loss:     38.81626313686371\n",
      "model test accuracy: 86.06\n"
     ]
    }
   ],
   "source": [
    "# evaluate\n",
    "scores = model.evaluate(x_original, y, verbose=0)\n",
    "print('model test loss:    ', scores[0]*100)\n",
    "print('model test accuracy:', scores[1]*100)\n",
    "model_accuracy = scores[1]*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ART exposes many attacks like FGM, NewtonFool, DeepFool, Carlini etc. The code below shows how to use one of ART's attack methods (Fast Gradient Method or FGM) to craft adversarial samples based on x_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of adversarial samples crafted: 10000\n",
      "adversarial samples saved to: adv_samples.npz\n"
     ]
    }
   ],
   "source": [
    "from art.attacks import FastGradientMethod\n",
    "\n",
    "# configuration\n",
    "epsilon = 0.2\n",
    "\n",
    "# create crafter object\n",
    "crafter = FastGradientMethod(classifier, eps=epsilon)\n",
    "\n",
    "# craft samples on x_test (stored in variable x_original)\n",
    "x_adv_samples = crafter.generate(x_original)\n",
    "\n",
    "adv_samples_filename = \"adv_samples.npz\"\n",
    "np.savez(adv_samples_filename, x_original=x_original, x_adversarial=x_adv_samples, y=y)\n",
    "\n",
    "print(\"Number of adversarial samples crafted:\", len(x_adv_samples))\n",
    "print(\"adversarial samples saved to:\", adv_samples_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following functions can be used for gathering metrics like model robustness, confidence metric, perturbation metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy.linalg as la\n",
    "import json\n",
    "\n",
    "\n",
    "def get_metrics(model, x_original, x_adv_samples, y):\n",
    "    scores = model.evaluate(x_original, y, verbose=0)\n",
    "    model_accuracy_on_non_adversarial_samples = scores[1] * 100\n",
    "\n",
    "    y_pred = model.predict(x_original, verbose=0)\n",
    "    y_pred_adv = model.predict(x_adv_samples, verbose=0)\n",
    "\n",
    "    scores = model.evaluate(x_adv_samples, y, verbose=0)\n",
    "    model_accuracy_on_adversarial_samples = scores[1] * 100\n",
    "\n",
    "    pert_metric = get_perturbation_metric(x_original, x_adv_samples, y_pred, y_pred_adv, ord=2)\n",
    "    conf_metric = get_confidence_metric(y_pred, y_pred_adv)\n",
    "\n",
    "    data = {\n",
    "        \"model accuracy on test data:\": model_accuracy_on_non_adversarial_samples,\n",
    "        \"model accuracy on adversarial samples\": model_accuracy_on_adversarial_samples,\n",
    "        \"reduction in confidence\": conf_metric * 100,\n",
    "        \"average perturbation\": pert_metric * 100\n",
    "    }\n",
    "    return data\n",
    "\n",
    "\n",
    "def get_perturbation_metric(x_original, x_adv, y_pred, y_pred_adv, ord=2):\n",
    "\n",
    "    idxs = (np.argmax(y_pred_adv, axis=1) != np.argmax(y_pred, axis=1))\n",
    "\n",
    "    if np.sum(idxs) == 0.0:\n",
    "        return 0\n",
    "\n",
    "    perts_norm = la.norm((x_adv - x_original).reshape(x_original.shape[0], -1), ord, axis=1)\n",
    "    perts_norm = perts_norm[idxs]\n",
    "\n",
    "    return np.mean(perts_norm / la.norm(x_original[idxs].reshape(np.sum(idxs), -1), ord, axis=1))\n",
    "\n",
    "\n",
    "# This computes the change in confidence for all images in the test set\n",
    "def get_confidence_metric(y_pred, y_pred_adv):\n",
    "\n",
    "    y_classidx = np.argmax(y_pred, axis=1)\n",
    "    y_classconf = y_pred[np.arange(y_pred.shape[0]), y_classidx]\n",
    "\n",
    "    y_adv_classidx = np.argmax(y_pred_adv, axis=1)\n",
    "    y_adv_classconf = y_pred_adv[np.arange(y_pred_adv.shape[0]), y_adv_classidx]\n",
    "\n",
    "    idxs = (y_classidx == y_adv_classidx)\n",
    "\n",
    "    if np.sum(idxs) == 0.0:\n",
    "        return 0\n",
    "\n",
    "    idxnonzero = y_classconf != 0\n",
    "    idxs = idxs & idxnonzero\n",
    "\n",
    "    return np.mean((y_classconf[idxs] - y_adv_classconf[idxs]) / y_classconf[idxs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the robustness check metrics\n",
    "\n",
    "1. Model accuracy on test data\n",
    "2. Model robustness on adversarial samples\n",
    "3. Reduction in confidence\n",
    "4. Perturbation metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"model accuracy on test data:\": 86.06,\n",
      "    \"model accuracy on adversarial samples\": 17.09,\n",
      "    \"reduction in confidence\": 42.64292120933533,\n",
      "    \"average perturbation\": 44.387608766555786\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "result = get_metrics(model, x_original, x_adv_samples, y)\n",
    "\n",
    "print(json.dumps(result, indent=4, sort_keys=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare original images with adversarial samples and test model predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://keras.io/datasets/#fashion-mnist-database-of-fashion-articles\n",
    "\n",
    "fashion_labels = {\n",
    " 0: \"T-shirt/top\",\n",
    " 1: \"Trouser\",\n",
    " 2: \"Pullover\",\n",
    " 3: \"Dress\",\n",
    " 4: \"Coat\",\n",
    " 5: \"Sandal\",\n",
    " 6: \"Shirt\",\n",
    " 7: \"Sneaker\",\n",
    " 8: \"Bag\",\n",
    " 9: \"Ankle boot\"\n",
    "}\n",
    "\n",
    "def get_label(y):\n",
    "    if \"fashion\" in dataset_filename:\n",
    "        return fashion_labels[y]\n",
    "    else:\n",
    "        return \"Predict: %i\" % y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1oAAAC/CAYAAADnw60yAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsnXeYLEW5xn9jQgGRLJJzzjnjISM5SVCSIioXQa8BxLAsooKKIBjI0QscL0GSkj2So+Sccxa8Akbs+8fsW/P22dqZ2d2e3Zlzvt/z8Jymdqanu7qquru+t96vVhQFQRAEQRAEQRAEQXW8Z7wPIAiCIAiCIAiCYEojXrSCIAiCIAiCIAgqJl60giAIgiAIgiAIKiZetIIgCIIgCIIgCComXrSCIAiCIAiCIAgqJl60giAIgiAIgiAIKiZetIIgCIIgCIIgCComXrSCIAiCIAiCIAgqJl60giAIgiAIgiAIKuZ9w/lwrVYrOnUgUwCvFUUx23C/NBZ1+p73NN6np5tuOgD++te/tv39aaedFoB33303lf3jH/+o6Oia0rV1+uEPfzhtzz777AD87W9/S2Xve1+9a3k9vfe97x20XRSNQ51mmmkAePzxxztwxImuq1PV1ayzzprK1Na8fib/2+R/r9VqpX8B/v3vfwPl9p7b5yjpujrNob7v40GuzhyVqx4B3nrrrU4dotO1dbrooosOKvvPf/6TtlW/3ve9zt///vcD5Xacq//HHnts9Adbpmvr1NHY6PWjsVFjBZT7ser/73//+1gcojOiOoXxe56aZZZZ0rbGhNyYCY36fP3118fo6OoURZEfkFoQz6hN6Yn+32O0VafDetEKmvL0eB/AUGgwBVh11VUBuPrqq9v+/uKLLw6UH7AeeeSRio6uKV1Rp34T0s1d9Qiw//77A3DXXXelsjnmmAMoPyxNP/30aXummWYC4F//+lcqW3DBBQHYdtttKzv2DF1Rp47q4rOf/Wwq+8tf/gKUX14n/xuUH7b0gPaBD3wglb3yyisATJo0KZX985//rOCoS3SkTvVw7g/xuQfydl8cl1lmGaA8HqiufBLA0QPuq6++msquvfbatn5vlHRdOxUnnHBC2tZDqb9UffCDHwTgqaeeGlQG8NGPfhQoj6eqf38h23zzzSs8aqBL6jQ3njozzDADAG+88UYqm3vuuYH8ZAw0Xgjuu+++ag+2NV1Rp5CfQPE6Et6u1lhjDaD8Auv1/uCDDwJw6qmnDtpPq+s43M8FHadr2uoURFt1GtLBIAiCIAiCIAiCiqkNZ4YhQohNuaMoipWH+6Wq6lQzpl/+8pdT2S677AI0IgYAs81Wj3K+8847qWzmmWduum/NFnp0QTNlf/zjH1PZSSedBMBll102/BPIM651KnymUNGF6667LpWtvfbaQ373//7v/9K2JJjQmEH066C/b7nllqnskksuGelhD0VX1KnzxS9+EYCjjjoqlf35z38G4MUXX0xlivg999xzqezRRx9N20sssQRQlg9dddVVANxzzz2p7Mwzz6zs2AfoSJ0OJeeD/Mywy1nXX399AFZcccVUttlmmwHw8MMPD9qPR1tdWvTaa68B8KEPfSiVKfpy8cUXp7KLLroIgGeeeWbIYx4mXddOFWlxaa8ipo76sY8b3iY1dnrfV+TQ97fBBhtUcdjOuNZpThKoNu4RQUkrvX7U/t58881Bn4NGZPHEE09MZd/4xjeqOOxWjKhOYeyfp5ZddlkA7r777lR24403AuWouUsHdW/ziGwuSqbrWFXEKqSDHaHrxtQpgLbqNCJaQRAEQRAEQRAEFRNrtHqYI444Im3vs88+QHlWWxEoj0QpUuAz1For4Os0fB2LZhZ9hlYzsFtssUUq23rrrQG46aabUtm66647vJPqQny2Tyy//PJpW3Wq2X9ozGq79t0XFGvW0KMWCy+8MNBYEwcdiWh1HTIT8TUtuVlTRbe8nXr0RREHjyLOOeecADz00EPVHfAYodnhVmsc1PfdpEF15Oc9ceJEoNx2FUnwduoRL9WlRxcUFZ9vvvlS2U9/+tNBnzvooIMAeOGFF4Y8x15Cs/p+DVRvPl5q29e6eJtVO/X9qO3n1iROKeT69E477QTAoYcemsoUedlhhx1S2U9+8hMAVlhhhVS24YYbpm1Frn/5y1+mMl0bj9BUHXnpVnQP0XpAgJdffhmA1VZbLZX19/cDjTYJ5T689957A+X7uKJc/vzRgXWvQTDFEBGtIAiCIAiCIAiCiokXrSAIgiAIgiAIgooJ6WAPIqmQL/Z96aWXgNY5bmTn7IuztZ3LSwLlRceTf8d/T9KQNddcM5VpwbwbPEwJuHmAJIMuv8jl0nH5kKSXuXxk88wzT7UH2+VI/ucW4jK+kCwTGrJYb3Mzzjhj2pYsyOWzasf33ntv1YfdcZrJnGQgAo36c+ml0ga43FdGC25go1QCGj+g3Cb1215/MtXwFA+y3Hc54WGHHQbAZz7zmaFOsafYfvvtgbJ50LPPPguUpZet8mhJgujf+chHPgLAxz72sVS20korAXDHHXdUcwJdiGR9zz//fCpTu/nd736XyjbddFMAFlhggex+1B+8D+SYEiWDaifbbLNNKlM7uuGGG1KZxkqXsEsmLPk2lKWDMs7wlBmSE/vzh9JnuFTZpfRBMDUTEa0gCIIgCIIgCIKKiYhWD/K9730PKC/618y9z5Iqaa6jBdo5O1dPZOp2rpoB84iMoleKzEBjBl6LbqGxiNYTTfbyTJcvLhaKHvhsqWawvc58Ubbq37+j6+mzi1MDTz9dz/m33HLLpTLVj7dTzbT6wmuPFCgq4xEH/b0XzTByES1FO+edd95U9sQTTwDlKKt4++2307bartuT67uLLLJIKvMZ71tvvRUoL4ZX9MHHCJnruJmDxp/ddtstlclavxeTmCqhtqccUBTW+6z6uZLsQjlKoDbtqgJ9x8cXJUXvlYhW7pp6JESpBjwKrfuHjIAAllpqKQA+8YlPpDLZunvdu/mLWGyxxQbt281YpM7we1TO7KjbcSOKq6++GijfVzXe3X///als/vnnB2D33XdPZWpbboDj/XqrrbYC4PLLL09lSmK8+uqrp7KNNtoIaCRABrjgggsAeOyxx9o/sSCYAomIVhAEQRAEQRAEQcXEi1YQBEEQBEEQBEHFhHSwB9HC6dxia5cLKqfICSeckMokFXAJhiQuf/3rX1PZM888k7Yli3HJlhbbPvfcc6lMx+OmEJIUydwAels6uPTSSw8qk3TQc5NJWum5Y1zmJlxaqPpzmeXUgKQ799xzTyqT5M3lSAsttBAAM800Uyrzvz/66KOD9i1pnMs2e4WcpEkSKz8fyYXdJESyKW9f+rtLt2Q48IMf/CCVufxP+3ZJsmRXLjVWn3epmNqz5z6SdLBX5IKOZGku5VOfd8Mg9XOXbXq9CBmI+LZfc+WA6xVy13TJJZdM26ussgpQlqmpz8p0ARr3Ize1kdHDnXfemcp8nNR18DqXSYzLEjVW61/orfuR7j+S9AEceOCBQNkIROODxj//u4+fp556KlC+P/t9TDn3brnlllSmHJEuyZSc2HP0ffWrXwXKxj1BMDUSEa0gCIIgCIIgCIKKiYhWD6LZal9M7TP74uCDDwbKM6ea4dasFDSsWSdMmJD9vQceeACAJZZYIpVpBnv//fdPZbLldZtuze6utdZaqUwL7HuRZZddFihH93QdvE51jTy651blwq+bvuOzslMDmgn36KjanLPDDjsAjZlqaCycB7j22muBcsRBM60eUXBjgl5D5+t93w1phNpQzsDG26Qi21dccUUq82iZvuML2tVmPXquiJcvpBeKZPQibreuc5RNPjSi/R7N0djgaRr8eimy6FEw7ds/l0v90Gt49ERtyCOhqktvkzJj8UjTyiuvDDQMQgDuu+++tD3bbLMB5SiYjJ/c3EURQ4/a9BKqB9ndA+y1115A2d5ddecmQIrIejRM9S6jDCgbu8hwxNu8yqQwgEYd+7h96aWXtn9iQTAFExGtIAiCIAiCIAiCiokXrSAIgiAIgiAIgoqZqqSDLqPJ5TESLsWRfMMX1I5HXojcYmpfOJ2TD51xxhkAbL311oP+5rmGJBk89NBDU5nn6Npll10GfUd5fCZOnJjKJB100wdJj3xBfC8j6YrXvSSDLrmSYcmf/vSnVOYLhSVrcXmQ9vPss89WfdhdjfKybLDBBoPKvH4kS3Hp6fHHH5+2VW8uQVQ9u8FDLyOjAJcD5/q+pD4uZ5U8zWWvkiK6EYn3cy14d2MGmWl4zidJEF3K+eSTTwJlyazGMT+GbsbPMSfplYzS25ekrbfffnsqcxMdSefcfEhjpo8hLiPsNZTTzaV8akt+P7r33nuBvOTUzV0ks3TJnxtaqP78fi6JsEuF1R+8X/QS66+/PtDoW9AwEvF7turO5ZXzzTcfUDbCUg4uf75xSesyyywDlJcDqE94LjI3yxEaq6aUHJpBMFIiohUEQRAEQRAEQVAxPRfR0gyimwh4dGGuueYCyhnKf//73wPtmwzkFiFvv/32aduzso8VOatfP+/c4l7VRY4dd9xxUJkiYFCeTVUk0C14tUjcZx2bscgii7T1uW5HhiA+m6rroFlcaMwarr766qnMZ1s1A+vRP80K5kwzpmQ0u+z9U0YLikg5Pnvq0RzVpbddRQh8xrzXTAY8qiK8rclwwKNSap8exRc+bqgu3LTAo+caZ73O1fe9HvUdt44X3sZlJuPRnm5GBgLQaEu5+4j3bdWPRwncllxmAp5CQ9fL00H0Wjt11A68fyoC4u1Zxgtep6pn78eK/vm4688AGjM90pMbYxWtyY0hvVDfMq9woxX1JY9U6ZzefPPNVKZr4lFTqXOkwIBydFZt1c1KNCb79fnjH/8IlJ+T1P7dvCgiWkEO78vazqU3acW6666btmWOVRVSIozErCwiWkEQBEEQBEEQBBUTL1pBEARBEARBEAQV03PSQTFUWHGdddYBYLXVVktlkt0dc8wxbe3b80hssskmQHmh6XjgC0pzSBLh0gpJB106IRTqdy6//PK07ZnilSPjE5/4RCr7wx/+AJTlhJIR+u9JpuA5d3oZSSxcfpGTDp5//vlN9yNJl0uFRM74ZEpGoXhfoK46dcms5D4uw3LJluSzvphb9ez9otdYYIEF0rb6mEuyJGnwupChhddFznBAfdXboY+tyk/k6LddfqVr5wYP+pz3FZ1Lr0gHF1988bStdup5oHRuLofLyaNuvvnmtL3ccssBeTOjXD6uXkTjpJ+DZGwuB9Z5u1xabdLrR23XZW0uXdPfXUIvEwy/n6vNupmMZHFu+NCtqA5d6rvZZpsB5XanenDDCvU9z5mlbc+R6XnH9Bxw8sknpzKNyWrHAOuttx4Aa665ZipT/efMeoLA8XEvZ1An/BlehmwA1113HVA21JKMuJW5mMYEv0+Jr3/962lby21kSAPtL52JiFYQBEEQBEEQBEHF9FxESzPU/vapbOnQmJnxmRwZMVxwwQWpTDNoPgP29NNPA+XFm5rtcsvo8UBWqY4vIBRuZasoks8M6ju+yPvwww8HypneHVlt++yurGL33XffVCYDEp+d1IxmM2OOXkKzo17PuRmYs88+e1CZL7ZWxMFnD0WvWg+PFNWlt9PcTJHK7rrrrux+1Jd9plt13ssRLZ+507nlotT+OY1lHlHQ2OkGGapTH099P7nxVnWaM8jwfqE697rX4vpewQ0tFAXxiLPqxSOvp5122qD9eETgC1/4ApA3KvF6zkW7ewX1RW9/Oh+/50qpoXQE0BhPc+Oq14/Xn/qDj7E5U41cnfaSjf4dd9wBwOmnn57KFEXyyLeeYdQvoREFc+WFDDLcht/rVdfHnz/0POWRXUW+PVKt6NvUZu4U5MmlYGgWvXJVlVK6+HOVp85Rv/bnqWOPPRaAbbbZpulx5SJZu+22GwA77bRTKlMf8efgdpUZEdEKgiAIgiAIgiComHjRCoIgCIIgCIIgqJiekA7mzBU8bO05oSQd8IXfCvm51E779LKllloKKC+e08LdXObzsSS3KN2lVs1kQd///vdTmRbHb7zxxqlMi1qXXnrpVOZSAoVKJTEEmDhxIgDLL7/8oOPyY9Ax+qL8XkayPpe25dqGzEKcm266KW1LZpmTD+XkhFMyaiMuMcvJh3JyQl8cL0lXLidPL8uwXJam83BzHi0291w3qlNvm/qutznVry9Y97/L3MIX30tq5RIwHY+b9kg65ON3brzoZrxO1da8Tap+fXw7+uijB+3HJSa6Nrn7Wk5q14vo/uvnoHbjxiFqV25OIdlbTqLp+/O6ytWp2qSMGqBhpOPXMCfB7yb8vrzzzjsDZQmVjt/boOrTx8yc9FXbQ92fdS/yfFzNrsVll12WyrR0YcKECanszDPPzP5Ot6E26HJMPXO6tPree+8F4POf/3wq0zm+8MILqczbdy43ZM4AJoeudTPJ3ViTO6ZcWe7c1P7cLE2SwJ/97Gep7Ec/+hFQzhXppi66Ng888EAq22ijjYCydPWHP/whUF5GpPa81lprpTItiXFZoYzfnn/++UHn0YqIaAVBEARBEARBEFTMmIVpfNZIb7k++5SbwW5mga0FxQAvvfRS2tasmb/tanbNDTK0b3/L1ky4z5RpRjNnpzySDNEjxRe1Cj921WVuVuvggw8e9F2fYVG9LLnkktnfVv16VC23gFjXLhfRcppd115EdZ4zDHCeeuqptL322msD+dlUvzZTA7Il9r6v9uyzr7k25zO2qkv/nGafRpJlvlvwxesam3xWVDOsF1544aDveJ0qYuhjmbZ93PDIoiI2rhBQXXo9P/TQQwBstdVWqUy/7eNpzmK+m/G60HjvdaoIt9+Dnnjiiab7VJTA+76up0cEe62uHLUrN0fR+XqUUPUmUwZo1K/3WV2H3DOD/z2nLthhhx3S9iOPPAKUow3dXs/e/zXzv+eee6YypV3p7+9PZTpPf+ZRH3djKqks/F7sNveKBjz22GOD/u5RbkUI3CZeShkZeMDYRLQmv6fmoj/+jKJz98jb/vvvD5QNwtTXfTx7/PHHgXJERqlz9ttvv1S24YYbpm2NkZ7yoVm0x3+vmyJZIndMuTI98zhqs94f9957b6B8HeeZZx4AVl111ewxSF3h37n00kuB8vOU3hv22muvVKZnCDfBk6rNVUj6bR+/XnzxxezxTE5EtIIgCIIgCIIgCComXrSCIAiCIAiCIAgqpiPSwZxMMBdKbCYrg7y0bJdddgHyi+egER53KYKkGr4oThKNofJHCEkVPLeR8kgMlc+nE+TMMByFl6+++upUtu666wLlHGCqU5dkSW6hhe+Tozp1eYzkFv4dhWh9wXvO2EGyToXdexFvz6qfVufj1yGXU2JqReF3b5PC+11uwbZLhSTtcqOIXJ/uNVzqJ0MGl6lqvPWFwOussw6QNxDxcVXjpEsRvU3qd1xCl5O7Sqrk10ufcxmtj8u9gN8zcu1Psi43AWiFxlG//+XkWL3cdtWXvV3lcjhKfuoyVLWh3PkPJUvXeJpr79tuu23aPvLII4GyHMuled2I92stA7jiiitSmdrO9ttvn8p0L/Z7jupr1113TWWSuXrOIjff0Tji11EyLn920pjxu9/9LpXJEMqPfyxpZjDhY+CKK64IwFe+8pVU9vDDDwMN0y9oGNq4FE2yTZlbQUP65m3RJZznn38+AE8++WQqO+KIIwC46KKLUpm30V5D+Qd9vNezu+ehOuyww4CyuZ2e7b1M93kfJ3J59FwGrDHoN7/5TSpT/foYJHnoM888k8r0HO0mMJ/85CeB/LKQVkREKwiCIAiCIAiCoGI6EtHKzdL7IlZt+6yCvpOLYvnCNb2JugW7LyDWrJlbD2tBvM/AaJbDF+vqbTgXkXM22WQTYGwjWrmZYJ+J08yVZ4zXbIufo/DrofMdysJedeAzuppl95n1U089FWht4azr1csRLZ/h18zLfffd1/Q7WpwJ8I1vfAMoX4epFbVPb6eKTnn9zDzzzIO+64Y0apM+69XLVvnqjx7py83yqy36guJc1Eljoo+xarteTz7maTsX0fJjefTRR4FyREvXzscV/Z6PXbkoRLfgEXtFm/x8NBv61a9+ddB3ve36rLpmst2UQIYwvu+55557VMfeDeTSELhltv7uM9Ha9jan+vN6zD0r5Mw3XImhOneb6G4fg6WgAVh00UWB8rnPPvvsQLk/atufg/QdRaSgYYDlJhYeQc9Zx8t8x8fj+++/HyhHbnTcyy67bCrzeu8UzZ4lc8isw80QPJLdDD1v+XOXcEO2b3/722lbz0f+PPrNb34TKPcNKT28nnU9vM3qWvuYr+t1zTXXpDJXfg2HaaaZJrUZRaVeeeWV9Hf1Vz8f/b7X46RJk4ByqgsZTHi/1pjgz5Y6X1d2uapN9xNv72rHXqbnBUUsAa6//nqgHLVVnXs0XOfi6Rbc4KwZ3T3CBEEQBEEQBEEQ9CDxohUEQRAEQRAEQVAxo5YO5sLuuQzRHhpsltPGF2Jut912QDn0J4mKS0881K3wby73gMtahIeXtcjNyxRq9GP2DNJjhYePc+ejBbG5rONeFwrpDseEIZcfS2Uua7rllluG/K4W8UNe1tRr5CRcvrg1h8smVG+5BfZjmZ+tG1B/cwlZTnbm+V2ExgNojBPeJrs9R04zJLHNSZm9XtS/vUzbLr/QOOlyDkmtvB36eCuJiI+JOh4vk8wlJ9fxvq/jd9mH5+jpNnI5wPzeo3aaW/A/lHGDZFY5CZ1LY3JjeTfjfU314gvHPf+MkJTUJX85Axb1af+ct/dczjbJBD0HZU6O2UvSQcmivb9qkf5BBx2UytTGfDG/ztPr8KyzzgJghRVWGPQb0Gijv//971OZcgv5M8lRRx01aD96PvHrqCUQflxV8v73v5+PfvSjQKPefPzRto9TRx99NFB+jlxzzTUB+MhHPpLK1L79/qI6XW211VKZxjaXzbpU7aqrrgLK9y4t/dhmm21SmYxIfD/qG95m1Q+8TPV82223MVpmm2029t13X6AhA80ZQnidyjDExzPVpcsOdc/3sVDSPJdW63z83uTjTW7Zi47Rr79koqusskoqU74zP36N5/6crL+P5H7V3SNMEARBEARBEARBDxIvWkEQBEEQBEEQBBUzbOmg5BAKozWTAUJeoqZw4nzzzZfK5K3vYX7JADx0qhCiyxBybnh+XPod/5xC1zlnIw/BKuzoMhA5US211FKpTKH6TuGugwqJeuhUIVh3DxIeEs3lKmolI5RUKOdG5seV20/OJadVTrBuRiF+l23qvN31LYfLuEROgji1SQeF98+cw1tOSuWSLcmCfGzIOW72CupbXgeS9Xi/kwOrO+RJkuWOazmJicYGH0tyTmXedrUfl9Bp22UhGk/9+PXbckqD7pYOutxXDlkuMZL8x+tZDHVvlPvol770pVSm6yXZE/SeY2ZuLPN7t0vghO6vLkVSG/H2pXY4lJNjTmost+GcE56Tk8a361g3Fqy00kppW7Jfd8iTE7P30QkTJgCN/HbQqM/11lsvld15551Aw80QymOLfufaa69NZcoZ5TJN5SBy6aDq312htd0p6eB//vOf9CykMc2fM3WP8OurPv7Zz3520P68r+t8XeKu8c5zNWkJgeTUw+H4449P27qP5cZeR89YuSUZVdTzG2+8wbnnngs02p87V+pe7a6Deo73XFhyYfQ8ZJIM+ud0vjkpossAfWyWo6H3Cy092njjjZuen447t7TIn8V0/UeSdy8iWkEQBEEQBEEQBBUz7IjW5DM9PgOnmQN/O9W2z5LqLdbfIBVZyi2I9wWJ2o+/5ft+NIPtM2SK4vgMg/bp39WMub+x6m3d32y12NHfnjtNzojC0WJL5XVx/POq05xhyVDkZvxUv35tfDZ78uP23/AZrl5Ds6NezzpHnxXMkcv0nps5zc2sTA14f1KkQLngoDzbJzw3iCIOijpCfpa9V1C/8zFR/c7b2kMPPTToc82ipx451G/4AnifNdSscM4wwBfDa3y89957U5lmCj0SqSjESGYFxwOfqf7MZz4DlPusZp3XX3/9VHbFFVcAQ4+rGqu9nebUFDnziF4hl6dSkRm/9+jv/nyg77raJDdOehvXd3J17vduRX+cXDSsmyJaN954Y9qW4ZTn8snlAdLfva/rPL2OcgYZrjjJ1Y326fczjT3eZhVx8LKcoVGVvPvuuymK4wYevYhHg8eTv//970mx9fTTTwP5aJ3fa3XfWHDBBVOZVAybbbZZKjvttNOAcnRKkfzc89JwuPjiiwHYdNNNU9ndd98N5PPx+f1TfcTfZfTc72Xt5luLiFYQBEEQBEEQBEHFxItWEARBEARBEARBxYw4j9aGG24IlPNeKdTvC50VevZQnT7ni7clJfH8Kgrf+YJEhcddYuEyFIUvXS6g3/FFeH6Mk5OTuri0QVLEnDynU7SSNWjR67rrrtv0u8LlAzmzi9xnvc5z5y4pjEticvJKXzTZaygvhZuOSM613HLLDXt/3rYn39/Uhi/SljTTZQa77bbboO/cd999aVtSNuXFgIYkQfkzegn1He9rGod8wbrO0SU/uT6mccDbnMZJH1N8rNbvuSxE47d/bt555wXg8ccfT2XKReNjp2SOvSKL83rRefv9RtfG26akg0PdH1577TUgL7v3fbucsxfIydT8nqL+6fceyXVcjqP7q7e5XK44R9fJ25qul5uK5L7fTHbYDbjBhPrX8ssvn8pkOpHLF+YmLRoT1FehYWrgeYw815jq09uqvu/1qucPbwM6Lq9zLcXwZ7Ggu3n33XfT9dK4vcEGG6S/6/q61FfyTb8/qy39/Oc/T2VPPPEEUDZp09KS3D1sqDxaGm98zFX78+cp5SaThBAabda/q7EntwyqXbmgExGtIAiCIAiCIAiCihlWRGuGGWZg9dVXBxpWmJqhhMYCOV/EpzdDX9iWW6CuqJO/2WqWymc/NeuUWzwLjbdTj4xpNsbt2PW5VvbaMibw2UX9PWf+0Cl8gXouoqU6kE0+NGYYcgvZW+Hf0axkbgGhs/DCCwPlWTRdB7/+vWxemRqPAAAgAElEQVT2IJvbvfbaK5WpnldcccW296P6y7W/blqIPRaoT3tdyIbZrb9zM/w+CyVjltVWWy2V+Qxrr6H25GNdboZZEfiVV145lWlxu/dZbeciBV7m2xoHfFZQ2173iub6TLXGLJ951KygH6usg7sdRZv8eqhNyohlOHi96Fr7/c9/pxfwY1cbykVjPcql9uLtWX/36J72nUsxAo1+7vctPYd4m3RbaqFjHMl9cizYfPPN07bGygMOOCCVXX755UA5aq++7mZBOvdbb701lcnkwM/d60vRCo8AKCrlyh8phH7605+mMhmPzDXXXKnshz/8IQBPPfVU/mSDrkapRPSvo+c/aIxtXqZ24/1WbcRVFmp/HiHTd/zZ3Nuf2qlHpfXs6QYsut/lotdu7iT8XUaRLFdttEt3jixBEARBEARBEAQ9TLxoBUEQBEEQBEEQVMywpINvv/12CjtLQrjMMsukv6+11lqDvqOwvBtfKATni8oULnT5gcJ7uSzoLj9zaaFCjG5MoMXiHq6WmYeHLHNmEDp+LayDRjhxLHPBuJwsJzdT6NTrSvKhdnMJDWWGIVyGlNvn1ltvDZTrWQt5/bsKIfciymniMja1keFISdUfciHsXs79NBLU7nKyqVbGIC4NVB/w3G5DLZ7vBSSTcImZZDi+UPiuu+4CygvktRg5J9P1Nqfxz9ucjzU6Bpf+qr17n55//vkBuOiii1LZKaecApRzUWl/uTws3c4NN9wAwK677prKZAjgOVjaRTlpoCFb8T7QrVK2ofB2lbuXyKzB5cD6nLc5tS+XHeak1rmx09ukePDBB9N2Lo9Wt0sHv/a1r6Xtm2++GSg/e0jK5AY5Gvf8PqUxwaX9eq7xc3dTDY2lfj0lG/NxSe32pJNOSmXK7+X7Vlkw5eH9OocbY0xtdOfIEgRBEARBEARB0MMMa7rXs24feuihg/6uWRZfjL7ooosCDatfaMx+LrvssqlMi6Rzs2I+S6Uo2L333pvKrrzyyrStbOCtrHE18+pWp7Ld9eibtn12TbPsjz76aNPfqBKf8fOZJCG7cZ8R1XH6rH4zK9uc5TvkZydzURddV8/yvcMOOwz6XC8bFGgW2hdJKirg10UZ0WVfOjla6JmLuExtES3hURNFqX3xa47crLe3L5+97TVOPfXUQWUaY9W+oNHGtt9++1SmhcI+862ZZY3h0LDS9TrzKL/aohszaDzwRcZSOBx//PGpTHbzHu3pNctyR7bEPqZpPPVoQqu+L/w+owil931f7N0L+D0jF1nSvdbTf+g7PnZqbMzZN/u9KGfYlDMQ8XrWeJuL4HZr9FupLqBxT/fjf/jhh4Gy5fZ2220HwEorrZTKlIpnjz32SGVqt/4c5KlLdA08yiWVipsH6BnMU0zI4MSfSRQh87EjCKZ0IqIVBEEQBEEQBEFQMfGiFQRBEARBEARBUDGVxsolEbn66qtTmbZ/9atfVflTo2arrbYa70MYFi6rysn+ZDDh0gl9JyfjyJUNlaNE2zlpoefcWGONNYBGlvih9t1r+WFy5ORVLpFoJR+SGYDkltCQxXbrouxO47niJFlpJTXL9QuvP8/FMSWgMdbluZKduRGO2pLLoV5++WWg3P/0He/bXqfqt97ecwYlMt1wEyLJuKcUZBzg0ktJ3r3vK6dWK+mg16PGb9+P13mv4ech1O5cci/ZXq6f+z0qZ5jjNMs9KFMoPwY3iZH8eKh9jzdqY9CQ5rlE7/bbbwfKObN0D5aBCzSWavg4O3HiRKCcY9T3o7H07LPPTmXK1+XSwcsuu2zQfnTcLl/u5RyaQTBSps4nuiAIgiAIgiAIgg7Snas/g0H4zLxmpHym6MgjjwTKC2I1e9dstg8as4UedWplO659urX+pEmTALjkkktSWV9f36Bj6NaZw6HIGYNccMEFqUx2zx5JWXvttQG46qqrsvvMmTzod3zGfGpCmdyh0dZaRffcaEEz4N5Offa2V/H2p/rw/qS2love+fnruwsvvHAqe/LJJwd9R4vY/bfdmEARAt+3oj3rrbdeKlNEq5WxTjeTO/YrrrgilckYw6OASnNxzjnnNN23jwG6Nt7ec2NwN+NtJHfPUfReKTIAFlhgAaBstqDolpuB5EwsPForM5ecoYW3U5kx+H7cUKcb8VQOc889N1Duw+qPm2yySSrT+Xl9qI7d7l5t2qN+Hi2XEYffk5TGxMcJ7duNR+abbz6g/JySM/IKgimdiGgFQRAEQRAEQRBUTLxoBUEQBEEQBEEQVExIB3sEX0QqWYZLhSTHUy4wgEUWWQRoZI6H5lKsoaQqKvfFyZJb+IJYSQr8GCY/ZmhICnqFnHzowgsvTGW77747UL4eyml0yCGHZPcpSUfOdKSXcw2NBpk1AMw+++xAa1mPy4vUxtxEQG2yl/E2kpNkLbbYYkDZmEbjgX9eOQ2feuqpVCb5mnLsQD5/kRtoqD+4XE7bLv/MHb++2ysSQh8vVZe/+93vUtmOO+4IlOVpkne1Ine9ZGICZXOTXsBlahrDXKKndiXzBsi3JdW5DEKg0U59LHaTCMnTvF3p99zcQXn1/BrJOKJb8zt6ztCbb74ZaPR5aNx3XGKoMkkloZHrzu/PG220EVCuSzdxUU5Uz1WqunMjJ9Xhtddem8qWXHJJoJxz0p9FgmBqISJaQRAEQRAEQRAEFRMRrR7BFxDLRt0jH5pR0qz1eCJrc2gsjvUow2233TbmxzQafFZbUT23rlZUxc8xZ5/v3HfffQAss8wyqUyz4h5dmJrwSMHKK68MtK5HX3ytmVOPyHj0ZkpAEYJchNhNZmSh7fX38MMPA+WoiWad/XM+s6/f8XpWJCZnRe6Rd5W5jXmvRbRy7c8ts2UC4pEDRfXc6v7uu+8etB+f6Ve9eQTXo7W9QC4672OZ2su5555b+W+//vrrQ/7NI2iK3LhplMZij+p0E08//XTaXn/99QGYd955U5naqLe3F154ASj3RxmP5FQA3uf9OxpLPVqmeppnnnlSmfq193WZZaiPTP7bQTC1EBGtIAiCIAiCIAiCiokXrSAIgiAIgiAIgooJ6WCPcOutt6ZthfZ9AXEridVY4jIEyYdcZuS5j3qBVnnInnnmGaCx2Bga8oo111wzlbn8UxIwl7mp3madddZRHnFv4lJY1Uurundk2OASIJetTAnkJHcHH3wwAF//+tdT2WabbQbAjDPOmMqUM8tNW1Rnr776aipzEwJJhtz0RpIgN3PQAvtjjz02lbmMSHTTONUOrSSO6vtbbrllKpP8T0YDkJcOuhzLzUaE5ynqBVzOJimlSyq/973vjfkxTc4xxxwDlPPHSerpEvFukrhJ2giw//77A7DKKqsM+twZZ5yRtnUv8vFThiEus5TM3yWrLh3UWOr9Vvdyr6OHHnoIgGWXXTaVSRbv8u1ekQwHQZVERCsIgiAIgiAIgqBiIqLVIzz33HNpW3a1HgGQ/a0ju12f1RrKwn2k+P70O4899lgqu/TSS4HyzKYsanuFVrNwJ5xwAtCY1QM455xzgHIUyznzzDOBcr3IcOC6664b+cH2MKoTgHXWWQcom4604qKLLhpU5tbIUwK5iJBMVA499NBBf/Mog4wvPFIywwwzAEOnfVDU3Ge8FcVxU4hei1JXxfe//32gYRsOjTqbNGlS0+9OnDgxbSu1wZtvvpnKrr766qoOc0zwe5CiHm6i0qw+cik0OsF5550HlNUgbkHfjXjfO//88wF48cUXB33OI1++LU455RQA7rjjjlSmyLdH/j0Cpd954IEHBv394osvHvQbvm+NVc8++2wqi4hWMDUSEa0gCIIgCIIgCIKKiRetIAiCIAiCIAiCiqkNJ5Rbq9VeBZ5u+cGpk/mKophtuF+KOm1K1Gn1RJ1WT9Rp9USdVk/UafWMqE4h6rUJUaedIfp/9bRVp8N60QqCIAiCIAiCIAhaE9LBIAiCIAiCIAiCiokXrSAIgiAIgiAIgoqJF60gCIIgCIIgCIKK6VgerVp/bRvgAmCJoq94qI3PPwWsXPQVr01W/lbRV0w/jN8d1ueb7GdP4Iqir3hhtPvqJLX+2rvAvdSv5YPAHkVf8U6Tz58GXFL0FefW+muTgK8VfcXtY3GsvUKtvzYLoCQ2cwDvAq8O/P+qRV/xz+wXgyGJOh0/av21bwG7Uq/z/wCfByaSH2+3ApYs+orDM/v5OPDPoq/IJ4ebQoi2Oj7U+mtzAEcDqwBvAi8DXy76ikeGsY8ZgV2LvuKXnTnK3iL6/ujI1V/RV9wyyn1OosVz15T0bNaJOrR9f5x6PW1Rxf46RScTFu8CXD/wb18Hf6dT7AncB3T1ixbwt6KvWB6g1l/7H+ALwE/H95Dq1Ppr7y36indbf7K7KPqK1wHV6SHAW0Vf8RP/TK2/VgNqRV8xOINsB6j1195X9BX/bv3J7iTqdHyo9dfWALYAViz6in/U+muzAh8Y6vNFX3ERMCjzc62/9j7g48BbwBT9sBVtdewZqM8LgNOLvmLngbLlgI8Cbb9oATMC+wJT/YtW9P3RMdz6CwbTzXU4lmNqR160av216YG1gQnAxQy8aA28fR4CvAYsDdwBfLroa1gf1vprHwLOB84v+ooTJ9vv14FPAtMAFxR9RfYFrtZfOwrYGHgJ2LnoK16t9deWB44DpgUeBz5T9BVv5MqBDYCVgf+p9df+BqxR9BV/G2W1jAXXAcvW+mvzU49aLQ1Q6699DZi+6CsOGeqLtf7aLsDBQA24tOgrDqz1174ALFT0FV8f+Mye1GfC9qv11z4N7E+909wC7Fv0Fe/W+mtvAccDGwL/Rf1le4qg1l9bmPqN6E5gBWCjWn9tAnAg9Xq7qOgrDh64Mb1W9BUzDnxvZ2DDoq/Ye2D729Rnd/5c9BUTBj7/I+p95oPAMUVfcVKtv7bhwGffAhYClhjL8x0Lok47zseo19s/ADSLXeuvAXyp1l/bEng/sGPRVzw0WR8/Dfg79evyPLAm8O5A3/9S0VdcN9YnM55EW+0oE4B/FX3FcSoo+oq7a/21Wq2/9mNgM6AADiv6iokDzxgXAjNRb7/fLvqKC4HDgYVq/bW7gCt175pKib4/Ooaqv+8CWwIfov7i+fmirygGolC3UG/LMwKfLfqK6waeaU8FlgMeGvgeA/v6FfUI7oeAc4d6pu1hhqrDp4DTqdejt8HpgGOpvx+8Hzik6CsuHHimPROYbmC/+00eXa3111YBTgB2oP7sn9vPnsB2wPTAe4H1OnLWk9GpNVpbA5cNhPxfr/XXVrK/rQB8GVgSWBBYy/42PfUXs7MzL1kbA4sAq1KfbVyp1l9bN/Pb0wG3F33FUsAfaUTTzgAOLPqKZalL7YYsL/qKc4HbgU8VfcXyvfCSNXCz3oz6OQz3u3MCRwDrU6/bVQakn+cB29pHdwLOqfXXlhjYXmsgmvYu8KmBz0wH3FL0FcsVfcUU85JlLA4cVfQVS1J/uDqM+sC6ArBWrb/WKoTdB2xQ9BXL0ajbfYBXir5iVeqD7n/V+mvzDvxtZeovsVPyQ1bUaee4Apin1l97pNZf+2Wtv+Y3lteKvmJF4FfA14b4/tzAmkVfsR31CamjBsbEqeFBK0e01c6gidfJ2Y76PWk56pN3P6711z5G/SVg24H2OwE4ciAqdhDw+EAbnZpfsiD6/mgZqv5+XvQVqwxMZH+IesRGvG+gH3+ZxjPmF4F3BvpwH+DPw98q+oqVgWWB9Wr9tWU7eULjwHDb4LeAawbqcAL1/j4d8Aqw0cDndwKO8R+p9dfWpN5Gty76iseb7AdgRWCHoq8Yk5cs6NyL1i7AOQPb5wz8v7i16CueG5Bc3AXMb3+7EDi16CvOyOxz44H/7gT+RP2Gt0jmc/+hrkEG+DWwdq2/9hFgxqKv+ONA+enAukOVt32W3cGHBmbvbgeeAU4ewT5WASYVfcWrA6HU/wHWLfqKV4Enav211Wv1dQuLAzdQj/itBNw28NsbUH9phvpL13mjOqPu5vGioZtejXpnfq3oK/4FnEXr9nMDcEatv7Y3jf63MbDXQF3eQn02TG37pqKveKbSM+g+ok47RNFXvEW9r+5DfZ3RxIFZPagrB6D+gDv/ELv436IH5b8dJNrq2LI29YnXd4u+4mXqk6erUH/J/UGtv3YPcBUwF3WZYTBA9P3R0aT+JtT6a7fU+mv3Up+cXsq+lqvXdak/i1L0FfcA99jnP1nrr/2J+nPtUtQDEFMMI2iDGwMHDYyFk6hH+OelHpU6caDO/5dyPS1BPZK1pY2VQ+0H6pHuP1d2km1QuXSw1l+bmXrjW6bWXyuoh+eKAdkfwD/s4+9Odgw3AJvW+mtnFX2DMinXgB8WfcXxwzykKT0jc1qjJWr9tX9Tfon+4Cj2fw51ueZD1OWaxcDM4elFX/HNzOf/PoUPzm+38Zn/UG+vwuv/c9Qf0LYA/lTrr60w8Nl9i77iavscA9Khdn6v14k67SAD/XESMGngRrXHwJ80Fk8+DjtTVV21QbTVznA/dclPu3wKmA1Yqegr/jUgRRrNfW6KJPr+6MjU3+epR59WLvqKZ2v1NZze7tqpVwBq/bUFqEdyVinqy1hOYwpsw8NsgzVg+6KveNj3MVDPL1OPbL+HekRbvEi93lag4akw1H5WYxzadSciWjsAZxZ9xXxFXzF/0VfMAzwJrNPGd78LvAH8IvO3y4HPDGizqfXX5qr112bPfO49NAbsXYHri77iL8Abtf6ajmE34I9DlQ9s/xX4cBvH3I28DMxe66/NUuuvTUM5tJ3jVuph61lr/bX3Uo9Aqh4uoC4F9Sjl1cAOqv9af23mWn9tvqpPoge4hfrs1iwD0s2dqber/1BvV4vU+mvvoSy/XLDoK24GvkO9rc9FvW3vO7APav21xQZ03VMjUacVMnDeHvlfHnh6hLvr5TGxE0RbrY5rgGlq/bV9VDAgo3oT2KnWX3tvrb82G/XowK3AR6jLLf9Vq6+T0/0n2ugA0fdHxxD1pwf31waeRduZHLiW+rMotf7a0tRf1ABmoP7Q/5daf+2j1Jd+TFGMoA1eTn39YG3g+ysMlH8EeHFgbN2NegBHvAlsDvywVveBaLafcaETL1q7UH84d86jLB9sxgHU5XA/8sKir7iCujTjpoG34nPJd/y3gVVr/bX7qEfWDh0o34O6TvMe6he7VflpwHG1/tpdvXbTG5CxHEr9hnQl9WhUs8+/SF3b/gfgbuCOgYXFFH3FG9Rt4+cr+opbB8oeoL5I+4qBeruS+qLHqYqir3iO+gPTJOoy2JuLvuLSgT8fSL2z3wg8Z187aqD93gv8oegr7qNuHvIocNdAu/0VnXUE7VqiTitneuD0Wn/tgYG+uiR1Q6KRcDGw7cCY2M7E2RRNtNXqGFCwbAtsWOuvPV7rr90P/JD6Pf8e6vela4BvFH3FS9Tl7SsP1OXuDNzjirpj5A21/tp9tbqJxtRM9P3RMVT9nUjdkfpy4LY29vMrYPpaf+1B6s9ld0Dd7IW6ZPAh6u38hqpPoAsYbhv8HnWZ4D0DY8D3Bsp/CexR66/dTX0JSykqNSAr3gL4xUDUaqj9jAu1YpBCLwiCIAiCIAiCIBgNnTLDCIIgCIIgCIIgmGqJF60gCIIgCIIgCIKKiRetIAiCIAiCIAiCiokXrSAIgiAIgiAIgooZlrPR+973vmKaaaYB4J133hn092mnnXbIv+U+5+g7/rdm+8ntI7e/dj47UiY7vteKophtuPuo1WrJjSRXf82OvVU953jve+uumO++O36prto9z3feeacjdVoFqkeA2WeffdBvvP/97wdghhlmSGWvv/562n7f++pdz81o1LdefPHFVNbu9R9GO+loneb6b6s+7XUk1D7/85//DPm3zDECMNtsjdNTXX7wg430JM0MgMarTofLSMa0v/3tbwC85z2N+TW13ddeey37Hf39X//6Vyp76623hvW7I+x7Y16n7TL33HOnbbUlb1Oqs+eff96PK23POeecQLkdq/79c2+++eag327V51p8vqP3qHbL2jjOEdOpe71TRZ1C+2211Zg6XGaZZZa0rTHBx0xvl3//ez1d0VDjQzPaHf8nP5d//OMf/Pvf/64N+mAb+DNq7jfHol22+o0qrmErhjjuysbUdp/Te51m7yoDtFWnw3rRmmaaaVhiiSUAuOOOOwb9vdnfcp9z9B3/W7P95PaR2187nx0pkx3fSPNTJHL11+zYW9Vzjg9/uO6In7uJjxXtnucdd9zRkTqtAtUjwOc+9zkA7rrrrlQ2xxxzALD++uunsrPOOittzzTTTAD885//TGULLbQQAIcddlgqa/f6D6OddLROc/23VZ9ec801gfLD6l/+8hegcaN3hmq7evndb7/9UpnqcpFFGqk8/KVhcsarTofLkksumbbbdY695557AJhuuulS2d577w3ASSedlP3Ol770JQBefvnlVHbttdcO61hH2PfGvE7b5Stf+Ura/ve//w3UHw7F/vvvD8A3v9nI5+4v+t/97neB8gvrySefDJRfgi+88MJBv92qz7X4fEfvUe2W+cuk2m67bcQnuHITLp261ztj3fdbjanC61XtKFdHu+++e9q+9957Adh3331T2RtvvJG2H3qonhnmlFNOafp7uTFIx+pj1e233z7oc5Ofy4MPPjjoM+3iz6iTH4f/ViefsVr9RrNrWBVDHHdlbbXd5/Rep9m7ygBt1WlIB4MgCIIgCIIgCCpmWBGtd955p9K319y+2t3/SI4j952VVlpp2PupkmmnnbbSqMuGG26YtnfddVegET0BmHXWWYGGZABg5plnBsp14ceiz3p0QTNlf/zjH1OZZsXnnXfeto4193tVXA+v007R39+fttdaay0Attpqq0Gf+7//+7+0ffDBB6dtSQdz8gLNbg9FJ/vIULRbp+32ab/OO+64IwCLL754Kvvzn/8MlGWUCy64IFCWZD3yyCNpW9//9Kc/ncpuueUWoBHNAfjQhwbnHx/vOm13vyuvvDJQnhlWXboEU5HUFVdcMZV94hOfAMoRGY0RLtHUGAGw/fbbA+U6+853vgPAxRdfnMp++9vfAvDMM8+ksvGY6ax6PM2x1FJLpe1XXnll0N8/8pGPALDRRhulMh87L7nkEqDc9+eff/4h9zcebdPJ9f2h7hXNynKydUVFbrrpplQm2bX/htq7R7M/8IEPpG1FqU888cRU9thjjw15TrljbXVOVT8rjKSt6hhafT53rMsuuyzQiLgCHHvssQBcd911qcwj/l/96lcBOP3001OZrp9HsZrVV7sR96rbb7vtcjS/m7seI+mvues1mmeidttJFYzlb/UqEdEKgiAIgiAIgiComGFFtHK0mgXK0exzI9nfaH6v1azCeNDucVx11VVpe4MNNgDg1ltvTWWKRHn0SvrrLbfcclTHcPXVVwOwxRZbpLKtt94aKM9O+ux5s313S907zWZqlltuubSt6IsvGNY6GF9z4WYYWtvhOveFF14YgMUWWyyV/fWvfx3WMXfjrFKra6uF2E899VQqy60tUBv/9a9/ncp8YbciCf7dj33sY0BjrUG30u6soGaHc3V6yCGHpG21u4cffjiVTZgwAYDjjjsulR199NFAed2W19WTTz4JwNtvv53KZPYw33zzDdqPR2k0W9+JMX0oXHXRqZlWn6FXZNrXWgqPxvraos9//vOD9nPooYcC5bFajGX9taJZ5KLVseX69E477QSUFQIaW/28pQzw3zjwwAPT9hFHHAHAxIkTU9lqq60GNMZaaIy3zdYUjQft1mG7x6jovvoqwKuvvgqU26LqXWMnlPuwIl7rrrtuKlt77bUBOPzww9s61lbjf5X1XpXqaryiNCOJuI3ls1Oz3+rGZ7iRojqv6v0gIlpBEARBEARBEAQVEy9aQRAEQRAEQRAEFTNi6WC3hglHE/Id7gLYKiQdowl1zzjjjIPKVl111RHta7hIxpVDkg1o2HS7rClHlWYYuTodzX5z13n66acf9HctcoeGPMNtn12yoVwfLuHUvt1M5P777x/xcVdJu3U6krZ83nnnAbDCCiuksgMOOGDIz7vZxamnnpq2m8lh3QzDf6cdxluulatnt7D/zW9+A5SlVLJt9zYnowU3sJlrrrkAeOGFF1KZy+AksbrvvvtS2WabbQbA1772tVQmswyXE8o6es899xx0LjlJRqcWw1d1r1JdyjwI4NlnnwUaEkLH+75LiIXLhiXdktQVpoxF5q3q/he/+AVQlq5ddtllAPz3f/93KpPJyuqrr57KvF4OOuigQfuWEY4fQ04yOCXUM8A222yTttU3ve/J2MbzTMlMyCWGLh18+um6e7WPszLV8OcP9f+c9NVp1v/FaOzdW5k2DXd5y2hkj8MZd3q97U0NjOYaRUQrCIIgCIIgCIKgYkYc0Wp3UexYzxaNtdHGaGnX4tX/9rvf/Q4oW4c3w78rMwy3gW8Xj9i4CUYzfBFtO4zldanaCCU3W+oRBV+U7XbaQtfTZxeHG9EayzZe1f5lguEGIznOPvtsoGyV7fXbDK/v4R53p+p0JNFsRTsvvfTSVKZkwh5hkkXzkUcemcr6+voAePzxx1OZthdddNFUpkXz0IjYeD+Wvb7PjCshr89oK+q7xx57pDK3iR4rqurnipBcdNFFqUx15X1WzD333Gk7N9PvNtoaGz760Y+mMqkTWiktms2+j+V4qtQD0LBjl1U7NI7PIyGKZit1A8Ddd98NNNIRAPzoRz8CygYju+yyS1vH5ekKdDwvvfRSKmvXflx0w3PEJz/5ybQtYyo3Y/rDH/4ANOoSYIcddgDg+uuvT2UyZvE0Gd6vZe9+2mmnpTJFnL74xS+mMlntu3mTjiFHpxRRVZnhVPEs2y3PmKNlLNLlTOlERCsIgiAIgiAIgqBi4kUrCIIgCIIgCIKgYoYlHWx3oeFI8iSNRs7T6lt9bP4AACAASURBVBiqCOGORab4dtHCYV9sneMzn/kMUF7IKvlLjkmTJqVtXxwvWYwv6FZ9SEYEsNVWWw3apxbJ5r7bqp10KvQ+mv3mDEiEy2QkBfLcMV4HvhBe6Hq61EVUJX/oZtZYY420LeMGrydtzzTTTKnM61TSyxlmmCGVKQ9UVYx3nSrX2mOPPZbKZMTw1ltvpTLJf/7rv/4rlenv3oZ///vfA7DOOuukMl8MrzbtZg+SKn7uc59LZZIVS0IEjfbs5iPjIR1sV2bXimuvvRYo173GN5mPOJ57zOtFuHRQpkHKRQiNa9OKTt+HWsmxmt0bl1pqqVS2yiqrAOXcbpKsee42jX8ue5PRw3DyCqrOr7nmmlSmOt9kk01SmUvuegU3/pEZzjzzzJPKlMvS24bu797/JZt0Aya1aWhIi/0+dt111wHlZwTJaX3sVZ+48847U1mnn6Palbm1ktmOJJ9Vu1R9DxmP3IStGK6cuao2UNW+m31nJPUdEa0gCIIgCIIgCIKKGbEZhqjKRn24+xvO747mbXksZwvaRbPVbgqQQyYDfv4yD7jyyitTmSxzJ0yYkMr8O1/+8pcBSjNFWni///77Nz0G7dtnzN1eultod8ZGhg25z2266aZp+6qrrgLKC4v//Oc/p+1c9E+f9ZnwdqnayrrTeJtfe+21gYbxAsADDzwAwPHHH5/KtJh7lllmSWWK2kJjplwLvAGee+65Qb/d7izbWNZpu8e09NJLAw3rdGjM3HtEVVEpr799990XgA9/+MOpTCYhl19+eSrzSIv6vEdxFFl04wZFvGSKAXDccccBcMQRRww6j/Gi3ZlKfc7t1mUIIjMiyJtgCI8w5MwwTjnllLTdyh6+2bFOfsydZLj3Ko8+qw15agy1P4/45SJM6vPextdcc820rYjqdNNNl8qajaMetana8ns0tNr3scceC8Buu+2WynRPkmEIwOuvvw6UI4WLLbYYAMsvv3wqkzpm/vnnT2XepmWSI4WBly200EKpTJHYr3zlK6lMhj1zzjnnoPPLRY+qTu0y3L7utGNDP/nfh0urSHu3KFHajRK2azrUjefYaSKiFQRBEARBEARBUDHxohUEQRAEQRAEQVAxw5IO5mRurcKA3SRpajfnzngx3DrK5WLya3DGGWcA5YzxYuaZZ07bkgx6bh7P0XXFFVcA5cXJWjyrhbgA2267benzABtvvDFQlitIOtipsHHVeR/8OCWZ9Fwkkr45ksf86U9/SmVeB+L8889P23PNNRdQltCJdsPtVecGE+0a4Qz3b9CQt7gkUBIUl09JTnjrrbemMpcOqt4kmQV48803m/52O4xlnbb6LbURz/0jud7JJ5+cyiSZnHbaaVOZDFo8n5ukSIccckgq87HhW9/6FlCW/xx00EEAzDHHHKlMMi1JG6HR9yVjgob0q1P3g9H0/dwx+TnmkIzSzQKUG8qNWNwUQrjMUqYuZ555ZipzmVw7dKqdjgSNfy5TVQ6snGw6l8vKDSvUp13y5xLXL3zhC4P+LrOGiRMnpjL9tn+ul1h//fWBskRS92WX5yvHmOdhnG+++YByfrLdd98dKMtc3UBnmWWWAeDwww9PZUcddRTQMMWBvPRVY5WP6502Hsn1/3bNcHrpWXYsGYkZRrt00gRjNPuoWvoYEa0gCIIgCIIgCIKK6YgZRietMdtFC+ehYZMre96RsMEGG6RtZWKvgtHMwOYiWs7cc8895N/cDEP4bKov3tZs6z333JPKtEjcLaWbIVvqVlTRTlrNwIzGXlXXKhfFcjR767ODt99++6DPbbfddmlbx+OmGb1Cu3Wa+5yiLr54XZGEN954Y9A+crOnACeddBJQNonx6E0v0G5UxSNVikB5xFmz/W5/Lzz6cvTRRwNly+ecFbmXqe//7Gc/S2WKzuTSH/gxKKp72223DfpcFVQ9+6qF/463U1nce0RG9bPTTjulMrVN55///Gfa1vVyk4PhWuF3k2mL2oGbAb300ktA2URF7crrQng/lsLCo1ie+kFRU9nkQ1lZMfnvnX322YP+NhLVwFgj+3SPQEkR4HbrGvc8oq9r4nUo3OxCEWtoKC58vJG5jUdkFXFceeWVU9mPf/xjIG+y1am22ioVQbsmF+MVuRnJ/bPTKUdG84zabpQol+4mF+VuhbdTT1PSjHbvF6O5r0REKwiCIAiCIAiCoGLiRSsIgiAIgiAIgqBiRi0d7KQn/khCopKorbXWWqlMC7lHIh1ccsklgbJEbjwWQ+ZCqzlcFqDcIrnv5nJZ5eQU0JBg/Pa3v01lv/zlL4G85EOL4KGRRd7z0XSakWSHb7eNKe+I8mQBbLjhhoM+N2nSpKb7UZ3+9a9/TWXKZZKTbvUyrfqLpFge9pe8zU0a1H9dwurGGAcccABQzuVStQFOp2VDOanFggsumMpUV54rzPM6iYMPPhgoGyq41EdIhikJHMCpp56atiXz2meffVKZ5GC5fXt71ud8TNK55KSDVdxLqjbCWXzxxQeVuRGB2qnLWXML/m+++ea0rbxHLv1Wn/e6z8leq87lM1zalc9onPT7g+4f++23XypTHsaf/OQnqUzySTcTUVtySbtLC9X+ciYXn/70p9O2rpMbRPl9bbxplR9NUkA3rFlvvfUAePXVV1OZnoMk1wRYYIEFgHLOLOF9xuXa6q8uw9K22rHv05+7zjrrrEG/M5aGEmPRL5o9o3biHKfUvFPtygSPOeaYtC1DNmj0cX8W+853vtPWPldbbTUgP95Kqgvwq1/9CoBzzjmnrf06EdEKgiAIgiAIgiComFFHtMYCnxnQjNQtt9ySyjxik7MeXWSRRQD49re/ncq0eNZn0Z9++mmgbEcqO2/ZJXtZFbSyzBc5Y4tcpMpnnhRF8tkCzYT7Im9Zt/pCVufLX/4yUI4IbrrppgDsu+++qUzWuW60oYXwsnqF9jO2jyXtzj7NPvvsQNlKN4eig75ftyrXjKRbXwtvk1MDarM+w/+pT30KgJ/+9KepTNEFr9NVV101bd91111Aedbb67wKOr3wOIfP3GkWP2dy4QvkFSFxwwpF+jzKp3p2EwGPfGmM8TFEdeqmGlINuJmDZtM9oqVxp1dmZnMmPh5xzs2CXnjhhUC5nbr1vqzIc9FW35/X7+S0MpwaS9VF7rc0huUUDx51mnXWWYf1W96WvP7UTnNGDzlzHI9iffzjHwfK0djx6Of+u0P9tsrcZOS+++4DypFv3adcSTLTTDMBDet9aJg1rb766qnMx09/FhKKPp577rmpTPczN3z65Cc/Oei7k58HjL9derNr3O6xjfYcqmhvYxHtHm5deVm7hhe539DnbrrpplTmqXM0VvrzlCJaOcWHkxvDdR/zsUqpKlzlkFOG5YiIVhAEQRAEQRAEQcXEi1YQBEEQBEEQBEHFjIt0sFmIM/c3Dznmwnw77rhj2pakxuUvCvn5AnLlgvAyLXJ2iaEWhuZy94ylrGC22WYbVJbLo+XHKQOP73//+6lMC9iVYd5xOabLMnSenh3+hz/8IdDIi+NstNFGaVumES5NaEYnzVWqQJIY5WYD2GyzzQZ9TnJVl6552HuNNdYA8vKh7bffPm3ff//9QHfWRY6RSCgU9nfZz/HHHw+UZS677rorAP39/dn9SNLleY56LY9WDsnyoLEg180rJCOSAQE0xoYvfelLqUz17G1On3Mpkv9dciofTyXF8vFHeY5OOOGEVKZF+scdd1wq8xxKk1N13x/uAvWcnEn3Dmj0ZZe75O4Lufwt/h3Vm8s/VVa11HUs8fqTtDonf/Q2IDmbozp1sxXtx/fndaX6vfTSSwftb+mll07bd95556C/y7hovCVsQ7HMMsuk7Z133hkoG1epHXlblBTY5ZAq83rVmOlyWJcO5nI6akz1Nq2x22XOyq0lQzGABx54IHeKXYvL03Tv9+cpyTa9LvQ5z2vm0uxcbkg947Zqgzkp93iQkwT6MeXK2pUJXn755UAjxyM0nldV3wA/+MEP0rbuWd6+9BzqctaJEycCZemw2q4buWhJzIknnpjKlKfy+eefH3TMrYiIVhAEQRAEQRAEQcWMqxmGR6pkxOBvvXpL9VksvQFrthTKxheHHnooUJ510Gysf0779FlZzYT7zLpmenzG1+19x4rcTHAuorXuuuumbUVdtDDWueaaa9K2Zguuv/767G9rVsyjajfccEM7h53wmYuxskJt9VtVH4dHBHMz05rhA1h77bWB/AJRn/3qxkhWs+jzSOpUC+a9Pcsw4De/+U0q80XtQikMAK6++upBn3N743aOP3fc430NPKqnMdGjdppFlgkDNCzwPWKvcc3HMtmJe+TLo4Cy4vYZb10nr+eHHnoIgK233jqVKQWEj985o5fxiCS0uxjf60J1npu9Pu+881LZE0880fS3tWDb+74iB9/85jdT2ZlnnjnoWMebdvuE2ovMqRy3TFb/3GOPPQZ9LhcxyZnA+N+9/ak/uELA00UIGTu5PbrIRVnHSnWh3/HnDfXdPffcM5XJqMvPTUYVPv7pmrgxlVQW3kfdQtsNNoQiDn6/O+mkk4CywkX277Kkh0bEoRujh64I0Li30EILpbKcwYtMyvz5TEoiGY0ALLbYYoP27SkfctEeRR79ebSbIlkid0y5spyB2E477QSUo3977703UO7rqmc3bDv//PPTtp5NfUzVs4E/Ty277LIAbL755qlMShn/PZlwuQpJxluucmiXiGgFQRAEQRAEQRBUTLxoBUEQBEEQBEEQVMyIpYPNPPM9fNcs1Jn7my/EVjjbw/PKw+L5IXyBqxaEzjjjjKlMUg03c1D+Dl9AmjMmUDjRJS8yOvC8UiNl2mmnLWVlnxyd+4033tjW/lweqWP2HGCqU5cCCV846yiE7fuWfGCTTTZpejySGY2lVCCXm8xpV+7RrgzuiiuuAOC1115ruj+/DmpX4y0FaJdO1qlwSZvGEM9j4Yu4c2WSdsmYYTTH0s7fxwqvF/Vfl7SpLT3yyCOpTOYzOTMQl2hqnHQ5jP9d33dpUU7uqt/2cVKfcxmt5DmdGg9G0k6blbkZgGSWzllnnQUMLbvOITmX1/Mrr7wClPPADCWTG2ta3aNynH766UBe7uu5Z/R3byPq87n7sd+vvf5UVy6fFW6GkSNnaJJjvORuvsD/xz/+MdC450Ajf6WX6XP77LNPKlN9SdoGMGHCBKCcL26bbbZJ25J7qX1CQ8blY4buY56z6A9/+MOgcxmPOszlB83JPw844IBUpqUpboah+nUp5Cc+8YnSPgB+9KMfAeV6Pu2009K2npk8l6n2edFFF6WyXE64bqHVs5Hak8wnAH7xi18A5TxUkr27PFYyzJxk3scT77caP9y0SffN//3f/01lql+XcsoEw+9ruk/de++9qUx54W699VaGS3eM5EEQBEEQBEEQBFMQw4potTuzlZul97fFnMmFvpOzg9WiSmjMAjz77LOpzLOX52bC+/r6gPIiNv2OW/HqbdgXkOfORTMSJ5988qC/dQqP0OmY/Y1faCYR4Bvf+AZQtiIXOUvyoWb2VAf+91wkS7M73/ve97L7EV/84heBhlV1r6OZE7cezeHWw7o2PmutGaFes8CtCu+LWozqs34zzzzzoO+4KYSuQ24WvdvTBgg/zpzNtcZOH099ZlnIntaj1Io2eZRLM4keHchZ8vrsqn7bIw7PPPNM6Tf8cz5u6Pd8LNYxduN18fqTAYGfz9NPPw3AV7/61VQ2//zzA/nIH8CTTz4JlE0JFA1/7LHHUpkiB91Eu9eoWaR+gQUWSNuKbnnUVvfhnAmAPx/knhUcRTLOOeecVCZjKI/QuKGOGG4KmtHSbN9S0EBjPPRz1zl5f9S2PwfpO25qpW2PMsjGGhpt2Meg+eabDygbReie5WORjlsGBNAw6RkLVJetbMVvu+02AA4++OBUlksTkMOft4Rsxb2d+74VEfExUM8BHlV78cUXgXJERiZRMh+BxvOUjze6Xrvssktb59EMVwnoudGjhOqvPgZqjHQrdJlOfOUrX0llq6yyClC+/2jM9fuU2rOUaABzzDFH2tazsLd3GTx5mcx5Hn744VQmNYLb7qvOPbordcNSSy2VymQ604qIaAVBEARBEARBEFRMvGgFQRAEQRAEQRBUzKjzaLWSEuTyY+UWaIs555wzbW+33XYAnHHGGans0UcfBcqyOc8BIxmhh7A/97nPAWXffZHLNO9lkiZ5aNMzSHcahb9dNqVjcpmO6vSII45IZRtssAFQlmA0W/irfCIjJbdIsJXhSS/QbrZ2SYKGIpefLWfw4HK4qQk/b0kk3HAml+dG4wE0JAJep1rAqjxP0D35sVohWc8JJ5yQyjQOecb6nXfeGYAtt9wylamf+1irNueLtCXX9DrzBcVqsz4mHn/88YM+Jylxbmx3KafGA5d9DGXC0w34fUTnKwMMgDXWWANo5MWDhunNUGPt/fffD5SlRaoDv8+4lGU8aWUwkutP7Rp56B7mfV+Schm6QH6c9PrN3Wf0LOHPFJJjunRQdGNuJyhLB3/2s58BZUmWxraDDjoolanPHXfccalM9ynv/3qOWmGFFVLZsccem7Ylg/XlGcotpLyFAN/97neBhrkBNK6t3/c6Kb+cHOUHdUm6tr2tHn300UBZvrrmmmsC5TyO6v+eH0vPMpLAQWNsc1MmNypSX/d7l5bC+D1OErV55pknlcko4jvf+U4qO+WUU4DGPRMaS03cZGKkzxUzzzxzkgxKBprLE6o8U9DIXeWmE5JK+n1Dx+RjofK4ubGKzsdlgL6f3Fjry22E+opLH3fbbTeg/PyqXH8+ruge6PLudomIVhAEQRAEQRAEQcXEi1YQBEEQBEEQBEHFDFs6mHMMbMbtt98OlEPFksQoLA2NEKPnx5Jsw6Ul8rfXvwAzzTRT2lb41yUYcsnxUKLyFnhYW2H2T33qU6lMofr//u//TmVVSl1ysoxcWN2PXSFnl6vofHKukC7nyUkwxGWXXZa2XUao8lbSwlZuk8KdY8RYSgqaMRI5mdraCy+8MOzv5mSUIwnxj4ccrurf8vC/3Hxc9pPrd+7QKBe3r3/966nMc7+Jdo97vCWGckPz8UiSDR//5Pjn9SPJiJ+/6tdlXRobXIbhkmSN8z5Oagxx1yxJkFz6ovHA27h+22XcyyyzDN2K51GRNMYlRhqLlRvL8XuQj2u6rvvvv38qkyRG0iAo59SaEsjdC/72t78BeafQnEtwLs8m5KWKypnj/UcyPM+PJIbad6fwe3+z+56PP+rDvpRAzo3eRyWbc3e1nMOopHTu7urPFbpX+/WRXNYdBPVs4C58ktB+61vfanpOnbrna0xzOaOkoy69lFvwIYccMmgfGluh8TzqjoRbb701UHYG1BICd7PMud62csLVs7PjObyE8pXl/lbFMoQ33ngjnZ/GJK/TAw88ECg/W+o53iXiP/jBD4CGrBAaTq7/8z//k8qU985zx+o7Gi+gLC2Wo6H3C+VC23jjjVOZ6tnb6XnnnQfApEmTaAe5Sg6HiGgFQRAEQRAEQRBUzLAjWs1mejSr5NEfzaL4IjYtfPOZU83G+EJNzYD5DLWMMXIzrNCYofEFnXqrVq4SaMwI//rXv05ln/70p0vHDLDgggsC5TduzYp4PgD58w8Xz02mt+3cbE8rAwnNXHluC+GRpmaLlFtFrK688sq0nXurV54BJ3fcnsejE+TqtBXDjV74QmCZDPis4J133tnWfnL9yfvFcOnU7GCrHHqjifromH3BtYwAPN+bG1oIX9Sq8cLNcfbZZ58RH9fkxwedj27l9u9joiJavshY9eIRrZwphfpiLqqtyD2U+6dmhXPGA9/85jdT2e677w6UZ7kVkXBTB82me7RCVFHPVfd9zw2z1157AeU+q0XTiiAAXHHFFcDQebR0DX22XHXq47P23S7jrQZwcuqGXF4jzVDnxjy/x+fGSW/jHqWZHJ/V934zOTkzp06SG1NzbdCfW5S3zcfKJZdcEiibhilKrHsTNPq/17/K3DDCx4dcLrxTTz0VKI+tGqNyhhK5yEyn8Dp1M4XJydXzYYcd1vTvwvuZPxc2+1yrsajd5w832JgcfyapciwoiiJF86Ru8vvuTjvtBJTbiO4bHiVSH/Z7utqIt3Epy26++eamx+Xn+MQTTwBldZvGa3+uzdWv2uy6666bytTufVxSPlp/Dm6XiGgFQRAEQRAEQRBUTLxoBUEQBEEQBEEQVMyI82htuOGGQHmRpEL58tqHvHmGPudSFy2m9sVzCt95OFoSAZdYeE4t/Z5/R2FthTihsYAwh0tdtPjPpY/ad7N8YFXjYVnVpYdBladBhgBDfXck6Dq0mxtFi2ChLHEQfr0mZyyMB3Jh+nZ/97bbbgMaOcoArrrqKgCWW265VNaudNDbqcjlqGhF7vg7LSXKSVxHc/20eBUaC9hdZuBSNaGFzAAf//jHAfjiF7+YyrRgdiTH1axOx6KdSvbj44zGITfDuPvuu4GykYL6mMuENA54m5Pkz3/jt7/9bdqWBMT7viRdLtfS4mg3J1G+QR879fecdHAsGG7f9z6k8953331TmerKF2mLVvcHv9fJGMrHxtw+mzEWBgPt7j8nm5Ssx/920kknDbkPl51LujTUvazZkgY3FRnuvXAsZcNDsfzyy6dtSQf9GUvGDt6eJJtbccUVU5n6nEsyc3mMZB4Gjb7r+eQkX5t99tlTmZYuuBnOz3/+c6BhgtDtdLL/tLu/bpL/NkNSPWj0KUn1IG82p+dwtQsoSwaFpOu77LLLoL+5lM/vK5Ig+pirfuHPU8ozKwMPgMsvvxxoLB2Cxv3OlxFpCdOPf/zjQcfViohoBUEQBEEQBEEQVMywpndmmGGGZJX42c9+FoAHH3ww/V321kceeWQq0+yVv9kqg7XPkuo7++23XyrTojif5cstUvOZVS3U9Nkdz6ItNDPkxyD7Xl88e//99wPlyJ3ezHPZ5asgN6vhv6/ZO/+cZvZ9ca3qvFUkSou3ffZ7k002yW4LzeRus802qUyRhK222iqVaYHxddddl8p07UZiedoOrWxz253BzpmTXHvttUA5E7zq2WcPTzvttKb7ztkVi2YLu1sdd6dmXUdSp+3OzqlPe13IhtmzsOcsoE844YS0Lavd1VZbLZX5NRkp4zXLqGP3mTtt+/imSL2bZiga4m1J/fvEE09MZXvssQdQjix4ZEyzlT4rqG2fmdSs+worrJDKdG/w49dY7v3HZ0dHSy5dhjOaKKWO3c0C1CZ174DyWNcMt9TXtfbrOhpTnCrHgVZGODly45rOJ2eU4VFW/d3v+7m0LU6ztCVuJz3PPPM0O+y2GOvI1uabb562NVYecMABqUypX9yyXsoLN0/QufvzTW7sdZWA+r9HCiZMmACUUx8oCqHnQoDFF18cKEcPNF5X2eeDoelURDZnfPbUU0+lbY35N9xwQypTu/Hos6Ki3v/Vbvz+8rnPfQ6AY445ZlTH7Sk1JseNNISr7nLn3C4R0QqCIAiCIAiCIKiYeNEKgiAIgiAIgiComGFJB99+++0kBbvrrruAsgGAFj87OeMLhZw9FKew3W9+85tUpqzk+hcaeTBcVuELsH/5y18C5TCp8pG4SYMkLh6y1OJzNzLQ5ySLhEY4vlMLunOLMj10n5NlSDohGQE0cmO0ysElOahyJAyFFg0OtU/JCD2ELGmCy0VyIdpOU1XYXKFwz6+jazMcKan6Q27ReLumI07u/MZr8fbkv99Keqe24XUh6UErYxD/jmQubhShfjGauhhLkwFH/dclZjK78bHn+uuvB8qL5rUYOSc/c7MQyThceuXSDskRfTG8/u55dDSmX3TRRans85//PADnnHPOoP29+OKLg45rvGhXtqy+7zkE1T5dttkuPk7mjCJGMg50gpwcs90+7XzsYx8Dyjl4hEsCJT979tlnU5nG2Fb3MjfPUd6bddZZJ5XpXqe22YqxGkObSVofffTRVCbzgJNPPjmVyZjJ7/3qzy631pjgBgS5/G3+e3p2cuMLXRcfl/Ts5BJk5ffy/FS9IhlsV2I8XrLydtvlWD4DuMw/h/qjo2cmr8drrrkGaLQfKEsQR0Oz/dx4442V/EaO7hjJgyAIgiAIgiAIpiCGFdF6991306yIOPzww9O27Jl9MboiUD6LIitbZS+HxsJXn9HTbIvPdikK5gsxZa8NjZme3MJ5Z4sttgAatsTQyGLt0Tdte0RJs5iyVK+a3CyJzzL7TJLQAkKPEuo43dI2t5hYZhi56MpQ5GYWdV3vueeeVDb33HMD5RnO3MLlTpk5VL2/Z555ZlCZ6tevy4ILLggMPYMnA42c3XCrWdscnTbD8AXxw/2NVp/X33NtxBdu5/DvqI94nb700kvDOtZWjOUMofqR96ccihBsv/32qUwpKnJGQj6GSy3gNraanYZGXXqdamGyWzmvscYaQHnh/vHHH9/0uEU3Wxr79dbibI9oaTz1aEKrvi88CpZLeeFpRpoxlikH2iV3n9G91g0aNt10UwCWWmqpVKax0U1UZKDl5jc+Tmoc8DFY9TKSaGMzxtryXe0JGs81XoeyVveUI1IauYnJnHPOCZRVAhpfdZ8GWHvttdO26tOvpyLnM888cyq78sorgfLzh0x1fDwRrYywupF2VSNVnUeVBmHdRKu0MMM1MBstY9HuIqIVBEEQBEEQBEFQMfGiFQRBEARBEARBUDHDS5OewcNuCtFfffXVqUzbYxX6ayUZFP39/ZX+bhU0258vRs9J/LSY2vMGKTt8q7xMWiCszw+Fy7Ryx7DooosCeUmlfze3QL+XQ+KSsLixSiv5kMwAJLfM7W8kdCrs3io/URUo9xM0ZHCeS6MVuZwwnr9vuDTrj52SuI7k+t1+++1AWcojCY/LdF9++WWg3P/0HS16h/JYo7HD5T/+d6F9eg4u/Z7TqTwwNgAAElZJREFU7fKgZjz//PNAfizz+lFOrVbSQa/HnJzax5N2GEuJUav95+Riyv3kZguS+7qcTfkzXa4viWtuv5N/dnJayY+7HZf1XnDBBQDsvPPOqcyXbwgZArl0UJI/zyum+nTpphvaaCzNSYdnmWWWVCZZqOc5ktnLr3/96yZn12Aslg2M5fhTlWlGNz4btTvWtDrvsTi33DGM5HqM5lgjohUEQRAEQRAEQVAxw4pojcTmdTRvgVUv8h3JfrplobHPzGvm34/ptNNOA+DYY49NZVpM7EYawg1ENEPrs1Za3Or4bKL2qdl0gDnmmAOASy65JJVp9suPodni2Crq2Y0bRNWzW37eOh+P8mlBsdez02yWdTQZyDtFu2YYo4moeaRFqQZaRfe23XbbtK3oi39HfWUks/2dXqRdVZRQ1su56J0s4qFh4bzQQv/f3tkHW1VdB/y3CrYYEK2CTklNNOJU0UFRSIwYJGqg2hAyCeKkGqXaYtJAxkQ66mAr2hrimGA69StoLDGxqJAifpRAQnxCkQYQEZGPVKPyZfygwuTR4iiu/nH2vnfdd8/9eve8e+97b/1m3rz99tlnn33W2x9n77X22ifk4l599VWg0JV2dMNt77GOCebMmQPArbfemouL2p4xY8bk4hYsWFDrq9RNI9p+dB4E+bpmtVMTJ04ECt3apz3bOmmwMo/U4969q8etSu0pbcyJ7ds6UYnjjLWCiFpWe1xGHJts27bjVeyD0zSDVlMesY4jrAVOK2KPcohOK4YOHVqULh79AOmaqKipsq7ahw8fDhT2E9bR2PHHH1+QH+T7WSvrWH+tI7HYh1urjbQ6nWVdrdSnNuI7rjNHrbTKd2YaaX2qpZ6+tNp7u7MlBLhGy3Ecx3Ecx3EcJ3N8ouU4juM4juM4jpMxdTvDSKPRpn6tcFJ3Z6mk6o7vY82qolmGNRVKM5mIG9xfeeWVXFw5c5RS52jFeOtUI24+thvw4wnw9sy0jmUGWLp0ackyZEEjHDcsXrw4Fx4/fjxQ+P+I57TNmjUr9f5o0mE31kfSnA1UMslrtplbuedWMjOK162JTzQFKrfJHQrPGop1zDoR2LlzZ01lbqTpRj1nk1miExp7ts4DDzwA5E13bbq4SR3yJqz2HCgrv9j2bVwM23oaw9F8uBQxv7R6nwW2ntazCbqcKY91GhTPD7N1t5JToYh1SpDm+ML2ra1GpfoanVJFxyAWa3Z99dVXA4VOVOJZWXZMu/TSSwG4//77c3HWSUQ0r0s7b3L9+vVFZTj22GPLlr+VsKZ80enV6tWri9Jde+21uXAci1auXJmLi+ebWtP0eCacdVhh2/BZZ50FFDq22bx5M1BoEhjl/8wzz+TiXn/9daDQcccFF1wAwJ49e4rKH9vcli1biq71dFrRZLC700rns7lGy3Ecx3Ecx3EcJ2M6rdGqZwW72nyrPRm7EW7W08rQiI3GkWeffTYXjqtH1pV93EycppWqVj6lNIP1uJ6eNm0akF9dBBgxYgQAjz76aFXlqpVqNQXVnlBuNxRHDcuSJUtycXHVthZ3zJs2bQLg1FNPLbqWtjG+WrpKu1tJptWuHpXTFFiZxg3blY4msJuvo4bA1rW4qppWhmpphMa82nxjXbMr97NnzwYK69/06dOBQrfZ27ZtAwpXtIcNG1b0DKtJiM+xzluiMwO7gT4+e+bMmbm4KDdbBqvN6EgWss2qnqaR1u9Ht9bRnTbkNQLR0QDAxo0bi/IZNWpULs46G4lYbW015bI0sp6mPT9qLIcMGVJ0beHChWWfEZ06WOcObW1tQKETCKsVSdOQRNLq3HnnnZcLR0dSaTTKZX65/9f27dtz4agR2rt3b9G9p512Wi5u9+7dQKElTNTi2XoVNYT2eIfohh/yrvjt9ahJtFrBaCljtdxRS2n76EY6emq2xUIttLIzjK4ki/640v1Z9Y/1/I9co+U4juM4juM4jpMxPtFyHMdxHMdxHMfJmC5xhmEpp26r1syoEp0x7SlnKtaKrFmzJheeMGECUKimr2RiVQ2l3r8eM9FoUmTPzipn5pEF1TrDqFT2GDdy5MiS1yBvnhY3xlvOPvvsXNiaf8ZzSdJMhgYNGpRaxp5E2ntZU9hyZ8CVIprJxA3eAPPnzweqb+eNMhXqSLXPSmvnN9xwAwDXXXddLu7CCy8E4JFHHsnFjRs3Dih0MBLNLOP5PJDfcA958zYbt3btWqDQmUN0gGPP8bMmg40izRlG1iYmNl0064p9MuTlG+UNhaaDEXs+Ulo/YB1ElCtPWrmagTWfjKZmjz32WC4utsVmEs92i+fHdYZGy9k6w4gmwdYsNTpfefDBB3Nx0YmFNc+/6aabgELzvXimnv2WsOaGMXzHHXfk4qI5qM0nhm25Yn2wTl3SzDhb2WyuK8aDRozprSjLclTaxpFGs8bqzuAaLcdxHMdxHMdxnIyRWtzsikjX+OTtGTynqsWqjwpYmZZb6YgOJCC/Id5qAOIG4xUrVuTiojvXWrQCtWLdxVutWyRujrernXH1e/ny5UXpO6xM1C3Teqh2M+1dd90FFLqljavVy5YtS807bmq2mr729nagUONgNQS1lqsELSfTiNXkRS2idZAR3TSXyiM6Fpk6dWoubvTo0RWfC3WviGUm02qd0FRdMPNeN954I1CoKYlahlLHPlx11VVAYT2M2rJVq1bl4uyG92rL05EO796l9TTr1dDJkyfnwtG1tpVz1ALa50VHGgBjx44F4OKLL87FXXLJJUCh05FIJ/uALhmj4nNtHYra1eioxZKWh3Xi1FWu/yHvTMNqcBYtWlQyfRXOuTolU4D+/ftrOcctaXKKx4ZY6wnrgKZjWS3RecWOHTuKnrFr165cnNXExu8JeyREGjGfqCGD/PdA7C8A7rnnnpL3RrZs2cL+/fvTz5qpgH+jlqWpY38PpSqZukbLcRzHcRzHcRwnY3yi5TiO4ziO4ziOkzG1mg6+DRQfTOMAfFxVB9d6k8u0LC7T7HGZZo/LNHtcptnjMs2eTskUXK5lcJl2Dd7+s6cqmdY00XIcx3Ecx3Ecx3Eq46aDjuM4juM4juM4GeMTLcdxHMdxHMdxnIzxiZbjOI7jOI7jOE7GNHaiJTITkZcQ2YjIBkQ+lWHeYxF5MrP8ehJpchd5DZFBKWm/gMj1JfIZi8jZXV3clkXkYJDfS4i8gMi1iPhiRT24TCsjclSQ0QZEfofILvP3H1a4t3S/KHI/IsNKXLsGkY90iLsekUsR+WLJ+7oLLtPWw8ep2sj3nZsQWVBUt4rTz0NkUgi3IdKp878crOxfQGR9r6hvWdEV84Bq6nMT63zfhj1J5NPA54EzUH0vdJ7lB7RGIdIX1Q8qJ+yG1Cp31ceBx1Py6QuMBdqBZ7uiqN2A/0P1dABEjgb+DRgI3FSQqifXp+xxmVZCdQ8QZTQLaEf1exnk+9ep8SJ9gGuAnwL2tNzxwGTgduBJYHPdZWgWLtPWwsepzmD7zoeArwFzmlqiiEgfVA82uxhdiJX9eGA2cG5TS9QdaOV5QBfSyJXjPwHeQfU9AFTfQXV3WLG6OawKvIjISQCI9EfkAUTWIPI8IhND/HGIrAzp01cSREaFe04ok88URB5H5FfA8kYIoEmkyz1heorcpyByZwjPQ+ReRH4NPErSkX8rrEJ8ptEv0lKovgVMBaYhIqn1SeTvEFkbVm5uDnH9EXkqrIRtQuSSEP9dRDaHtPV/8HVHXKb1IXIuea3M84gcFq4MQGQhIlsReQgRCenzK3wi7Yh8H5EXgJnAEOBpRJ4O1weSDIgnAl8Abg/POQGR0xH5ryDnRYj8scn/n8mvun+yccLICJdpo/Bxqj5WAkPD99GmXKzIDJKFhNKIfCXIdhMit4W4ryFyu0lj5X0ZyffUBkR+SLKI0LG+fzrLl2txBgLvAiAyAJHlpr5OzKUS+XtEtiHyn4jMR2RGk8rbTErNA/4hjOubEJnboT+9LdS33+Tas8ihiDyMyBZEFgGH5p4gcg8i60i0Zjc3+gVTUdXG/MAAhQ0Kv1G4W+HcEP+awvQQ/luF+0P4OwqXhfAR4b7+Ch9R6BfiT1RYF8JjFZ5UOFvhOYWPVchnisJOhSMbJoNm/NQu9ykKd4bwvCDTPuHvWQozmv5OzZNle0rcXoVjiuoTjFOYqyAKfxDkOEbhywr3mfsPVzhKYZuG4xYUjmj6u7pMW/OnXBuEJxRGh/AAhb6hX9yn8KdBZqsVzglp2hRGhrAqTDZ5vaYwyPz9JYVbQniewiRzbaPpV25R+IHJ/74QHqOwqenyc5m25o+PU52RWXv43VdhscLXFY4rqBMwQ2FWUR2L9RSGKGxXGBzy+ZXCF8PfL5t8liico3ByaBOHhPi7FS4PYS2o7z35Bw6G+ro19AVnmv/FwBAepPByGK9GhfT9FA5T+O9eUUeL5VaqnR9p0vxEYYKpp98P4YsUfhnC31Z4IISHK3xg+t34vdAn3D/c5DWyGe/dOI2WajtwJsmK9dvAI4hMCVf/Pfx+DjguhMcB1yOyAWgD+gEfAw4B7kPkRWABYO3aTwbmAhNQ3V4hH4BfoPo/mb1jK1K73DuygJ5tApAltj6NCz/PA+uBk0hWrl8EPhdWaT6D6j5gH3AA+BEiX6LQtKi34zKtnlXAHES+CRxB3tRyDao7Uf0Q2EB6Wz8I/KxM3n8OLCmKFTk8POuZEPNjYIxJMR8A1RXAQESOqPptWgOXaSPwcaozHBq+a9YB24EfdSKPUUAbqm+Huv0QMAbVt4HfInIWIkeR9LWrgPNJ/k9rw7PPBz4R8qpU33sSiemg6kkk7fjBoIUR4DuIbAR+CXwUOAYYDSxG9QCqvweeaFbBm0rpdv5ZRH4dvuvPA04xd6W1/zEkZtiguhHYaNJPRmQ9yXfCKRTOEZpC4/ZoAaEjbAPagkCvCFfeC78PmjIJ8GVUtxXkkajB3wROIzF9PGCuvkEykRoBRLODUvl8Cthf5xt1D2qTe0d6h4w6g8gnSGT3VoixshJgNqo/TLnvDOAi4J8QWY7qLcEE6HxgEjCNpLPpfbhMq0fkG8DfhL8uQvW7iDxFIodVJHsHIN/OoXRbP1DhQ/WTwNc7UUqt8Hdr4TJtHj5O1Up+n1BE5AMKt4T0qyP/h0n2D24FFqGqYTLxY1RvSElfqb73TFRXk+w1GkzSTwwGzkT1fUReo77/Qc+juJ1fDQwHRqK6I3zjW5lV0/4TRI4HZgCjUH0XkXm0gPwbp9ES+TNETjQxpwOvl7ljKYltdrTVHBHiDwfeCCuJXwX6mHv2An8BzEZkbIV8ege1y70cvwcOq5iqNyAyGLgXuBPVtA+dpcCViAwI6T+KyNGIDAH+F9WfkmyAPyOkORzV/wC+RbKI0PtwmdaG6l1hVfV0Ejv3E1B9EdXbgLUkq9CdJd/WRU4BtpqPqPy1RHv4rtkL81XgGZNP3C93DrAvpG9dXKbNwceprHgTOJrEq+YfkTgeKMca4FxEBoW9Vl8hX9cWARND3MMhbjkwicRxEYgcicjHM36H7kWyb7APsIfk+/StMMn6LBBlswqYgEi/MDZV+r/0TNLbeVSCvBNkM6mKnFYAfxnyPJVkogbJfrn9wD5EjgEuzKLY9dJIjdYA4F+CmcMHwMsk6sNSFe4fgR8AG0ncPb8a0t4N/AyRy4Gf03ElS/VNRD4PLEHkyjL59BZqlXs5ngAWkmzwnI7qyuyK2S2IphqHkMjyJ5Ty8qS6DJGTgdUkc/x24DJgKMmm9w+B90lWtA8DFiPSj0Rr8+0ufo9WwmWaHdeEwf1D4CUSs7TObkqfC/wckd3AUyR9beRhEvPtb5IMilcA95K4l/4t8Fcm7QFEnif5/17ZybI0E5dpY/BxKguSD/xbSCZQu0i0UeXSv0HiJv9pkn7yKVQXh2vvIrIFGIbqmhC3GZEbgWXhe+p94Bt0flLcXYnjFiRyuwLVgyTeH58Impp1RPmrrkXkcRITtzdJzN27/wJJ7ZRq53uBTcDvSBa0KnEP8K+hfm4hMSsE1RdC37gV2EEywW06kr5w7DiO4ziAyC+Ay1F9o8b72oAZqK7rimJ1a1ymjtO7EBmAantYPFkBTEV1fbOL5XQ9jd2j5TiO43QvVD/X7CL0OFymjtPbmEtyKHk/kn1uPsnqJbhGy3Ecx3Ecx3EcJ2MaeWCx4ziO4ziO4zhOr8AnWo7jOI7jOI7jOBnjEy3HcRzHcRzHcZyM8YmW4ziO4ziO4zhOxvhEy3Ecx3Ecx3EcJ2P+H26doTxvk1oAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x216 with 20 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# x_adv_samples = np.load(\"adv_samples_from_cos.npz\")\n",
    "# x_original    = x_adv_samples[\"x_original\"]\n",
    "# x_adversarial = x_adv_samples[\"x_adversarial\"]\n",
    "# y             = x_adv_samples[\"y\"]\n",
    "\n",
    "x_adversarial = x_adv_samples\n",
    "\n",
    "x_orig = ((x_original   ) * 255).astype('int')[:, :, :, 0]\n",
    "x_adv  = ((x_adversarial) * 255).astype('int')[:, :, :, 0]\n",
    "\n",
    "y_pred_orig = model.predict(x_original,    verbose=0)\n",
    "y_pred_adv  = model.predict(x_adversarial, verbose=0)\n",
    "\n",
    "fig    = plt.figure(figsize=(15, 3))\n",
    "cols   = 10\n",
    "rows   = 2\n",
    "images = list(x_orig[:cols])      + list(x_adv[:cols])\n",
    "preds  = list(y_pred_orig[:cols]) + list(y_pred_adv[:cols])\n",
    "labels = list(y[:cols])           + list(y[:cols])\n",
    "\n",
    "for i in range(0, len(images)):\n",
    "    ax = fig.add_subplot(rows, cols, i+1)\n",
    "    y_pred = np.argmax(preds[i])\n",
    "    y_orig = np.argmax(labels[i])\n",
    "    ax.set_xlabel(get_label(y_pred),\n",
    "                  color = \"green\" if y_pred == y_orig else \"red\")\n",
    "    ax.tick_params(axis='both', which='both',\n",
    "                   bottom=False, top=False,\n",
    "                   right=False, left=False,\n",
    "                   labelbottom=False, labelleft=False)\n",
    "    plt.imshow(images[i], cmap='gray')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Summary and Next Steps <a id=\"summary\"></a>\n",
    "\n",
    "This notebook only looked at one adversarial robustness technique (FGM). The *ART* library contains many more attacks, metrics and defenses to help you understand and improve your model's robustness. You can use this notebook as a template to experiment with all aspects of *ART*. Find more state-of-the-art methods for attacking and defending classifiers here:\n",
    "\n",
    "https://github.com/Trusted-AI/adversarial-robustness-toolbox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acknowledgements\n",
    "\n",
    "Special thanks to [Anupama-Murthi](https://github.ibm.com/Anupama-Murthi) and [Vijay Arya](https://github.ibm.com/vijay-arya) who created the original notebook which we modified here to showcase how to use *ART* with *FfDL*. If you would like to try *[Watson Machine Learning (WML) Service](https://console.bluemix.net/catalog/services/machine-learning)* with *ART* check out Anupama and Vijay's notebook here:\n",
    "\n",
    "[https://github.ibm.com/robust-dlaas/ART-in-WML/Use ART to check robustness of deep learning models.ipynb](https://github.ibm.com/robust-dlaas/ART-in-WML/blob/master/Use%20ART%20to%20check%20robustness%20of%20deep%20learning%20models.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2017, 2018 Trusted-AI. This notebook and its source code are released under the terms of the MIT License."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}