chp10_EarlyStopping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2a858fde-dcd9-430d-83c2-25b9f30f8dda",
   "metadata": {},
   "source": [
    "Early stopping is a technique used to prevent overfitting when training machine learning models, including XGBoost models. Overfitting occurs when a model is trained for too many iterations and starts to memorize the training data instead of learning to generalize from it. \n",
    "\n",
    "Early stopping works by monitoring the model’s performance on a validation dataset during training. If the performance on the validation dataset stops improving for a certain number of iterations, the training is stopped early and the best iteration is used as the final model.\n",
    "\n",
    "In XGBoost, you can use early stopping by specifying a validation dataset and setting the early_stopping_rounds parameter when calling the fit method. For example, if you set early_stopping_rounds=10, training will stop if the performance on the validation dataset doesn’t improve for 10 consecutive rounds.\n",
    "\n",
    "Early stopping can help you save time and computational resources by stopping the training process early if it’s not likely to result in further improvements. It can also help you avoid overfitting by preventing the model from being trained for too many iterations.\n",
    "\n",
    "we can achieve early stopping in XGBoost models when we call the `.fit` method.\n",
    "\n",
    "## 10.1 Early Stopping Rounds\n",
    "\n",
    "Here are the results of XGBoost using out-of-the-box behavior. By default, the model will\n",
    "create 100 trees. This model labels about 74% of the test examples correctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "79ef1480-1e47-455c-9bcc-8dfc3308f58f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import dtreeviz\n",
    "from feature_engine import encoding, imputation\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import base, compose, datasets, ensemble, metrics, model_selection, pipeline, preprocessing, tree\n",
    "import scikitplot\n",
    "import xgboost as xgb\n",
    "import yellowbrick.model_selection as ms\n",
    "from yellowbrick import classifier\n",
    "import urllib\n",
    "import zipfile\n",
    "\n",
    "from EffectiveXGBoost import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "14126273-090c-410b-a30b-75095add1eed",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "G:\\My Drive\\SisengAI\\AlgorithmicTrading\\code_rebuilding\\MattHarrison\\EffectiveXGBoost\\EffectiveXGBoost_MyTake\\EffectiveXGBoost.py:56: DtypeWarning: Columns (0,2,8,10,21,23,24,25,26,27,28,44,56,64,83,85,87,107,109,123,125,150,157,172,174,194,210,218,219,223,246,249,262,264,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,304,306,325,326,329,341,368,371,384,385,389,390,391,393,394) have mixed types. Specify dtype option on import or set low_memory=False.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7458563535911602"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path_url = 'https://github.com/mattharrison/datasets/raw/master/data/kaggle-survey-2018.zip'\n",
    "file_name = 'kaggle-survey-2018.zip'\n",
    "dataset = 'multipleChoiceResponses.csv'\n",
    "\n",
    "raw = extract_dataset(path_url, file_name, dataset)\n",
    "\n",
    "# Create raw X and raw y\n",
    "kag_X, kag_y = prepX_y(raw, 'Q6')\n",
    "\n",
    "# Split data\n",
    "kag_X_train, kag_X_test, kag_y_train, kag_y_test = (model_selection\n",
    "                                                    .train_test_split(kag_X, \n",
    "                                                                      kag_y, \n",
    "                                                                      test_size=.3, \n",
    "                                                                      random_state=42, \n",
    "                                                                      stratify=kag_y)\n",
    "                                                   )\n",
    "\n",
    "\n",
    "# Transform X with pipeline\n",
    "pline = pipeline.Pipeline(\n",
    "    [('tweak', PrepDataTransformer()),\n",
    "    ('cat', encoding.OneHotEncoder(top_categories=5, drop_last=True,\n",
    "    variables=['Q1', 'Q3', 'major'])),\n",
    "    ('num_impute', imputation.MeanMedianImputer(imputation_method='median',\n",
    "                variables=['education', 'years_exp']))]\n",
    "    )\n",
    "\n",
    "X_train = pline.fit_transform(kag_X_train)\n",
    "X_test = pline.transform(kag_X_test)\n",
    "\n",
    "# Transform y with label encoder\n",
    "label_encoder = preprocessing.LabelEncoder()\n",
    "label_encoder.fit(kag_y_train)\n",
    "y_train = label_encoder.transform(kag_y_train)\n",
    "y_test = label_encoder.transform(kag_y_test)\n",
    "\n",
    "# Combined Data for cross validation/etc\n",
    "X = pd.concat([X_train, X_test], axis='index')\n",
    "y = pd.Series([*y_train, *y_test], index=X.index)\n",
    "\n",
    "# Default training\n",
    "xg = xgb.XGBClassifier()\n",
    "xg.fit(X_train, y_train)\n",
    "xg.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c8d03b1-7312-4318-a72b-922fa720aedd",
   "metadata": {},
   "source": [
    "Let us provide the `'early_stopping_rounds parameter=20'` when n instance of the `XGBClassifier` class is created.\n",
    "\n",
    "Next, the `.fit` method is called on the `xg` classifier object to train the model on the training data` X_train` and `y_train`. The `'eval_set'` parameter is set to a list containing two tuples: one with the training data and one with the test data. This tells XGBoost to evaluate the model’s performance on both the training and test datasets during training.\n",
    "\n",
    "During training, XGBoost will monitor the model’s performance on the test dataset and stop training early if the performance doesn’t improve for 20 consecutive rounds. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1171c038-2417-46cc-b70c-413caaabd84b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tvalidation_0-logloss:0.61534\tvalidation_1-logloss:0.61775\n",
      "[1]\tvalidation_0-logloss:0.57046\tvalidation_1-logloss:0.57623\n",
      "[2]\tvalidation_0-logloss:0.54011\tvalidation_1-logloss:0.55333\n",
      "[3]\tvalidation_0-logloss:0.51965\tvalidation_1-logloss:0.53711\n",
      "[4]\tvalidation_0-logloss:0.50419\tvalidation_1-logloss:0.52511\n",
      "[5]\tvalidation_0-logloss:0.49176\tvalidation_1-logloss:0.51741\n",
      "[6]\tvalidation_0-logloss:0.48159\tvalidation_1-logloss:0.51277\n",
      "[7]\tvalidation_0-logloss:0.47221\tvalidation_1-logloss:0.51040\n",
      "[8]\tvalidation_0-logloss:0.46221\tvalidation_1-logloss:0.50713\n",
      "[9]\tvalidation_0-logloss:0.45700\tvalidation_1-logloss:0.50583\n",
      "[10]\tvalidation_0-logloss:0.45062\tvalidation_1-logloss:0.50430\n",
      "[11]\tvalidation_0-logloss:0.44533\tvalidation_1-logloss:0.50338\n",
      "[12]\tvalidation_0-logloss:0.43736\tvalidation_1-logloss:0.50033\n",
      "[13]\tvalidation_0-logloss:0.43399\tvalidation_1-logloss:0.50034\n",
      "[14]\tvalidation_0-logloss:0.43004\tvalidation_1-logloss:0.50192\n",
      "[15]\tvalidation_0-logloss:0.42550\tvalidation_1-logloss:0.50268\n",
      "[16]\tvalidation_0-logloss:0.42169\tvalidation_1-logloss:0.50196\n",
      "[17]\tvalidation_0-logloss:0.41854\tvalidation_1-logloss:0.50223\n",
      "[18]\tvalidation_0-logloss:0.41485\tvalidation_1-logloss:0.50360\n",
      "[19]\tvalidation_0-logloss:0.41228\tvalidation_1-logloss:0.50527\n",
      "[20]\tvalidation_0-logloss:0.40872\tvalidation_1-logloss:0.50839\n",
      "[21]\tvalidation_0-logloss:0.40490\tvalidation_1-logloss:0.50623\n",
      "[22]\tvalidation_0-logloss:0.40280\tvalidation_1-logloss:0.50806\n",
      "[23]\tvalidation_0-logloss:0.39942\tvalidation_1-logloss:0.51007\n",
      "[24]\tvalidation_0-logloss:0.39807\tvalidation_1-logloss:0.50987\n",
      "[25]\tvalidation_0-logloss:0.39473\tvalidation_1-logloss:0.51189\n",
      "[26]\tvalidation_0-logloss:0.39389\tvalidation_1-logloss:0.51170\n",
      "[27]\tvalidation_0-logloss:0.39040\tvalidation_1-logloss:0.51218\n",
      "[28]\tvalidation_0-logloss:0.38837\tvalidation_1-logloss:0.51135\n",
      "[29]\tvalidation_0-logloss:0.38569\tvalidation_1-logloss:0.51202\n",
      "[30]\tvalidation_0-logloss:0.37945\tvalidation_1-logloss:0.51352\n",
      "[31]\tvalidation_0-logloss:0.37840\tvalidation_1-logloss:0.51545\n",
      "[32]\tvalidation_0-logloss:0.37538\tvalidation_1-logloss:0.51581\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7558011049723757"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xg = xgb.XGBClassifier(early_stopping_rounds=20)\n",
    "xg.fit(X_train, y_train, \n",
    "       eval_set=[(X_train, y_train), (X_test, y_test)])\n",
    "\n",
    "xg.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b32aed91-88d9-40ba-865e-0c6a183d5307",
   "metadata": {},
   "source": [
    "We can ask the model what the limit was by inspecting the `.best_ntree_limit` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8abfcde0-cc63-4652-a5b5-10df3eedd273",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xg.best_ntree_limit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4998d13-34dc-41a1-98c0-b07bb9a65b7e",
   "metadata": {},
   "source": [
    "## 10.2 Plotting Tree Performance\n",
    "We will explore the score that occurred during fitting the model. The `.eval_results` method will\n",
    "return a data structure containing the results from the `eval_set`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ead6fcd0-70aa-432c-b43c-446b7ea56396",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'validation_0': OrderedDict([('logloss',\n",
       "               [0.6153406503923696,\n",
       "                0.570456665259967,\n",
       "                0.5401074955389963,\n",
       "                0.5196461799091072,\n",
       "                0.5041859194777588,\n",
       "                0.4917588337620288,\n",
       "                0.4815858466047529,\n",
       "                0.4722135674049504,\n",
       "                0.4622124607211323,\n",
       "                0.4570046101736514,\n",
       "                0.4506211909637632,\n",
       "                0.4453310159057096,\n",
       "                0.4373589512455096,\n",
       "                0.4339914069427133,\n",
       "                0.43004427399415684,\n",
       "                0.42550266016301175,\n",
       "                0.42168949386987714,\n",
       "                0.418539318900061,\n",
       "                0.4148519256973189,\n",
       "                0.4122836277372164,\n",
       "                0.4087179540923545,\n",
       "                0.40489826790427885,\n",
       "                0.40279635313690876,\n",
       "                0.39941699934982083,\n",
       "                0.3980718076986071,\n",
       "                0.3947315311104598,\n",
       "                0.39388538918257093,\n",
       "                0.39039599370074535,\n",
       "                0.38837148117914494,\n",
       "                0.38569152608129825,\n",
       "                0.3794510695498472,\n",
       "                0.37840359450022193,\n",
       "                0.37538466163330503])]),\n",
       " 'validation_1': OrderedDict([('logloss',\n",
       "               [0.6177459120091813,\n",
       "                0.5762297137336836,\n",
       "                0.5533292924830927,\n",
       "                0.5371078263330196,\n",
       "                0.5251118483629016,\n",
       "                0.5174100387985535,\n",
       "                0.5127666982333304,\n",
       "                0.5103968677599786,\n",
       "                0.5071349117019888,\n",
       "                0.5058257413338562,\n",
       "                0.5043005665239378,\n",
       "                0.5033770955852053,\n",
       "                0.5003349147737026,\n",
       "                0.500343639228637,\n",
       "                0.5019165393520785,\n",
       "                0.5026775172031717,\n",
       "                0.5019612925713728,\n",
       "                0.5022262001183728,\n",
       "                0.5035970172726483,\n",
       "                0.5052709661197761,\n",
       "                0.5083886559280819,\n",
       "                0.5062287504265075,\n",
       "                0.5080608451852153,\n",
       "                0.5100736727207406,\n",
       "                0.509867396365172,\n",
       "                0.5118910036538599,\n",
       "                0.5117007328783939,\n",
       "                0.512182518838807,\n",
       "                0.5113475983128857,\n",
       "                0.5120185821651589,\n",
       "                0.5135189278200475,\n",
       "                0.5154504013839513,\n",
       "                0.5158137125354157])])}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# validation_0 is for training data\n",
    "# validation_1 is for testing data\n",
    "results = xg.evals_result()\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a05d1519-a2fa-45e8-9efa-365c0ec38ebf",
   "metadata": {},
   "source": [
    "We will plot that data and visualize what happens as we add more trees:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "88e90c50-2933-4822-9d5e-b18e9bfce7d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUAAAAETCAYAAABHit/zAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABOXElEQVR4nO3dd1yVdf/H8dc5h8M8IDJEFEFAcYALJypucyuiOHOUWlpWNlzlSk1t3XdZ/izLu2GWJjlzkzPcuEBFHMgQZAscNudcvz/QkwSKFnAO8H0+Hjw451zrc13Am2t8r+8lkyRJQhAEoQaS67sAQRAEfREBKAhCjSUCUBCEGksEoCAINZYIQEEQaiwRgIIg1FgiAKuYXr16ERoaqrflx8bG0qZNGwBiYmJ47bXXynX+CQkJjBkzpsT8H11uZZgwYQL79u0r13lWxPZ6KDQ0lNdff71C5l2diQAU/rG4uDgiIyPLdZ4ODg5s2rSpwuavTxW5Pi1atGD16tUVMu/qTARgJTh06BABAQH4+fkxZswYLly4gEajoXv37oSFhenGmzVrFj///DPJycm88sorjB49ml69ejFhwgRSUlKeennz5s1j+vTpDBo0iI8//pj8/HxWrFjB8OHDGTp0KPPmzUOtVgPw888/M3ToUEaMGMG4ceO4efMmUHJP8+/vNRoNCxYsIDo6milTplBYWMjixYsZMmQI/v7+vP7662RlZRWrKygoiHHjxune9+vXT/dHe+/ePbp27UpMTAxt2rQpMf+Hy1y0aBHDhw+nT58+7N+/v9T1P3/+POPGjWP48OGMGDGCw4cPA5Cdnc2cOXMYPXo0/fr1w9/fn9u3bwNFe3wzZ85k4MCBbNiwQTevtWvX8vbbb+venzt3Dj8/vydu/1u3bjFmzBj8/f0ZPnw4GzduLHV9goKC8PPzY+jQoYwdO5bLly8D8MUXX/DOO+/w/PPP069fP9544w3dz6tXr158+umn+Pv707dvX37++WcATp8+zeDBg3U//+XLlzNhwgT69u3LzJkzdT+Lo0ePMmTIEIYNG8a8efPo1q0bsbGxT1yfak0SKlRkZKQ0ePBgKTU1VZIkSYqIiJC6dOkiZWVlSZ9//rn0/vvvS5IkSffv35c6dOggZWRkSN9//7309ddfS5IkSVqtVpo6daq0fv16SZIkqWfPntLly5efuMy5c+dKkyZN0r3/4osvpFWrVklarVaSJEn69NNPpcWLF0uFhYWSp6enlJCQIEmSJG3btk3atGlTqct5+D4mJkZq3bq1JEmSdOrUKWnQoEGSJEnS2bNnpf79++uW8dFHH0khISHF6srJyZG8vb2l9PR0KSYmRurSpYs0evRoSZIk6aeffpIWL1782PnHxMRIHh4e0r59+yRJkqQDBw5IvXv3LrHu9+/fl5577jkpJiZGkiRJunfvntStWzfp7t270t69e6Vly5bpxl24cKG0dOlSSZIk6fnnn5fmz5+vG/b8889Le/fulZKTkyVvb28pLS1NkiRJmj17tvTLL788cfvPnz9f9/NLTEyUZs2aJWk0mmLrc/PmTalz585SdHS0JEmSdOLECalLly5SZmamtHr1aqlbt25SUlKSpNFopLfeektatWqV7uewcOFCSavVSvHx8VLHjh2l8PDwYvOeO3euNHr0aCkvL0/Kz8+X/Pz8pMDAQCk1NVXq0KGDdO3aNUmSJGnr1q2Sh4eHblvVREb6DuDqLjg4mMTERCZPnqz7TCaTER0dzYgRIxg5ciTz5s3j999/p1evXlhaWjJp0iTOnTvHd999x507d7hx4watWrV6puW2bdtW9/rIkSNkZmZy4sQJAAoKCrC1tUWhUNC/f3/GjBlDjx496Nq1K927d/9H6+nh4YFCoSAgIICuXbvSr18/WrZsWWwcU1NTOnfuTHBwMGlpaYwePZrNmzeTmZnJoUOHmDp16hOXoVQq6devHwBNmzYtda/44sWLJCUl8eqrr+o+k8lkXL9+nf79+9OgQQM2bNhAVFQUZ86cKXZesV27diXmZ2trS48ePdixYwd+fn78+eefLF68+Il19u3bl7lz53L58mV8fHxYsGABcnnxg61Tp07RqVMnGjRoAICPjw82Nja6I4L+/ftjZ2cHwMiRI1mxYgVz584FYNy4cchkMurWrYuvry/BwcF4enoWm7+vry/GxsZA0c8mPT2dc+fO4e7uTtOmTQEYPnw4y5cvf+K6VHciACuYVqvFx8eHzz77TPdZfHw8derUQaFQ0Lx5c44cOcLWrVt59913Afj444+5fPkyI0aMoGPHjhQWFiI94y3b5ubmxWp49913deGWlZVFXl4eAJ988gkRERGcOHGCdevWsWPHDj7//HOAYsvMz89/4vKsrKzYsWMH58+f59SpU8yaNYspU6Ywfvz4YuP16dOHY8eOkZGRwdSpU7l9+zZBQUFERETQoUMH4uPjH7sMpVKpey2TyUodR6PR4O7uzpYtW3SfJSQkYGNjw88//8yvv/7K+PHjGTJkCNbW1sUO/x7dZo8aP348S5YswcjIiOeeew4LC4snbouePXuyf/9+Tpw4wcmTJ1mzZg1bt24tNo5Wqy2xDpIkUVhYCIBCoSg27qMBamRk9NhhD5mamupey2QyJElCoVCU+D0qbdqapGavfSXw8fEhODiYW7duAUXnYIYOHUpubi4Ao0aN4ptvviEnJ0e31/bnn38yadIk/Pz8sLW15cSJE2g0mn9cQ9euXdm4cSP5+flotVoWLlzIf/7zH1JTU+nevTvW1tZMnjyZWbNm6c7zPbo3cvr0aZKSkkrMV6FQUFBQAMDhw4eZPHkybdq04bXXXsPPz6/Y+c2HevXqxcmTJ7l27RotW7akS5cufP7553Tr1q3YH/3f5/+0WrduTVRUFGfPngXg2rVr9OvXj4SEBP7880+GDx9OQEAArq6uHDp06Km2q7e3N3K5nPXr1+uuUD/J22+/zZ49exg0aBCLFy9GpVIRHR1dbH18fHz4888/iYmJAeDkyZPEx8fr9vT/+OMPMjMz0Wq1/Prrr/Ts2VM3/+3btwNFF1WCg4Pp1q3bU20bb29v7ty5Q3h4OAD79+8nIyPjsf9MagKxB1jBGjVqxNKlS3nrrbeQJAkjIyPWrl2r24vo1asX77//PtOmTdNN8+qrr/LRRx/x+eefo1Qq8fb2Jjo6+h/X8Morr/Dhhx8yfPhwNBoNzZo1Y968eahUKmbMmMHkyZMxNTVFoVDoDoneeecdlixZwubNm/H09CxxiPVw3UxMTBg5ciSbN2/m2LFjDB48GHNzc2rVqsWyZctKTGNpaYm7uztmZmYoFAp8fX157733eO655544///+979Pta42NjasXr2ajz76iLy8PCRJ4qOPPsLJyYkXX3yRRYsWERgYCBSFZURExFPN19/fnz179ugOH5/klVde4b333mPz5s0oFAr69OlD+/btSU9P163Pli1bWLx4MTNnzkSj0WBqaspXX32FpaUlAHZ2dkybNo20tDTat2/P9OnTdfOPjY3F39+f3NxcFixYgJubW6n/oP7O2tqa//znP8ydOxe5XI6XlxdGRkaYmZk91TaojmTSsx5bCUINU1hYyMyZMxk6dCgDBw6s8OV98cUXpKWlsWjRohLDevXqxeeff06LFi2eeb5qtZr/+7//47XXXsPMzIwrV67w8ssvc/z48Rq7Fyj2AKug27dv8+abb5Y6zNXVtdj5RuHfuXnzJmPHjqVPnz70798fKAqSv5/bfMjCwkLXNMXQqFQqlEolI0eOxMjICCMjIz777LMaG34g9gAFQajBxEUQQRBqLIM4BNZqtWRlZaFUKmv07rggCOVLkiQKCgqwsLAotcmPQQRgVlbWU1+NEwRBeFYeHh66K+yPMogAfNjA1cPDQ9d6HSAsLAwvLy99lfVEhlwbGHZ9hlwbiPr+DUOrLT8/n4iIiGKN6B9lEAH48LDX2NgYExOTYsP+/t6QGHJtYNj1GXJtIOr7NwyxtsedWhMXQQRBqLFEAAqCUGOJABQEocYSASgIQo0lAlAQhBpLBKAgCDVWlQzA3IIszkXuIbcgq+yRBUEQHqNKBmBSZgxhd48Rce+svksRhBorLy+vWM/bT7J161b++OOPxw5ft26d7qFQlalKBqAWGwBi0m7ruRJBqLmSkpKeOgD9/f3p3bv3Y4e/9NJLJZ4hUxkM4k6QZxV6L59EtRKtdAdJ0iKTVckcF4RyM2dXCIGXosp1niNbufDRkLaPHf7VV19x8+ZNmjZtSufOncnOzmbcuHEcOXKEsLAwsrKycHd3Z+XKlXzxxRfY2dnh5ubGN998g1KpJDY2loEDBzJjxgzmzZvHwIEDSU5O5ujRo+Tm5hIdHc20adPw9/fn8uXLvP/++1hYWGBra4uJiQmrVq361+tYJZOjdb3a3Eg1Ry7L53522V2BC4JQ/qZPn06jRo149dVXcXNzY9OmTdSuXRsrKyu+++47Nm3axMWLF0lISCg2XVxcHF988QWbN2/m22+/LTFftVrN119/zdq1a1m3bh0AixcvZtWqVfz44484OzuX2zpUyT1AJ2sLkrKsgXQSMiKpbeGg75IEQa8+GtL2iXtrFc3V1RUoup8/NTWVt956C3Nzc7Kzs0s82MrDw0PXI/WjT6976OFzVxwdHXVPI0xMTKRx48ZA0SNf9+zZUy51V8k9QABr86L/ApHJt/RciSDUTHK5HK1Wq3sNcOnSJeLj4/nPf/7DW2+9RW5ubolHcZbV52dpw+vWrcvNmzd1yygvVXIPEMDT0QV1/glkGXf0XYog1Ei2trYUFBToHvEK4O7uzv79+xk1ahTGxsY0aNCAxMTEf72sxYsX8+6772Jubo5SqcTBoXyO+qpsAHZwsWfbeXNaO2aSlZeOhUktfZckCDWKiYkJO3bsKPaZtbU1v/32W4lxHz7zGqBjx46618HBwQClXtAwMTHh0KFDAISGhvLVV19hY2PDf//738f27/esquwhcDsnG26mmAOQmFG+V78EQTAstra2vPjii4wbN47w8PDHPpXvWVXZPUALEyVamQOQwL30SFztK78NkSAIlaN///66x5KWpyq7BwjQ0NaFAo2MqFRxIUQQhGdXpQOwvbMDkWlm5OQnkV+YW/YEgiAIjygzALVaLYsWLWL06NFMmDCBqKji59suX77MuHHjGDt2LK+//jp5eXllTlNeOrnYcyPFHBkSSZnRFbIMQRCqrzIDMCgoiPz8fDZv3szbb79d7GqNJEksXLiQlStX8ssvv+Dr68vdu3efOE15amJvRVymFQCJojmMIAjPqMwADAkJwdfXF4DWrVsTFhamGxYZGYm1tTU//PADzz//PPfv38fNze2J05Rr8XIZ1uZOaCW4mxZZIcsQBKF0z9IbzENnz54lPDwcgJkzZ1ZEWc+kzKvAarUalUqle69QKCgsLMTIyIi0tDQuXLjAwoULcXFxYfr06Xh5eT1xmicpLShDQkKeOE09Ixl3M0yQEcW5c2crtWOEsmrTN0Ouz5BrA1Hf00hKSuL777/Hzc2t2OdPqm3dunX4+PiQlZXFCy+8oPf1KDMAVSoVWVl/dTyq1Wp1QWZtbY2LiwuNGjUCwNfXl7CwsCdO8yReXl7FnikaEhJSrAFlaYaZxbLl3GUa1ErDpUld7C0blLmc8vA0temTIddnyLVB1azvbOQe7iSXb396De1a0t514GOHL1iwgPj4eE6ePElERARpaWmo1WpWrVpFkyZNmDdvHtHR0eTl5TFlyhScnZ25evUq9+7do3///gQEBBAcHMyECRNo2rQpN27cQK1W8/nnn1O/fn3WrFlDUFAQNjY25OTk8MYbbxRrRP008vLynngEWubukre3N8eOHQPg4sWLeHh46IY1aNCArKws3UWOc+fO0bhx4ydOU946OttxQ9cg+k6FLUcQhOIe9gaTk5NDp06d2LBhA1OmTGHJkiWo1WpOnz7Nl19+yTfffINGo8HLywtfX19mz55NvXr1is2rZcuWfP/993Tp0oXdu3cTHh7O8ePHCQwMZM2aNSQlVUyvT2XulvXt25fg4GDGjBmDJEmsWLGCXbt2kZ2dzejRo/nggw94++23kSSJNm3a0KNHD7RabYlpKoq9ypQ8rT1wl4T0O3jW962wZQmCoWrvOvCJe2sVKSIiglOnTrF3714yMzMpKChApVKxcOFCFi5ciFqtZujQoU+cR/PmzYGiTg+Sk5O5desWLVq0QKFQoFAo8PLyqpDaywxAuVzO0qVLi33m7u6ue+3j40NgYGCZ01Skpg71SM0OQ6mIRJKkMnubEATh33vYG4ybmxtDhw5lyJAhHDp0iIiICBITE7ly5Qpr1qwhLy+P7t27M2zYMGQyWYneYUrTqFEjNmzYgFarpbCwkKtXr1bIOlTZW+Ee1cnFnqux5tiYZ5CRm0wtM3t9lyQI1d7D3mCysrLYu3cvv/76K4mJicyZMwd7e3uSkpLw8/PD3NycF198ESMjI1q1asUnn3yCk5PTE+fdpEkTunfvzqhRo6hduzZKpfKpriM8q2oRgB2c7dhxyZyODTJIzIgSASgIlaC03mAevUBT2lHgmDFjGDNmDPBXTzAbNmzQDR87diwAKSkpWFlZERgYSH5+PoMGDcLR0bHc16FaBGDr+jbcuV/U7CYx4w6NHdrpuSJBEP6N2rVrExYWxogRI5DJZAQEBJS4cFIeqkUAmhgpcLCsR07Bbe6l39F3OYIg/EtyuZyVK1dW/HIqfAmVpIOLPTdTzMnMTSYnX63vcgRBqAKqTwCK9oCCIDyjahOAHV1EAAqC8GyqTQC62qhQF1ij0cpIEF3kC4LwFKrFRRAoepReG6c63LlvipH8LgWafJQKY32XJQiCAas2e4DwVwepElqSM2P0XY4gCAauWgVgB2e7R54Ud0e/xQiCYPCqVQC2b2DLzdSiAEwQASgIQhmqzTlAgFpmxjSwtiVBbYKRPAqtpEEuU+i7LEEQDFS12gOEosPg68lmFGrzScu6p+9yBEEwYNUuADu62HElsei+4NuJF/RcjSAIhqxaBuCFOEvyNSbcSDhHgSZf3yUJgmCgql0AejpYY6o05nxcHfI1uWIvUBCEx6p2AWikkNOugS1brpghk8m5Fn/iqXqgFQSh5ql2AQhFD0q6n6PE1LgR97MTuJd+W98lCYJggKplAPb2KOo59nJC0ffw+JP6LEcQBANVLQOwu7sDtc2M+el8FjYW9YhOuUpW3n19lyUIgoGplgGoVMgZ4ulEbHoOpiYtkNASHn9K32UJgmBgqmUAAgxv4QzA0UgLTIzMibh3lkJtgZ6rEgTBkFTbAOzbxBELYyN+uxxHY4f25BVmcSfpsr7LEgTBgFTbADRTGjGgWX1uJGciUzRDhkw0iREEoZhqG4AAw1s0AGBPeAYNbJqRor5LkugnUBCEB6p1AA5sVh9jhZxtodE0q9cZgPD4E3quShAEQ1GtA9DK1Jg+Ho5ciksjq6AO1uZ1uJMcSnZ+pr5LEwTBAFTrAIS/rgbvCIuhqWNntJKGiHun9VyVIAiGoNoH4FBPJ+QyGdtCY3Cv0walwoTr906j1Wr0XZogCHpW7QPQTmVKd/c6nIxKIkmtoZFDO3LyM4lKCdN3aYIg6Fm1D0AofhjczNEHgGviYogg1Hg1IgCHeRU1h9kWGo2VmR31a3uQmBFFoniAuiDUaGU+FEmr1bJkyRKuX7+OsbExy5cvx8XFRTf8u+++IzAwEBsbGwDef/993Nzc8PPzw9LSEgAnJydWrlxZQatQNidrCzo423LkVgIpWXm0bNCLu2kRnL61k0GtX0UuqxH/BwRB+JsyAzAoKIj8/Hw2b97MxYsXWbVqFWvXrtUNv3LlCh9++CFeXl66z/Ly8gDYsGFDBZT8zwxv4cyZ6BR2XYllcgd33O3bcCvpAhH3ztDUsZO+yxMEQQ9kUhn3hq1cuZKWLVsyaNAgAHx9fTl+/Lhu+IABA2jcuDFJSUn06NGDl19+mUuXLjFnzhzq169PYWEhb731Fq1bt37sMvLy8ggLq9iLElEZeQT8fgvf+io+7e5MgZRLRO5eZMjwMB2AkcykQpcvCIL+eHl5YWJS8m+8zD1AtVqNSqXSvVcoFBQWFmJkVDTpoEGDGDduHCqVipkzZ3L48GHq1avHlClTCAgI4M6dO0ybNo19+/bppnnaIkNCQmjbtu1Tr+STtAW8zqVwJiEDD8+WWJoqsborcSbydwprxdOx8Yhnml951lYRDLk+Q64NRH3/hqHVVtbOVZknv1QqFVlZWbr3Wq1WF2SSJDFp0iRsbGwwNjame/fuXL16FVdXV4YOHYpMJsPV1RVra2uSkpLKYXX+neEtnMkr1LI3/C4ATev5UNu8LjcSzol7hAWhBiozAL29vTl27BgAFy9exMPDQzdMrVYzePBgsrKykCSJ06dP4+XlRWBgIKtWrQIgISEBtVqNvb19Ba3C0xve8q+rwQBymYKO7kMBiVO3dqCVtHqsThCEylbmIXDfvn0JDg5mzJgxSJLEihUr2LVrF9nZ2YwePZo333yTiRMnYmxsjI+PD927dyc/P5/58+czduxYZDIZK1asKPPwtzK0dKyNm62KPdfuklugwVSpoG4tN9zs23A76QI37p2liWNHfZcpCEIlKTOV5HI5S5cuLfaZu7u77rWfnx9+fn7FhhsbG/Ppp5+WT4XlSCaTMbyFM58euUrQjXgGN3cCoJ3rAGJSr3I+aj8udl6YKi30XKkgCJWhxjWAe3hXyLbL0brPzI2taO3ch7zCbM7f2a+v0gRBqGQ1LgA7OtvhaGXGzisxFGr+OufXzLEz1uYORCScJVlcEBGEGqHGBaBcLsPPqwGp2fnsuXb3kc8VdHIfBkicFBdEBKFGqHEBCDC9c9GV7I8OXSn2jJCiCyKtSVHHciPhrL7KEwShktTIAPRyrM0QTydORiVx7HZisWHtGg5EqTDh/J39ZOdl6KlCQRAqQ40MQIB5vYvuXV4ZFFrsc3MTK7xd+pNXmM3h8J/QaAv1UZ4gCJWgxgZgJxd7ejZy4GBEPCExKcWGNXXshJt9G5Iyozl5c5t4lKYgVFM1NgAB5vVuAcCqQ8XvFZTJZHRu5I+tyombiSHiSXKCUE3V6ADs3bgu7RrYsi00mvCE9GLDjBRKejV7HlOlijO3dxN//5aeqhQEoaLU6ACUyWTM6+2FJMFHh6+UGG5hYk3Pps8jk8k4Er6RzNxUPVQpCEJFqdEBCDDMswHNHGqxMeQ20WlZJYY71GpIR/eh5BVmc+jqjxRo8vVQpSAIFaHGB6BcLmNOL08KtRKfHim5FwjQpG5HmtTtSFr2PYJvbBEXRQShmqjxAQgwto0rLrUt+PbUTRIzc0odp4PbEBysGnInOZTQ2COVW6AgCBVCBCCgVMh5p4cnuYUaVh8PL3UchdyIHk2fx8KkFuejDpChiavkKgVBKG8iAB94oaM7dVSmrAm+TnpO6ef5zIxV9Gw2AYVcQUz+aaJTrlZylYIglCcRgA+YKY2Y1a0ZGbkFrD1x/bHj2amc6NZkLBISh679yIWoA0ii4wRBqJJEAD5iemcPapkq+ezYNbLzH38LnIutJ+4mvVCZ2HAp5hB/XP2BvMLsSqxUEITyIALwEbXMjHmlSxOS1Hl8d+bmE8c1k1szpPVM6lk3JjbtOr9fXENa1r1KqlQQhPIgAvBvXvdtiplSwSdHrpJfqHniuCZKc/p4vkALpx5k5qaw+9IaIpMuVVKlgiD8WyIA/6aOpRkv+TQmOi2L/xwt+yKHXCanbcP+9Gz6PMhkHL3+C2cjd6OVnhyegiDonwjAUizs2xIHS1OWHQjlVnLmU03jYufF4FavYmVmx5W7x9kXuo576bcruFJBEP4N/T+r0gDVNjfhv8PaM+6n48wIPMX+l/sgk8nKnM7a3IHBrWYSfCOQqJQw9oWuo45VQ1o69aR+bY+nmocg1FS5BVlk5qZSqMlHoy2gUFug+16oKXptbV4HZ1vPclumCMDHGNXahR/O3WJ/eBwbz0fyfFu3p5rO2MiUns2eJzEjissxh4lNCyfo6nfYWNSjZYOeONt6IpeJHW+hZsstUJOsvkvKI19ZeffLnM5UqaKBTfNy25kQAfgYMpmMNf4daPHxLt7ecY4BTetja2Hy1NPXsXKhj+dkUtVxhMYe5U7yZY6Eb8TKzJ4WTt1xt2+DXK6owDUQBP0rKMwjPSdJ93U/6x4pWXfJyive/Zyp0oL6tT2oZVYHpcIEhVyJkUKJkVxZ9FquxEhhTC2zOuV6JCUC8AlcbS1Z0q8Vc38/z5xdIawf0/mZ52Gjqkf3pmNpk9OX0Ngj3Eq8QPCNQM5F7qGOVUPq1nLFoZYrNhb1xJ6h8K/kFmSRmhWPVtIglymQy+QPvj94LVcgQ06hNp9CTT4FmrwHX0WvCzV5aLSFGBuZYaq0wNRYhZlShalShanSAoX8r7jQSloKCnPJK8whvzCbvMIc8gpzSC6M4OTNGNJzEsnISSY7v+RzdcyUljjVboqtqh62qvrYqpwwN7bSyykiEYBleKNbM34+H8n3Z28xoZ0bPRrV/UfzsTKzo0vjkbR27kPY3ePEpFwlJrXoC0CpMKGOVUMcrFypW8sVW1X9Yr9wgvCoAk0eKeq7JGfGkqyOJTkzFnVexfZXqVSYYmxkSoEmj/zCXKD0XpHiHzSHtTCxpp51Y2qZ2VPL3P7B9zqYG1tVaJ3PQvyFlUGpkPNVQCc6r97LjMDTXHh7MKbKf37oamFiTUe3IXR0G4I69z4JGZEkpEdyL+M2d9Ouczet6DY8Y4UpzraeNLRriaO1uwjDakqStGTkpBSFmDqWVPVdCjUFyORyFA/33uQK3Z6cTCYjPjeKsJOBSI8EkImROfVre2Crqo+R3BitpEEradFKGiTtI68lCYVCiVJhjFJhglJhgpH8wWsjExQyI/IKc8gtUJNbkEVOgbrodX4WuQVq8jU5mBtbUdvcAWMjc0yMzDAxMsNYWfQ6LiaBVs07UMvMDiOFsR637NMRf1VPoYOzHa92acKXf15n1R9hLOnfqlzmqzK1RmXaBvc6bQDIzs8sCsP028SkXuVmYgg3E0MwNjLD5WEY1nIX5w71pFCTT9z9mxRq8rGzbIClqc0zHbZJkkRW3n1d2KVkxpKsvkuBJlc3jgw5CoURWq3msW1J5SioY9UQO0sn7FRO2Fk6oTJ5tloqSlZcCLaqevou46mJAHxKywa0ZltoDKsOhTG6TcMKWYa5sSWu9i1xtW9JJ2koSZkxRCZfJio5lBsJ57iRcA4TI3OcbZvjWKsRlmY2WJraYmJkbhC//NVRfmEusanhRKWEcTftOoXaAt0wU6UKe8sG2Fu6UMfKGVuVE8oHez1arYb0nCRSs+JIUceRmhVHalY8+YXF+5u0MrOjgarpgyBrgI1FPYwUSqAoMCW0aLXaB3t0RV9XL0fQrmW7ytsI1ZgIwKdkZWrM6uHtGfH9UWYEnuLjjrYVujyZTE4dKxfqWLnQwXUQiRnR3Em+zJ1HwvAhpcIES1ObB1+2qExtUGvSyM7LwMzYUoTjMyqU8oi4d4aolCvE37+p2xOzMrPDxdYLU6UFSZkxJGVGEZN6jZjUa0DR3ltti7rIZDLSshLQSsU71LAytaOedSNsLOpjZ+mErao+JkZmj61DJpMhQ4FcoQCUxT4XyocIwGfg18KZoZ5O7LwSy047Be0r6Z+wTCbHoVZDHGo1pL3bYJIyoknNiiMzN5XM3BQyc1NJz0kmNSu+2HSRZ4+iVJjoTj7XMvvrRLSFibVub6WmKjokTed+9j3SshO4n3WPtOx7pObGw4O+MGws6uFi64mLnVepTTCy8tJJyowu+sqIJll9F4DaFg7YWDhiY1EfG5UjNuaOKI2evhmVUDlEAD6j1cM7cOjmPb64kMDUvmpcbFSVunz5I2H4KEmSyClQFwViTgrXI8Mwr6V4cBgWT7I6tsS8lAoTzJSWmBmrMDO2wkypwszYEjNjSxysGmJlZldJa1WxCrUFqHNTychJITM3hfTsJNKy73E/O4ECTV6xcRVyI8zltjR37oiLnSeWpk/e07cwqYWFSQsa2hU9Y1qrLdpbFOdpqwYRgM+oQW0LPhnajulbTjHqx2McfbXfv7oqXF5kMhnmxpaYPwiv9Fho26wtAFpJgzo3jfTsRF2D1Ky8DHIKMsnJzyQjI4XSmjRYm9fB2cYTZ9vm2KqcyuXQS6MtJFebwb302+QVZJNbmEVuQRZ5BVnkFmSTV5hFgSYPY4UZJsqHVxnNMVFaPPhujrHCBI2kQastRKPVoJEKH7wuRCtpyCvM0f0jyMhNJisvo8T6yZBTy9wOa/O61DZ3wNrcgdoWdVGZ2nDh/AW8nNr+o/UTwVe1lBmAWq2WJUuWcP36dYyNjVm+fDkuLi664d999x2BgYHY2NgA8P7779OwYcMnTlPVTe3YiN9DrvH77RTe2H6GrwN89F3SE8llCqzM7IpOuJcyXCtpipo85BcFojrvPnfTrhN3/waXYw9zOfYw5sZWONsWhWFdK7cy/9CL7gBI5H52IvdzEovCNzuJzNwUJCRuhD5pahmPa2P2rMyNa1G3liuWprZYmdk++G5HLTN70bRIKDsAg4KCyM/PZ/PmzVy8eJFVq1axdu1a3fArV67w4Ycf4uXlpfvswIEDT5ymqpPJZMxp50hsrpxvT92kg7MdUzo21ndZ/5hcpsDc2KpYA9Wmjp0o0OQTlxZBdMoVYtLCCY8/SXj8SZQKE4yfcPJeK2nIyS/Zi46JkTl1rFzIz5LRoJ4rpg/27EyVFpgozTE1ssBEaYGRXEmBJo+8wuyir4Li3ws0echlChRyI+QyIxTyotcKmRFyudGDi0K2WJra6K6oCkJpZFIZD7lduXIlLVu2ZNCgQQD4+vpy/Phx3fABAwbQuHFjkpKS6NGjBy+//HKZ0/xdXl4eYWFh5bE+lequOp9J+26TWyjxzXMNaWbz+FCo6iRJS5Y2mQzNXdTaBLQ8vr9DGTKMZRaYyKwwkVthIrPCVG6FkUxcBBD0w8vLCxOTkr9/Ze4BqtVqVKq/TvQrFAoKCwsxMiqadNCgQYwbNw6VSsXMmTM5fPhwmdM8bZEhISG0bfvPzsVUtJCQEIZ29+HnOs4MWX+IRacTOfvmoGfqMKEiVeS2S7tzBxMrK8wfnPZ4Vob8cwVR379haLWVtXNV5t33KpWKrKws3XutVqsLMkmSmDRpEjY2NhgbG9O9e3euXr36xGmqmwHN6rP4uVZEpWUx/qfjaLTV/wlxMcHBfNGoEd+0b89v48ZxZMkSbuzbR0FO6Q+VFwRDVWYqeXt7c/jwYQYOHMjFixfx8PDQDVOr1QwePJg9e/Zgbm7O6dOnGTFiBLm5uY+dpjp6r08LTkcns/faXd7ff5mlA1rru6QK1WLcOEK++oroP/8k7txfDbJVdeti06gRNh4e2DZujGvPnji2bYu8mv7zE6q+Mn8z+/btS3BwMGPGjEGSJFasWMGuXbvIzs5m9OjRvPnmm0ycOBFjY2N8fHzo3r07Wq22xDTVmVwuY8O4LnT4bA8fBIXS3tmWIZ6lXW+tHmQyGS3Gjyf6zz+Lfa6+dw/1vXt/fS6XY+3igk2jRtRr147e1fz3QKh6ygxAuVzO0qVLi33m7u6ue+3n54efn1+Z01R3tc1N2DKpO11W72PSz8GceXMgjewMp9uf8uY9bRrnv/mG+PPnHz+SVsv9yEiykpJo8+KLlVecIDwl0QNnOWpd34a1AR1Jzy1gxHdHScvOK3uiKkquUNBs5MinGrfDzJl4jRlTwRUJwrMTAVjOJrZzZ2bXJoTdu8/gbw+RmVtQ9kRVVKdZs7Br2vSJ47g99xy9P/igkioShGcjArAC/HdYe8a3deVUVDJ+/ztMTkFh2RNVQUozM5r87fTH38WdPcvxlSspo7mpIOiFCMAKIJfL+N/ozgxv4cyRWwmM+P4oeYXV80HpXebMoZaz82OH56alcXjRIraOH09+dnYlViYIZRMBWEGMFHJ+fr4r/ZvWY394HON/+pNCTfVrI2hWuzaNBw9+8khaLWG//MJPffuSHB5eOYUJwlMQAViBjI0UBE7uTg93B7aFRvPCphNotdXvULDLnDmY16mje99u+nSaDBtWYryYEyfYOHAgcUFBlVmeIDyWCMAKZqY0YvuLPenkYsfP5yN55bfT1e58mLWLC40HDgSg0YABDPjyS0Zv3UqXuXNRWlgUG/d+ZCSXly3j0MKF1W47CFWPCMBKYGmqZPe03rSuV5tvTt3g7Z3nqt0ff6dZs3Bo2RK/779HrlAgk8vps2oVQ7/5hlp/6wqtMCuL48uXsyUggLzMkr3GCEJlEQFYSazNjNn3ch+aO9Ti82PhLNp3sVqFYN1WrZh05AgWjxwKA3iNHcvYXbuo37FjiWmu/fYbP/buzb0LFyqrTEEoRgRgJbJXmXJgeh/cbS1ZERTG/N0XqlUImtWuXernDi1aMCEoiOYBAfC3XqXjzp7ll2HDuLxhQ2WUKAjFiACsZI5W5hx6pS8e9lZ8fPgKb+6ofofDpTFRqRi5eTPdFy3CSFX8OSoZMTH8PmMGB2bPRqoBvekIhkMEoB44WVtw+JXn8Kxbiy+OhzM98FS1vDr8dzKZjB5LltBq8WJqP3I/OUBBVhYnP/mETcOGkZ2SoqcKhZpGBKCe1LUy49CM52hT34ZvT93khU0nqmU7wdI49uzJ+L17adClS4lhEb//zoY+fYg9eVIPlQk1jQhAPbJTmRI0oy8dne34KeQ24zf+SUENCUHbxo2ZcPAgLZ5/Hpmi+AOW7l28yK8jR3Ju3To9VSfUFCIA9czazJj9L/fB160OgZeiCPih+t4293dKMzOG//gjvT74ANO/da+fGRfHvtdfZ8/MmWgLq+e91IL+iQA0AJamSnZP7UXvxnXZdSUWv/8dITu/ZvzRy2Qyus6di//Gjdg2aVJsmCYvj7Nr1rBx0CDU9+7pqUKhOhMBaCAsTJTsnNKLgc3qc+B6HEPXH6oxIQjQuH9/xu/bR8OePUsMu33gAD/26cOdw4f1UJlQnYkANCCmSgW/Te6OX4sGHL6ZgP93R8gtqBmHwwC1Gzbk+X378J42Dbmy+PN8k65cIXDMGE6tXq2n6oTqSASggTE2UvDL874Mal6fgxHxBPxwlPwack4QQGFszJB163juk09K3FWSlZjIwdmz2Tl1Kpr8fD1VKFQnIgANkLGRgl8ndqevhyN7rt1l7E/Ha8zV4Yc6vv46Izdtoo6XV7HPtfn5XFi/np/69eN+VJSeqhOqCxGABspUqWDbiz3o2ciB7aExTPy5evYn+CQNe/ZkwsGDuD/3XIlhd44c4ad+/bixb58eKhOqCxGABsxMacSOF3vS1bUOv16MYsrmkzXiweuPUtWty7jdu2n/6qsoTEyKDUu5fp2t48fz54cf1ojbCYXyJwLQwFmYKNk1taeusfSMwNM14ra5R8mNjBj45Zf0X70ay3r1ig3LTU3l0HvvsW3iRApycvRUoVBViQCsAqxMjdnzUm/aOtmw/vRNXt92pkbu8bR76SVGBQZSt3XrYp9LGg2hP/3Ehr59SblxQz/FCVWSCMAq4mF/gi0da7P2RASTfznBvYyat8fj5OPDhKAgPEp5DklMcDAbBwwgfNs2PVQmVEUiAKsQG3MTDkzvQ6t6tfkp5DaNV25j0d6LZOTWrCYh5ra2jNmxA5933inR5X7arVtsf+EFjixZUiP3koVnIwKwirFXmXJ61kDWjOiIlYkxHwSF0uiD7Xx29GqNuYcYQCaX89zHHzN47doSj+XMS0/n6NKlBI4eTZ5aracKhapABGAVpFTImd7Zg4j5w1g+oDUFWi1v7wyh6aod/HjuVo26UtxywgTGbN9Ovfbtiw+QJK5u2cKGPn1ICA3VT3GCwRMBWIVZmCiZ36cFN98dzlvdm5OQmcMLv5zA+9PdnI6vOXs+ddu0YWJQEM1GjCgx7O7p0/wyZAihv/yih8oEQycCsBqwtTDh46FtCZ/nx+T27lxNSOe1w9FM3XyC+zk14/ygiZUVAVu24LtgAcZWVsWGpUdFsWvaNILmzRNd7gvFiACsRpxrW7B+TGfOzBqIR21TvjtzC6+PdrLrSoy+S6sUMpmMXsuWMWz9eqxdXYsNK8jKIvjDD9ns70/u/fv6KVAwOCIAq6E2TjZ838+VZQNak5yVh9//jvD8T8dJVufqu7RK0XzkSMbt3o2Tj0+JYdd37ODH3r25e/asHioTDE2ZAajValm0aBGjR49mwoQJRD3mBvSFCxfyySef6N77+fkxYcIEJkyYwPz588uvYuGpGMllvNunBSFvDaJ9A1t+uXAHr493suVSzehAwL5ZMyYEBeE1dizIi/+ax58/z2Z/fy58952eqhMMRZkBGBQURH5+Pps3b+btt99m1apVJcbZtGkTERERuvd5eXkAbNiwgQ0bNrBy5cpyLFl4Fp51rfnztf58NNibzNxCxvx4jIAfjhKfka3v0iqcsbk5/hs30nPpUkz/9szizNhY9rz6KvtmzUKrqTnNh4TiygzAkJAQfH19AWjdujVhYWHFhl+4cIFLly4xevRo3Wfh4eHk5OTw4osvMnHiRC5evFi+VQvPxEgh5+2enlx4ZzBdXeuw9XI0TVbuYMm+S2TmFui7vAolk8no9t57DN+wAVsPj2LDCnNyOP355/wyZAhZiYl6qlDQJ5lURnP59957j+eee47u3bsD0KNHD4KCgjAyMiIxMZH58+fz5ZdfsnfvXm7fvs0777zD9evXuXTpEgEBAdy5c4dp06axb98+jIyMSl1GXl5eiWAVKoZWkthx6z5fX04kNVeDjamCqV72+DWqjZFcpu/yKlRWbCyXli4l9fz5EsMsGzemxdy52PztPmOhevDy8sLkb70JAZSeSI9QqVRkZWXp3mu1Wl2Q7du3j7S0NF566SWSkpLIzc3Fzc2NwYMH4+Ligkwmw9XVFWtra5KSknB0dHymIkNCQmjbtu1Tr2RlMuTa4Mn1tW8H8/wK+O/Ra3x8+AofnbvHtjtZLB/YhhEtnZHJKjYI9bbt2ralc//+7J4xg0sbNiA98rS5zBs3uLRgAYP+7//Idnausj9bfTO02srauSrzENjb25tjx44BcPHiRTweOYyYOHEiW7duZcOGDbz00ksMHjwYf39/AgMDdecKExISUKvV2Nvb/9t1EcqRykTJwudacuNdP2Z09iAyVc3oH4/R9Yt9HL+doO/yKoyRiQnD/vc/+n70EeZ2dsWGWdati0u3bnqqTNCHMgOwb9++GBsbM2bMGFauXMn8+fPZtWsXmzdvfuw0I0eOJDMzk7Fjx/Lmm2+yYsWKxx7+CvrlYGnGlyM6EjpnKP4tnTkVlUyPNQd4ecvJav0sEp8332TEL79g17w5AJb16jFo7VrMbW31XJlQmcpMJblcztKlS4t95u7uXmI8f39/3WtjY2M+/fTTcihPqCwe9lZsmdSdU1FJvBp4mm9P3SQ8IYMtk7pRx9JM3+VVCLc+fZhw4AA7XnwRz5EjcerUSd8lCZVMNIQWiunkYs/x1/oT0MqFPyMT6fj5Xi7eTdV3WRXGqn59Juzfj/e0afouRdADEYBCCebGRvwywZdlA1oTnZaF75f7CKwhDagBkpKSaNasGcOGDWPYsGEMGTKEgIAAQkJC/vE8Y2JieO2118qxyqezdetWXn755UpfblUhTswJpZLJiu4kae5Qi4k/BzP6x2Ms7NuSRc+1RF7Nm8sAmJqasmPHDt37PXv2MH/+fA4cOPCP5hcXF0dkZGR5lSeUE7EHKDyRXwtngl/vj6uNimUHLzPqx2Oo86p34+nS3L9/v1hLhkOHDhEQEICfnx9jxozhwoULANy6dYsxY8bg7+/P8OHD2bhxIxqNhgULFhAdHc2UKVNKzHvChAl8+umnjB8/nl69evHee++h1WqJjY2lTZs2uvEefb9161amT5+ua30xZcoU9u/fz4QJE/D19eV///ufbrqkpCSmTJnCkCFDmD59OklJSQBkZmYyb948/P39GTJkCCtWrKDwQdMgLy8v3njjDfr160doNe5PUQSgUKYWjrU59cYAerg7sC00Gt8v9nPoRny17ng1NzdXdwjcs2dPVqxYwUsvvQTAnTt3+O9//8u6devYvn07y5Yt47XXXiM7O5v169fTq1cvtm7dyrp16zh37hwymYzly5fj7OzM+vXrS11edHQ0GzZsYOfOnRw7dowzZ86UWeO5c+d4//332blzJ/fu3WP37t388MMPfPPNN3z22WdoH/x8IiMjWbRoEbt27cLDw4MPPvgAgBUrVuDp6cnWrVvZvn07aWlpfPfg/uiCggJ69uzJ/v37adGiRXlsUoMkDoGFp2KnMmXfy314c/tZ1p6IoO9XQThamRHQyoUxbRrSwdmuwhtQV6a/HwKfOHGCV199lZ07dxIcHExiYiKTJ0/WDZfJZERHR9O3b1/mzp3L5cuX8fHxYcGCBcjlZe9n9OzZE7lcjkqlwsXFhfT0dJycnJ44TYsWLXQ3Fzg5OdG1a1fkcjkNGjQgLy+PnAePCe3cuTMuLi5AURO1kSNHAnDkyBFCQ0MJDAwEikL/Ue3atSuz7qpOBKDw1JQKOV+O6Mio1g3ZeP42v12KZvXxcFYfD8fVRsXoNg0Z06YhLRxrlz2zKqZz5844OzsTGhqKVqvFx8eHzz77TDc8Pj6eOnXq0LRpU/bv38+JEyc4efIka9asYevWrWXO39TUVPdaJpMhSZLu+0MFBcVPPRgbGxd7/7i2tgqFQvf60Tu5tFotn3/+ua5ZW0ZGRrF/Yubm5mXWXdWJQ2DhmXVzd+DrAB/iloxk55SejPN2JVGdy6o/wmj9ye+0/Hgnnx29Smp2nr5LLTeRkZHcvXuXZs2a4ePjQ3BwMLdu3QLg6NGjDB06lNzcXN5++2327NnDoEGDWLx4MSqViujoaBQKRYkAK4uVlRUFBQXcvHkTgN27d/+j2k+fPk1cXBxQ1HNTtwd3u3Tt2pXvv/8eSZLIz89nxowZ/PTTT/9oGVWV2AMU/jFjIwWDmjsxqLkT2fmF7L52l80X7rDnWixv7wzh3T0XCGjVkJd9GuPT0L5KHSI/PAf4kFarZenSpbg+6Gl66dKlvPXWW0iShJGREWvXrsXCwoJXXnmF9957j82bN6NQKOjTpw/t27cnPT0dExMTRo4cyZYtW55qW1haWjJ79mymTZuGjY0N/fv3/0fr4uHhwbvvvktycjJubm66Gxvee+89PvjgA4YMGUJBQQGdO3dm6tSp/2gZVVWZvcFUhoc3LIvOEMqPPutLycrjx3O3+PpEBDeSMwFo4WjNyz4ejG/ryo0roWLb/QuGXJ+h1fa4bHlIHAIL5c7WwoQ3uzfn2rxhHJzeh5GtXLiWkM7MrWdwev83VpyO41pCur7LFARxCCxUHJlMRq/GjvRq7Mi9jBy+O3OTb07dYPut+2z/aCdDPZ2Y29uLTi6ipyBBP8QeoFAp6lqZMb9PC26868eHvk50cLZl55VYuqzeR881+9l77S4GcDamTPHx8QwePJhhw4bpGj8/tGXLFjZu3KiXutRqNVOmTCnWlCU+Ph5fX19SU/+6l/vUqVP4+/szdOhQRo0axeXLl3Xjzpw5U9d2sKYQAShUKoVcTs8GVpx4fQB/zOhLv6b1OHY7kcHfHsL70938fD6SQo3h/hGePn0aOzs7duzYUewuDSg6//X3tnSV5ZNPPiEgIEDXnGb79u2MHz+exEe6+s/Pz+fNN99k+fLl7Ny5kxkzZjB79mwAHB0dadq0KT///LNe6tcXcQgs6IVMJqNHo7r0aFSXi3dT+fjwFX69GMWEjX+ycO8FpnVqzKT27jha6act2ubNm9mwYQNyuRw7OzsWLlxIQkICn332GZmZmUyYMIENGzboxj948CCHDh0iODgYU1NTUlNTuXjxIomJiTRp0oRPPvmEtWvXcuDAAbRaLfXr12fx4sU4ODiQmZnJBx98QEREBAUFBfj4+DBnzhyMjIxYvXo1Bw8eRKlUUrt27VIfMBYfH8/hw4dZsGABUNQJcVBQEOvXry925djY2Jhjx46hVCqRJImYmBhqP/KwqICAAEaOHMmoUaNKtDGstiQDkJubK507d07Kzc0t9vm5c+f0VFHZDLk2STLs+h5X263kDOmVwFOSxdyNkvytHyXlOxskv/WHpF1XYqSCQk2l1ffDDz9Iffr0kVJSUiRJkqTffvtNGjBggKTVaqXffvtNeumll0qdbu7cudK3334rSZIkrV69WurXr59UUFAgSZIkbdu2TZo1a5bu/aZNm6SpU6dKkiRJ8+bNk3788UdJkiSpsLBQeuedd6R169ZJcXFxkre3t5SXlydJkiStX79eOnjwYIntt2HDBmnu3Lml1uTh4aFbj4eSkpKkrl27Sp6entLBgweLDRs8eLB08uTJp9tQpTC037vHZctDYg9QMBhutpasGdGRDwa24ZcLkaw/dZOdV2LZeSWW+rXMmdzenRc7NqKhjapC67h06RIDBw7ExsYGKOrs94MPPiA2NvaZ5tO6dWvdXReHDx8mNDSUESNGAEXtCh/eqva4W9IcHBxo2rQpw4cPp1u3bnTr1g0fH58S3XLdvn0bZ2fnp67Lzs6O48ePc+XKFSZPnoy7u7uufaOTkxORkZF0qiGdw4oAFAyOtZkxMzo3YUbnJpyPTWH96Zv8fD6SD4JCWfFHKL0a1SWgdUP8vBpgrzIte4bPqLQLAZIk6XpKeVqP3kqm1WqZOnUq48aNA4rOx6Wnp+uGlXZLmlwu56effiI0NJSTJ0+yYsUKfH196d27d7HlyGSyp7p4kZmZyalTp+jbty8Anp6eNG3alIiICF0AKpXKYrfOVXfiIohg0LydbFkzoiOxi0bwvzGd6dKwDn/cuMf0LaeotySQ3v93gDV/hhOXXn4Pem/VqhV79uzRXT397bffsLa21nUo8DgKheKxIdm1a1cCAwNRq9UAfP7558yZM0c3rLRb0sLDwxk8eDDu7u68/PLLTJ48udSuqVxdXYmJiSlzveRyOe+++65uD/LGjRvcvn2bVq1a6caJjY3Fzc2tzHlVF2IPUKgSLEyUTGrvzqT27kSmZLItNIatl6M5ciuBI7cSeH3bWXxc7PFv6cyIls64/IvD5BYtWjB58mQmTZqEVqvFxsaGr7/+usxeXbp166Z7GuLfBQQEkJCQwKhRo5DJZDg6OurGfdwtaUqlkgEDBjBixAjMzc0xNTVlwYIFukPnh/r06cO3336LRqN54t6bhYUFa9as0fX7Z2xszCeffELdunUBSE5OJiUlBW9v72fZXFWauBXuHzLk2sCw6yvP2u6mZ7M9NJqtl6M5djsR7YNf5x7uDkzq4M6IFs5YmCj1Vl9FKK2+hQsX4uPjw8CBA//xfL/44gtsbGwYP358udamT+JWOKFaq1/LnFe7NuWPV57j7uIRfBXQiR7uDhy5lcALv5yg/vu/MW3zSYIjE6tEQ+t/avbs2WzZsuUft0OMj4/nypUrjBkzppwrM2ziEFioNupYmjGtU2OmdWrM7ZRMfjh7ix/P3eZ/Z27yvzM38bC3YlJ7N8Z7u9GgtoW+yy1XVlZWut6c/wlHR0e++uqrcqyoahABKFRLbraWvN+/NYufa8Whm/f4/sxNtoXG8N6ei7y35yLNHWrR28ORXo3q0t3dgVpmNaThr1CMCEChWpPLZfTxcKSPhyP3c/LZfPEO20NjOH47gS+Oh/PF8XAUchkdGtjR26MuvRo7YqKpvofKQnEiAIUaw9rMmJd9PHjZx4O8Qg2nopL5IyKeQzfucSYmmZNRSSw/GIpKKWdMVAET2rrTxbVqdeQqPBsRgEKNZGKkoLu7A93dHVg6ADJy8zl6K4GgiHh+DbnFt6du8u2pm7jZqni+rRvPt3XD3c5S32UL5UwEoCAAVqbGDPFswBDPBox3kpNRqz4bzt1ma2gUSw9cZumBy3RpaM/z7dwY1boh1uKcYbUgAlAQ/kbxyHnDNXkd2BYaw4Zztzh08x7Bd5J4c/s5RrRyZmrHxvi61RGHyFWYCEBBeAKViZIJ7dyY0M6N2PtZbAyJ5LszN9kYEsnGkEg87K2Y0rERE9u5UcfSTN/lCs9INIQWhKfkZG3B3N5eXJs3jEOvPMc4b1ei0tTM/f08DZb+RsAPR9kfHoemhvWqXJWJPUBBeEYymUx3AWX18PZsDInk29M32Hq56JY8azNjOjjb0dHZjg4uRd9tLUrehiXonwhAQfgXapubMNO3Ka92bcLZmBT+d/omh2/e48D1OA5cj9ON18jOko4PwrCPhyNN6tTSY9XCQ2UGoFarZcmSJVy/fh1jY2OWL19eardACxcupFatWrzzzjtPPY0gVBcymYwOznZ0cLYDIFmdy5mYFM5EJXMqKomzMSm684YAberbMKZNQ0a1bohzNbstryopMwCDgoLIz89n8+bNXLx4kVWrVrF27dpi42zatImIiAjat2//1NMIQnVmpzJlYLP6DGxWHwCtViIiKYOTUUlsC41mf3gcF+6mMvf383R1rcOYNg0Z2cqlQjp4FR6vzAAMCQnB19cXKOriOywsrNjwCxcucOnSJUaPHs3t27efahpBqGnkchlNHWrR1KEWL3RoRGp2Hr9djmbzhUiO3Ergz8hE3th+lt6NHenuXgc3W0vcbS1pZGcp7lOuQGUGoFqtRqX6q3PJh73eGhkZkZiYyJdffsmXX37J3r17n2qaJyktKP/+/ANDYsi1gWHXZ8i1QeXU560E7w62JHlZERSdwYGo9BLnDgFqmShooDKmvkpJA0tj2jpYoDl7DoXcMNsfGvrP9lFlBqBKpSIrK0v3XqvV6oJs3759pKWl8dJLL5GUlERubi5ubm5PnOZJRIeo5ceQ6zPk2kA/9T18eGVUqprQe/e5nZzJrZRMbqWouZWcyfVUNWEpRT1BfxuWjKOVGQGtXBjdpiEdne0MpjG2of1sH3aI+jhlppK3tzeHDx9m4MCBXLx4EQ8PD92wiRMnMnHiRAC2bt3K7du38ff3Z//+/Y+dRhCEx3OxUZXanb9GqyX2fjZXE9JZf+Q8R+OyWX08nNXHw3GpbcHo1kUXVFrXr20wYVgVlBmAffv2JTg4mDFjxiBJEitWrGDXrl1kZ2czevTop55GEIR/TiGX68KxTvY9fmndhqCIeF33Xh8dvsJHh6/gYW+Fl6M1dVSmOKhMsbc0xUFlRh2VKXUsTalraYqVqTin+FCZASiXy1m6dGmxzx4+vu9R/v7+T5xGEITyo1TIGdCsPgOa1Sd3pIa94XfZfOEOv1+NJSIp44nTdnKxY7y3GwGtxVVn0RBaEKo4U6WC4S2cGd7CGY1WS0pWHonqXBIyc0lU55KkztW9v52SybHbiZyKSubNHWfp17Qe473dGOLphLlxzYuDmrfGglCNKeRy6liaUcfSDC/H0seJz8hm84U7bDwfye6rd9l99S6WJkr8Wzoztk1DurrVwUxZM6KhZqylIAg6jlbmzOrenFndm3P13n1+Ph/Jz+cj+eHsLX44ewuFXEaLuta0c7alXQM7Ojjb4ulgjZGi+vWdIgJQEGqw5nWtWT6wDUv7tyb4TiLbQqM5E5XChbupXIxL49tTNwEwUypoU9+GDs529Gpc9CAp1TM+b9kQiQAUBAG5XIavmwO+bg4AFGi0hMXf52xMMudiUjgbncLp6GRO3Enis2PXUCrkdHW157km9ejrUY9W9WojN9CG2U8iAlAQhBKUCjltnGxo42TDSz5Fn2XnF3I6OpmgiHgOXI/j8M0EDt9MYP7uC9RRmdK3iSPNTPJp3UaLQl41DpdFAAqC8FTMjY3o2aguPRvV5YOBbUhS53IwIp6D1+M4GBGv6+lmR8w+1o3yoWW92nquuGwiAAVB+EfsVaaM83ZlnLcrkiQRdu8+8wOPs/dOCu3/u5vZPT1Z0LclpkqFvkt9rKqxnyoIgkGTyWS0cKzN+53rs3taL+rVMmflH2G0+fR3jt1K0Hd5jyUCUBCEctW/aX1CZw/hdd+m3EjOoOf/HeCVwNOk5+Tru7QSRAAKglDuVCZK/uvXnj9f649n3Vp8fTICr492sjHkNtcT01HnFei7RECcAxQEoQJ1crHn3JuD+OjwFT44GMrEn4N1w2qZKnGyNqeelTlO1uY41bKgkb0l/ZrUq7R7lEUACoJQoYyNFCzo25KRLV3YfPEOsfezuZuRzd372cSmZ3PlXnqx8eUyGV1c7Rnm1YChng1wt7OssNpEAAqCUCmaOtRicb9WJT7PyivgbkYOsfezCIlJZeeVGP6MTOT47UTe2RmCV11rhnk1YJhXA7ydbMq1v0MRgIIg6JWFiRIPeyUe9lb0auzI7F6eJGTmsOtKLDvCYvjjRjwfBIXyQVAoXRrac+y1/mXP9CmJABQEweA4WJoxtVNjpnZqjDqvgP3X49gZFoulSflGlghAQRAMmspEyYiWLoxoWf7PFhfNYARBqLFEAAqCUGOJABQEocYSASgIQo0lAlAQhBpLBKAgCDWWCEBBEGosg2gHKEkSAPn5JbvLycvLq+xynpoh1waGXZ8h1waivn/DkGp7mCkPM+bvZNLjhlSizMxMIiIi9F2GIAjVlIeHB5aWJTtVMIgA1Gq1ZGVloVQqy/VGZ0EQajZJkigoKMDCwgJ5KQ9qMogAFARB0AdxEUQQhBpLBKAgCDWWCEBBEGosEYCCINRYIgAFQaixDKIh9KO0Wi1Llizh+vXrGBsbs3z5clxcyr8jxH/Dz89P16bIycmJlStX6rkiuHTpEp988gkbNmwgKiqKefPmIZPJaNy4MYsXLy61CYC+6rty5QrTp0+nYcOGAIwdO5aBAwfqpa6CggLeffdd7t69S35+PjNmzKBRo0YGsf1Kq61u3boGs+00Gg0LFiwgMjIShULBypUrkSTJILbdU5MMzP79+6W5c+dKkiRJFy5ckKZPn67niorLzc2Vhg0bpu8yilm3bp00ePBgKSAgQJIkSXr55ZelU6dOSZIkSQsXLpQOHDigz/JK1Pfrr79K69ev12tNDwUGBkrLly+XJEmSUlNTpe7duxvM9iutNkPadgcPHpTmzZsnSZIknTp1Spo+fbrBbLunZXDRHBISgq+vLwCtW7cmLCxMzxUVFx4eTk5ODi+++CITJ07k4sWL+i4JZ2dnvvjiC937K1eu0KFDBwC6devGiRMn9FUaULK+sLAwjhw5wvjx43n33XdRq9V6q61///688cYbuvcKhcJgtl9ptRnStuvTpw/Lli0DIC4uDjs7O4PZdk/L4AJQrVajUql07xUKBYWFhXqsqDhTU1OmTJnC+vXref/993nnnXf0Xl+/fv0wMvrrbIYkSbo7aiwsLMjMzNRXaUDJ+lq2bMmcOXPYuHEjDRo0YM2aNXqrzcLCApVKhVqt5vXXX2fWrFkGs/1Kq82Qth2AkZERc+fOZdmyZfTr189gtt3TMrgAVKlUZGVl6d5rtdpifzz65urqytChQ5HJZLi6umJtbU1SUpK+yyrm0XMuWVlZWFlZ6bGakvr27YuXl5fu9dWrV/VaT3x8PBMnTmTYsGEMGTLEoLbf32sztG0H8OGHH7J//34WLlxYrCMEfW+7p2FwAejt7c2xY8cAuHjxIh4eHnquqLjAwEBWrVoFQEJCAmq1Gnt7ez1XVVzz5s05ffo0AMeOHaNdu3Z6rqi4KVOmcPnyZQBOnjyJp6en3mpJTk7mxRdfZPbs2YwcORIwnO1XWm2GtO22b9/O119/DYCZmRkymQwvLy+D2HZPy+DuBX54FTgiIgJJklixYgXu7u76LksnPz+f+fPnExcXh0wm45133sHb21vfZREbG8tbb73Fr7/+SmRkJAsXLqSgoAA3NzeWL1+OQqEwmPquXLnCsmXLUCqV2NnZsWzZsmKnPSrT8uXL2bt3L25ubrrP3nvvPZYvX6737VdabbNmzeLjjz82iG2XnZ3N/PnzSU5OprCwkGnTpuHu7m5wv3tPYnABKAiCUFkM7hBYEAShsogAFAShxhIBKAhCjSUCUBCEGksEoCAINZYIQMHgTJgwQd8lCDWECEDB4Jw5c0bfJQg1hOHcYybUOKdPn+brr7/G1NSUW7du0aRJE2xsbAAICAhgy5YtdOrUCS8vL5KSkggMDOS7775j7969aDQaunbtyuzZs5HJZGzfvp0ffvgBrVaLp6enrhumd999lxs3bgAwbtw4Ro0apc9VFgyM2AMU9OrChQssWrSIvXv3EhcXR7du3QDYsmULAGlpaUybNo0dO3Zw8uRJwsLCCAwMZPv27SQkJLBz505u3LjBr7/+yqZNm9ixYwe2trasX7+eCxcukJ6errtl69y5c/pcVcEAiT1AQa8aN25M3bp1AXB3dyc9Pb3EOK1atQKK7n29fPky/v7+AOTm5lKvXj0yMzOJiorS7d0VFBTQvHlzxo4dS2RkJFOmTKFbt27MmTOnktZKqCpEAAp6ZWJionstk8ko7c5MU1NToKgH4kmTJvHCCy8AkJGRgUKhIDAwkAEDBrBgwQKgqBcSjUaDlZUVu3fvJjg4mKNHjzJ8+HB2795t8D2UCJVHHAILBudxfUB26tSJHTt2kJWVRWFhIa+++ir79++nY8eOHDx4kJSUFCRJYsmSJfzwww/88ccfzJ49mx49erBgwQLMzc2Jj4/XwxoJhkrsAQoGp3fv3gwbNoytW7cW+7xXr16Eh4czatQoNBoNvr6+DB8+HJlMxsyZM5k0aRJarZZmzZrx0ksvIZfLOXDgAIMGDcLExIShQ4fSpEkTPa2VYIhEbzCCINRY4hBYEIQaSwSgIAg1lghAQRBqLBGAgiDUWCIABUGosUQACoJQY4kAFAShxvp/+jjJS+q74t0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Testing score is best at 13 trees\n",
    "results = xg.evals_result()\n",
    "fig, ax = plt.subplots(figsize=(8, 4))\n",
    "ax = (pd.DataFrame({'training': results['validation_0']['logloss'],\n",
    "'testing': results['validation_1']['logloss']})\n",
    ".assign(ntrees=lambda adf: range(1, len(adf)+1))\n",
    ".set_index('ntrees')\n",
    ".plot(figsize=(5,4), \n",
    "      ax=ax, title='eval_results with early_stopping'))\n",
    "ax.annotate('Best number \\nof trees (13)', xy=(13, .498), \n",
    "            xytext=(20,.42), arrowprops={'color':'maroon'})\n",
    "ax.set_xlabel('ntrees');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37b4ba82-255c-49eb-8ae0-895136f6e6a9",
   "metadata": {},
   "source": [
    "This is an example of how to visualize the performance of an XGBoost classifier during training using the `.evals_result` method and the matplotlib library in Python.\n",
    "\n",
    "First, the `.evals_result` method is called on the `xg` classifier object to get the evaluation results for each round of training. The results are stored in a variable named `results`.\n",
    "\n",
    "Next, a new `figure` and `axes` are created using the subplots function from the `matplotlib.pyplot` module. The figure size is set to 8 by 4 inches.\n",
    "```python\n",
    "fig, ax = plt.subplots(figsize=(8, 4))\n",
    "```\n",
    "Then, a new DataFrame is created from the results dictionary. The DataFrame has two columns: one for the training log loss and one for the testing log loss. A new column named `ntrees` is added to the DataFrame using the `.assign` method. This column contains the number of trees used in each round of training.\n",
    "\n",
    "The DataFrame is then plotted using the plot method. The ntrees column is set as the index of the DataFrame and used as the x-axis of the plot. The training and testing log loss values are plotted on the y-axis. The figure size is set to 5 by 4 inches and the title of the plot is set to ‘eval_results with early_stopping’.\n",
    "\n",
    "```python\n",
    "ax = (pd.DataFrame({'training': results['validation_0']['logloss'],\n",
    "'testing': results['validation_1']['logloss']})\n",
    ".assign(ntrees=lambda adf: range(1, len(adf)+1))\n",
    ".set_index('ntrees')\n",
    ".plot(figsize=(5,4), \n",
    "      ax=ax, title='eval_results with early_stopping'))\n",
    "```\n",
    "\n",
    "An annotation is added to the plot using the annotate method on the axes object. The annotation text is ‘Best number of trees (13)’ and it is placed at coordinates (13, .498) on the plot. An arrow pointing to these coordinates is added using the arrowprops parameter.\n",
    "\n",
    "Finally, the x-axis label is set to ‘ntrees’ using the set_xlabel method on the axes object.\n",
    "\n",
    "```python\n",
    "ax.annotate('Best number \\nof trees (13)', xy=(13, .498), \n",
    "            xytext=(20,.42), arrowprops={'color':'k'})\n",
    "ax.set_xlabel('ntrees');\n",
    "```\n",
    "This code generates a plot that shows how the log loss on both the training and testing datasets changes as more trees are added to the XGBoost model during training. The plot can be used to determine the optimal number of trees for the model based on its performance on the test dataset.\n",
    "\n",
    "Let us train a model with 13 trees or estimators and see how the model\n",
    "performs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f22011c4-74af-4447-be73-0a1e2fe70a09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tvalidation_0-logloss:0.61534\tvalidation_1-logloss:0.61775\n",
      "[1]\tvalidation_0-logloss:0.57046\tvalidation_1-logloss:0.57623\n",
      "[2]\tvalidation_0-logloss:0.54011\tvalidation_1-logloss:0.55333\n",
      "[3]\tvalidation_0-logloss:0.51965\tvalidation_1-logloss:0.53711\n",
      "[4]\tvalidation_0-logloss:0.50419\tvalidation_1-logloss:0.52511\n",
      "[5]\tvalidation_0-logloss:0.49176\tvalidation_1-logloss:0.51741\n",
      "[6]\tvalidation_0-logloss:0.48159\tvalidation_1-logloss:0.51277\n",
      "[7]\tvalidation_0-logloss:0.47221\tvalidation_1-logloss:0.51040\n",
      "[8]\tvalidation_0-logloss:0.46221\tvalidation_1-logloss:0.50713\n",
      "[9]\tvalidation_0-logloss:0.45700\tvalidation_1-logloss:0.50583\n",
      "[10]\tvalidation_0-logloss:0.45062\tvalidation_1-logloss:0.50430\n",
      "[11]\tvalidation_0-logloss:0.44533\tvalidation_1-logloss:0.50338\n",
      "[12]\tvalidation_0-logloss:0.43736\tvalidation_1-logloss:0.50033\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7558011049723757"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using value from early stopping gives same result\n",
    "xg13 = xgb.XGBClassifier(n_estimators=13)\n",
    "xg13.fit(X_train, y_train, \n",
    "         eval_set=[(X_train, y_train), \n",
    "                   (X_test, y_test)])\n",
    "xg13.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45678b63-b5bc-4216-a0f7-af973079cd48",
   "metadata": {},
   "source": [
    "let’s look at the model that creates exactly 100 trees. The score is worse than only\n",
    "using 13"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f4058e18-de3e-412c-8391-568c771bf77a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7458563535911602"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# No early stopping, uses all estimators\n",
    "xg_no_es = xgb.XGBClassifier()\n",
    "xg_no_es.fit(X_train, y_train)\n",
    "xg_no_es.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f525226a-1f2b-4a59-a5d6-21b73920f53c",
   "metadata": {},
   "source": [
    "The early_stopping_rounds parameter helps to\n",
    "prevent overfitting and improves performance.\n",
    "\n",
    "## 10.3 Different eval_metrics\n",
    "The eval_metric options for XGBoost are metrics used to evaluate the results of an XGBoost\n",
    "model. They provide a way to measure the performance of a model on a dataset. Some of the\n",
    "most common eval_metric options for classification are:\n",
    "1. Log loss ('logloss') - The default measure. Log loss measures a model’s negative log-likelihood performance that predicts a given class’s probability. It penalizes incorrect predictions more heavily than correct ones.\n",
    "2. Area under the curve ('auc') - AUC is used to measure the performance of a binary classification model. It is the Area Under a receiver operating characteristic Curve.\n",
    "3. Accuracy ('error') - Accuracy is the percent of correct predictions. XGBoost uses the error, which is one minus accuracy because it wants to minimize the value.\n",
    "4. You can also create a custom function for evaluation. In the following example, I set the eval_metric to 'error'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0d0042a9-f9ee-4982-899a-3e12e9cd4ad7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tvalidation_0-error:0.24739\tvalidation_1-error:0.27072\n",
      "[1]\tvalidation_0-error:0.24218\tvalidation_1-error:0.26188\n",
      "[2]\tvalidation_0-error:0.23839\tvalidation_1-error:0.24751\n",
      "[3]\tvalidation_0-error:0.23697\tvalidation_1-error:0.25193\n",
      "[4]\tvalidation_0-error:0.23081\tvalidation_1-error:0.24530\n",
      "[5]\tvalidation_0-error:0.22607\tvalidation_1-error:0.24420\n",
      "[6]\tvalidation_0-error:0.22180\tvalidation_1-error:0.24862\n",
      "[7]\tvalidation_0-error:0.21801\tvalidation_1-error:0.24862\n",
      "[8]\tvalidation_0-error:0.21280\tvalidation_1-error:0.25304\n",
      "[9]\tvalidation_0-error:0.21043\tvalidation_1-error:0.25304\n",
      "[10]\tvalidation_0-error:0.20806\tvalidation_1-error:0.24641\n",
      "[11]\tvalidation_0-error:0.20284\tvalidation_1-error:0.25193\n",
      "[12]\tvalidation_0-error:0.20047\tvalidation_1-error:0.24420\n",
      "[13]\tvalidation_0-error:0.19668\tvalidation_1-error:0.24420\n",
      "[14]\tvalidation_0-error:0.19384\tvalidation_1-error:0.24530\n",
      "[15]\tvalidation_0-error:0.18815\tvalidation_1-error:0.24199\n",
      "[16]\tvalidation_0-error:0.18531\tvalidation_1-error:0.24199\n",
      "[17]\tvalidation_0-error:0.18389\tvalidation_1-error:0.23867\n",
      "[18]\tvalidation_0-error:0.18531\tvalidation_1-error:0.23757\n",
      "[19]\tvalidation_0-error:0.18815\tvalidation_1-error:0.23867\n",
      "[20]\tvalidation_0-error:0.18246\tvalidation_1-error:0.24199\n",
      "[21]\tvalidation_0-error:0.17915\tvalidation_1-error:0.24862\n",
      "[22]\tvalidation_0-error:0.17867\tvalidation_1-error:0.24751\n",
      "[23]\tvalidation_0-error:0.17630\tvalidation_1-error:0.24199\n",
      "[24]\tvalidation_0-error:0.17488\tvalidation_1-error:0.24309\n",
      "[25]\tvalidation_0-error:0.17251\tvalidation_1-error:0.24530\n",
      "[26]\tvalidation_0-error:0.17204\tvalidation_1-error:0.24309\n",
      "[27]\tvalidation_0-error:0.16825\tvalidation_1-error:0.24199\n",
      "[28]\tvalidation_0-error:0.16730\tvalidation_1-error:0.24088\n",
      "[29]\tvalidation_0-error:0.16019\tvalidation_1-error:0.24199\n",
      "[30]\tvalidation_0-error:0.15782\tvalidation_1-error:0.24972\n",
      "[31]\tvalidation_0-error:0.15972\tvalidation_1-error:0.24862\n",
      "[32]\tvalidation_0-error:0.15924\tvalidation_1-error:0.24641\n",
      "[33]\tvalidation_0-error:0.15403\tvalidation_1-error:0.25635\n",
      "[34]\tvalidation_0-error:0.15261\tvalidation_1-error:0.25525\n",
      "[35]\tvalidation_0-error:0.15213\tvalidation_1-error:0.25525\n",
      "[36]\tvalidation_0-error:0.15166\tvalidation_1-error:0.25525\n",
      "[37]\tvalidation_0-error:0.14550\tvalidation_1-error:0.25525\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.7624309392265194"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xg_err = xgb.XGBClassifier(early_stopping_rounds=20, eval_metric='error')\n",
    "xg_err.fit(X_train, y_train, \n",
    "           eval_set=[(X_train, y_train), (X_test, y_test)])\n",
    "xg_err.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e3887492-68a4-4711-8f0c-b2b9fc0dc5d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xg_err.best_ntree_limit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aac9d8ae-e763-46dd-869e-4c03bba56734",
   "metadata": {},
   "source": [
    "Now, the best number of trees when we minimize error is 19."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ea834d1-4330-447c-8f54-843ef75c9c14",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}