Continuous_Control.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Reacher.app\"`\n",
    "- **Windows** (x86): `\"path/to/Reacher_Windows_x86/Reacher.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Reacher_Windows_x86_64/Reacher.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Reacher_Linux/Reacher.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Reacher_Linux/Reacher.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Reacher.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Reacher.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_size -> 5.0\n",
      "\t\tgoal_speed -> 1.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "# env = UnityEnvironment(file_name='Reacher20.app', no_graphics=True)\n",
    "env = UnityEnvironment(file_name='Reacher20.app')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, a double-jointed arm can move to target locations. A reward of `+0.1` is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.\n",
    "\n",
    "The observation space consists of `33` variables corresponding to position, rotation, velocity, and angular velocities of the arm.  Each action is a vector with four numbers, corresponding to torque applicable to two joints.  Every entry in the action vector must be a number between `-1` and `1`.\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Size of each action: 4\n",
      "There are 20 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726624e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  \n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 0.019999999552965164\n"
     ]
    }
   ],
   "source": [
    "env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "while True:\n",
    "    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "    dones = env_info.local_done                        # see if episode finished\n",
    "    scores += env_info.rewards                         # update the score (for each agent)\n",
    "    states = next_states                               # roll over states to next time step\n",
    "    if np.any(dones):                                  # exit loop if episode finished\n",
    "#         print(f\"actions: {actions}\")\n",
    "#         print(f\"states: {states}\")\n",
    "#         print(f\"rewards: {rewards}\")\n",
    "#         print(f\"dones: {dones}\")\n",
    "        break\n",
    "print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "%matplotlib inline\n",
    "\n",
    "unity_env = env"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env.action_size: 4\n",
      "env.state_size: 33\n",
      "env.num_agents: 20\n"
     ]
    }
   ],
   "source": [
    "from unity_env_wrapper import EnvMultipleWrapper\n",
    "\n",
    "env = EnvMultipleWrapper(env=unity_env, train_mode=True)\n",
    "print(f\"env.action_size: {env.action_size}\")\n",
    "print(f\"env.state_size: {env.state_size}\")\n",
    "print(f\"env.num_agents: {env.num_agents}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import progressbar as pb\n",
    "def train(env, agent, episodes=2000, max_t=1000, print_every=50):\n",
    "    widget = ['training loop: ', pb.Percentage(), ' ',  pb.Bar(), ' ', pb.ETA()]\n",
    "    timer = pb.ProgressBar(widgets=widget, maxval=episodes).start()\n",
    "\n",
    "    scores = []\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    for i_episode in range(1, episodes+1):\n",
    "        states = env.reset()\n",
    "        agent.reset()\n",
    "        score = np.zeros(env.num_agents)\n",
    "        \n",
    "        for t in range(max_t):\n",
    "            actions = agent.act(states)\n",
    "            next_states, rewards, dones = env.step(actions)\n",
    "            agent.step(states, actions, rewards, next_states, dones)\n",
    "            states = next_states\n",
    "            score += rewards\n",
    "            if np.any(dones):\n",
    "                break\n",
    "            \n",
    "        scores_deque.append(np.mean(score))\n",
    "        scores.append(np.mean(score))\n",
    "        \n",
    "        print(f\"\\rEpisode {i_episode}/{episodes}\\\n",
    "            Average Score: {np.mean(scores_deque):.2f}\\\n",
    "            Score: {np.mean(score):.2f}\\\n",
    "            Max Score: {np.max(scores_deque):.2f}\", end=\"\")\n",
    "        if i_episode % print_every == 0:\n",
    "            timer.update(i_episode)\n",
    "        \n",
    "#         if (scores_deque[0]>30) and (np.mean(scores_deque) > 30):\n",
    "        if np.mean(scores_deque) > 30:\n",
    "            print(f\"\\nEnvironment solved in {i_episode-100} episodes!\\t Average Score: {np.mean(scores_deque):.2f}\")\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth') \n",
    "            break\n",
    "    \n",
    "    timer.finish()\n",
    "    return scores\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "sns.set(style=\"whitegrid\")\n",
    "\n",
    "def plot_scores(scores):\n",
    "    episodes = np.arange(start=1, stop=len(scores)+1)\n",
    "    \n",
    "    data = pd.DataFrame(data=scores, index=episodes, columns=[\"Score\"])\n",
    "\n",
    "    fig = sns.lineplot(data=data)\n",
    "    fig.set_xlabel(\"Episode #\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ddpg_agent import Agent\n",
    "from network import Actor\n",
    "from network import Critic\n",
    "from replay_buffer import ReplayBuffer\n",
    "from noise import OUNoise\n",
    "\n",
    "buffer_size = int(1e5)\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "learning_rate_actor = 1e-4 \n",
    "learning_rate_critic = 1e-3\n",
    "batch_size = 128\n",
    "discount = 0.99\n",
    "seed = 2\n",
    "\n",
    "action_size = env.action_size\n",
    "state_size = env.state_size\n",
    "num_agents = env.num_agents\n",
    "\n",
    "def create_actor(state_dim, action_dim):\n",
    "    return Actor(\n",
    "        state_dim = state_dim, \n",
    "        action_dim = action_dim, \n",
    "        fc1_units = 256, \n",
    "        fc2_units = 128, \n",
    "        seed = seed)\n",
    "#     return Actor(\n",
    "#         state_dim = state_dim, \n",
    "#         action_dim = action_dim, \n",
    "#         hidden_units = (256, 128), \n",
    "#         seed = seed)\n",
    "#     return Actor(\n",
    "#         state_size=state_dim, \n",
    "#         action_size=action_dim, \n",
    "#         seed=seed, \n",
    "#         fc1_units=256, \n",
    "#         fc2_units=128)\n",
    "\n",
    "def create_critic(state_dim, action_dim):\n",
    "    return Critic(\n",
    "        state_dim = state_dim, \n",
    "        action_dim = action_dim, \n",
    "        fc1_units = 256,\n",
    "        fc2_units = 128,\n",
    "        seed = seed)\n",
    "#     return Critic(\n",
    "#         state_dim = state_dim, \n",
    "#         action_dim = action_dim, \n",
    "#         hidden_units = (256, 128),\n",
    "#         seed = seed)\n",
    "#     return Critic(\n",
    "#         state_size=state_dim, \n",
    "#         action_size=action_size, \n",
    "#         seed=seed, \n",
    "#         fcs1_units=256,\n",
    "#         fc2_units=128)\n",
    "\n",
    "agent = Agent(\n",
    "    create_actor = create_actor, \n",
    "    create_critic = create_critic, \n",
    "    replay_buffer = ReplayBuffer(buffer_size = buffer_size, seed = seed),\n",
    "    noise = OUNoise(size = (num_agents, action_size), seed = seed),\n",
    "    state_dim = state_size, \n",
    "    action_dim = action_size,\n",
    "    seed = seed,\n",
    "    lr_actor = learning_rate_actor,\n",
    "    lr_critic = learning_rate_critic,\n",
    "    batch_size = 128,\n",
    "    discount = 0.99)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   0% |                                          | ETA:  --:--:--\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 19/500            Average Score: 2.16            Score: 7.74            Max Score: 7.74"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   4% |#                                          | ETA:  2:31:29\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 39/500            Average Score: 8.07            Score: 24.35            Max Score: 24.35"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   8% |###                                        | ETA:  2:16:50\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 59/500            Average Score: 17.01            Score: 36.44            Max Score: 37.08"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  12% |#####                                      | ETA:  2:13:03\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 79/500            Average Score: 21.47            Score: 36.57            Max Score: 37.08"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  16% |######                                     | ETA:  2:06:09\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 99/500            Average Score: 24.39            Score: 37.22            Max Score: 37.66"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  20% |########                                   | ETA:  1:58:15\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 116/500            Average Score: 29.94            Score: 33.25            Max Score: 37.66"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop: 100% |###########################################| Time: 0:34:37\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Episode 117/500            Average Score: 30.26            Score: 35.66            Max Score: 37.66\n",
      "Environment solved in 17 episodes!\t Average Score: 30.26\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "scores = train(env=env, agent=agent, episodes=500, print_every=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEPCAYAAABIut/fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XdgVFXe//H3zCSZVAKpJCSEEDqBCKEFEBaUgCQBFXVBbD/7roq6z9pY1H0siC67rD667vPsYltdlbXQRBSlCIQiAQIB0gtppPcymXJ/fwQGIoFMwiQzGb6vv8idO3e+JzP5cObcc89VKYqiIIQQwuGobV2AEEKI7iEBL4QQDkoCXgghHJQEvBBCOCgJeCGEcFAS8EII4aAk4IUQwkFJwAshhIOSgBdCCAclAS+EEA5KAl4IIRyUU0+/oMlkoqGhAWdnZ1QqVU+/vBBC9EqKoqDX6/Hw8ECttqxv3uMB39DQQHp6ek+/rBBCOIRhw4bh5eVl0b49HvDOzs5Aa5EuLi4WPy8lJYXIyMjuKqvHSXvsn6O1ydHaA47Xpsu1p6WlhfT0dHOGWqLHA/7csIyLiwtarbZTz+3s/vZO2mP/HK1NjtYecLw2ddSezgxty0lWIYRwUBLwQgjhoHp8iOZyamtrKS0tRa/XX/SYk5MTp06dskFV3aOj9nh4eBASEmLx2XIhhPgluwn42tpaSkpKGDBgAG5ubheNMzU0NODh4WGj6qzvcu0xmUwUFhZSXl5OQEBAD1cmhHAUdtM9LC0tZcCAAbi7u1/18+PVajWBgYHU1NTYuhQhRC9mNwGv1+txc3OzdRl2w9nZGYPBYOsyhBC9mN0EPHRu+o+jk9+FuBrtPlrIAyu3UVzecEXH0RuMPLZ6B+t3ZVqpst7JrgLe3mzdupWbb76ZBQsWkJCQwD//+U9blySEw0o/XcWaTw9zpqKRTXuyLX5eXnEtz/1tD9mF54c0dx0uILe4lq93ZmIwmrqj3F5BAv4SSkpKeP3111m7di0bN27ks88+Y8uWLfz444+2Lk0Ih2A0mqioacJoNFFe3cQr7x3Ap48rE0cF8uPPp2lsvng2XW5xLWs3plDb0AJAQ7ORl947QEpWBe9+mYyiKJhMCl/tzMTd1YnKWh0HUs70dNPsht3MorE3VVVV6PV6mpubgdZpi6tWrUKr1ZKYmMiqVatQFIXg4GD+/Oc/4+7uzsqVK9m3bx8qlYoFCxbw4IMPcuDAAf70pz9hMpkYOnQoL7zwAi+99BKpqakAPPDAA8THx9uyqb3eP9YfR+ui4a75o2xdiriEz7al0ddTy7yYQeZtH3xzkvW7slCrwNlZg1ql4uWHp9KsM/D7t3azI6mAuGnh5v2bWwys+vAghWUN7D5ayBOLx/H57gqqaw0kXDuYTbuz2XWkEHetE/kl9Ty5ZDyfbD3FlsQcpkUF26DVtme3Ab/90Gm2HTxt/tloNKLRaKxy7DmTBjJ7wsDL7jNixAiuu+46rr/+ekaOHMnkyZNJSEggKCiIJUuWsHbtWkaOHMlf/vIXvv76a9RqNcXFxWzcuJGWlhbuvPNOhg0bhpubG7m5uezYsQMvLy9Wr17N6NGjeeGFF1AUhcWLFxMVFUVoaKhV2na1OZJWysbd2ahUMGNcCIOC+ti6JPELLXoj//kxAzethusnDcRJo8ZoNLEzqYCRg3yIGupPVV0zs6JDCevfB0VRGBLal2/2ZjN/6iDz+ah/fXuKwrIGHrgxks17cnj+f/cB8NQd0UyPGsDJnAo+3HwCX283Avq5MWPcAMqrm/jXt6coKK0jJMCyBbociQzRXMZ///d/s337dpYsWUJRURG33XYbH374IYGBgYwcORKA3/3ud9x5550cOHCAm266CY1Gg5ubGwkJCezb1/oBDA8PN6/+lpiYyGeffcbixYtZunQpjY2NZGRk2KyNvYmiKHy5PYOTORUAGIwm/rHhOP193XHTOvHJVse5EO5y6htb2Lg7i39/l8o/N6RwIKX4kvvWNrTYfAw6Na+SFr2RmvoWjqaXAZCSVUF1vY6FMyJYOm8Ej956DaMH+wKtEwzip4WTX1LPsYxyAI5nlbPxp2zip4Wz4NoI/vrkTOKmhTMv2psZ40JQq1U8sHAM5TXNpJ2uYuHMCJw0auZMHoiTRsW3+3Jt1Hrbstse/OwJbXvZPX2h086dO2lsbGT+/PksWrSIRYsWsW7dOjZt2tRmv7q6OhoaGjCZ2v4RKYqC0WgEwNXV1bzdZDLxpz/9iUGDBuHh4UF5eTne3t7d3yA7VlbVREZ+FVPHXv5r9MmcSj745iROGhWP3HINDc168kvqef7eyWQV1vDv71LJyK9iaGi/LtVRU69j77EiZo4LwcPN8hX7etrW/Xl8+M1JAJw0ajb8lMXiOcO5fe7wNrOvDp0q4fWPfiYk0Is/3j/louOUVTXx9n+OUlLZgK7FiJeHC6/9drrV2340vQy1WoWb1oldhwuYMDKQ3cmFuLpoiB7Z/oV8114zgPc2nWD1v5Pw9XaltLKRID8P7o5rHYZzd3Xm4ZvHkpSUZH7O6MG+zBwXQnJGGXMmhQHQz8uVmDHB/HjwNIoCri4aoob6EzXU36pttFfSg78EV1dX/vznP1NQUAC0BnZmZiaRkZFUVlaSmdk6/eqf//wnn376KVOmTGH9+vUYjUaamprYtGkTkydPvui4U6ZM4dNPPwVaL+5asGABxcWX7oFdDT745gSvffgzNfW6y+63dV8u7q5OjB7sy5ufH+GDzScZPzyAiaMCWThjMF7uLnz8bSqZ+dX8Y8Nx/vrZYapqm83P337oNI+t3kFFTVO7x/+/9cd598tjPLDyBzbuzsJgVKzZTKvJKqgmwMed9X9awLqVcVw3MZTPtqWx+uMkMvKraG4xsHVfLi+/dwD/fm7kFdfy7Dt7qGk4f11F3planvqfn0jNqyRiQF8iI/zIKaplcydmr1jqaHoZwwf2Y3pUMPtSiqlv0pN4rIjJo4NwdWm/j+nirOGhm8YwfGA/+nm5Mircl2funICr9vJ90scXj+Odp2fjdsF+N/0qAg83Z7b/fJovt2fwl38fRlGs996m5VVyIrvCasezJrvtwdvalClTePTRR3n44YfNa+Nce+21/P73v2fWrFk8/fTT6PV6Bg4cyBtvvIGLiwu5ubksXLgQvV7PggULmDNnDgcOHGhz3EcffZQ//vGP3HrrrSiKwlNPPcXAgZc/H+DIGpv17D87y+F4VjnTowa0u19NvY49yUXMiwnjvgWR/N/Xx9l1pID7F0aiUqlwd3XmltlDeH/zSQ6nleKkUaNWweHUUp66YwKH00r5YnvrUFhqXhXTxra9qC79dBU/HSlk9oRQyqub+Mf6FDRqCNtdz6DgPiydO4IAH/fu/WVYKLuwhogB3mjUKjRqFY//ehwhAV58tOUkPx0tRK0CkwLRIwJ45q6JZBZU88p7B/jHd40knT6Er7cb3x/Iw8VJzapHphMe3PoNsqFZz/pdWSRcOxh3V+v04usbW8gsqGbJnOGMHerPd/vz+N+vjlHXqOfaay7/jW3GuBBmjAvp1Os5O6lxdmp7n4mhof1YuyIWgG0H8nhr3VHyztRd9nyNyaSwN7mIKWOCcHa6dD/YYDSx6sOfcdU68e4z13Wq1p4gAX8ZN910EzfddNNF2ydNmsRXX3110fYVK1ZctG3y5MltevKenp6sXr3a4dbW6ap9x4tp0RtRq+BYxvmAVxSFtNOtwy0atYoff87HYDQxL2YQTho1v70lioduGoNGc/6Pb/60cMqqmxgU1IdpY4Mpr2lm1YcHWf7uXgBmTwhl+6F8Sisb29SgKArvbTpBX08tD900BjetE0fTy9i6+zjNipbEY0Wk5VXxxmPX0sfD8pvUdIfGZj1F5Q3MmnD+pLxKpeKW2UOZcc0AMguqyS2uxdVFw8IZEWg0asZE+LHyN9N469P9pOVVUV5dRLC/Jy/cN5n+vuc/g4vnDOe/3vyJb/bmcOt1w6xS77HMchQFoob5MyLMh4B+buw8XICHqxPjR/T8OkvnXvNwasllA/5YZhlvfHyI3yway/yp4Zfcb29yEeU1zahVoNMb0TpbZyKItUjAC5vakZRPf193QgK8OJZZZt6+52gRb3x8iPEjAnhqaTRb9+cyerAvYf3P/1FeGO4Ari5OPHTTWPPPnu4u/OWJmXz4zUkGBXszb0oYB1KKKflFwB84cYYT2RX8dtFYc8913PAATPV9iY6O5kR2Bc//byIvr93PK7+ZRmOTniPppYwK920TkD0hp6gWgMEDLj5vE+DjToCPe7vnMiJC+nLXbH+io6MxmhTUqouvlh42sB/RIwL4emcW8dMHtxnm6KqjGWW4aZ0YNrAfarWKmeND+M+PGUyODMLZqefD0NfbjUFBfTicVsrNs4Zecr+001UA/HSk8JIBrygK63/KMn9jyj9Tx5DQvkDrN860vComje7f5jmpuZUMHdjaaekJFo3Bv/nmm8yfP5+4uDjef/99AJ577jliY2NZuHAhCxcuZNu2bd1aqHA8FTVNHMss51fjQ4ka6kdhWYN5fPyHn0/j4eZMcnoZv3ljO8XlDW3mUFvK3dWZ3yyK4oaY1ul2AT7ubQJeURQ+/OYkIQGexE4Oa/cYowf78l+3R5N2uoqHXvuBu1/6jjWfHuGTraldaveVOHe1ZkQ7AW8pjVp1yaUwFscOp66xhS93XDyzy2A0UVXbTH5JHXqDZTNzjqaXMSbCD6ez/xlfP3EgHq5Ol/xd94TxwwM4kV1Jk+7Saz1lnK4G4ER2BaVVje3uczKnksz8ahKujQAgt/j8lbRf78zk5fcOtDkHdCK7gqf+Zzc5RT23iGCH/0UfPHiQ/fv3s3HjRgwGA/Pnz2fmzJmkpKTw8ccfy3K2ost2HS5AUWBWdAjNLa0zjo5lljN2iB9H00u59bphXDPMn9c+/BlvTxemjQ264tcM9HGnsOz8Oidl1U0UlNbz8M1jL/pGcKFpUcH8pjGK7T+fZl7MIA6nlvboH+o52YU1eHu64NPHteOdu2BEmA/TooL5fFs6NfUtPHhjJAWl9XzwzUkOp5aa97txZgT3Lbj8vVBLKhspLm8gfvr5HnCwvyefvRrXLbVbavzwAL7amcnxrHImjep/0ePnhgdHhftwMqeS3UcKWTT74t7+hp+y8HJ35va5w/l2Xy65xXXmx07lVgKQWVDNxLOvkXp2W0C/njuX02HAT5o0iY8++ggnJydKSkowGo1otVqKiop4/vnnKSoqYs6cOTz66KNXfHMKk8kkN7g4y5pn+e3VjqQChg/sR7C/JyaTgpe7M8cyyqmoacaktI6ZB/t78renZ9OkM1jlK32gjwdH0stQFAWVSkVBST0AYf07vgjmhphB3HD2W0SL3shXOzLRG4xdqiunqIaw/n1Qd/KrenZhDYODvbt1MbqnlkbT38edL3dkcji1hLLqJjxcnVk0awj+/dz58efTHDpV0mHAHzrZevLc3qYkjhrsg9ZFw+HU0nYDvry6meo6HYuvH4bRpLDrSMFFAZ9+uor9KcXcet0w3F2dCevvZe7B6w0mMvNbvwFk5p8P+IyCavr7uvfoeRyL0tTZ2Zm33nqLuLg4YmJiMBqNTJkyhZUrV7Ju3ToOHTrEF198cUWFeHh4UFhYSEtLy1URbpejKAoVFRVt5s87msKyenKLa5k5vnWWhFqtIjLCj+TMMrYfOs3IQT4E+3sC4O2ptdpYd4CPG7qW1otuAPJLW3tdnb3KMTzYG6NJIf/sfxCdkXemlmV/3sme5MJOPU9vMHG6pLbd8Xdr0mjU3BM/muX3TEKjUXPzr4bwj+XXc0/8aOKmhTM9agAFpfWXnG7aWquRL3dmMiS0LwMD7esKUmcnDWOH+LX5RnKh9PzW8fehA/sxc1wIOUW15J1pPfehNxj517enePp/dtPXU0v82aUUBgX1Ibe4dZ+cohpazg5hZRac/5aXkV/NkJC+3dau9lh8FmXZsmU88MADPPzww+zbt4933nnH/Nidd97J+vXrue222yx+4ZSUlHa3V1ZWmntXVytFUc4ummSisrLS1uVY7MKLTjrcN7N1mMTFUEpSUusfVD9tE2VVraGRMKlvp45nqfrK1uPvSjxMiJ8LR05U4eaiJjPteLufuUvV0FDTOnV2R2IyVYM795/P4azWtu/Yn4qHqcTi5xVXtmAwKqhaKrv8u+nM81yAh2L7Ac2knTp+frux9T/H9d//zDWXaPvB9HrKqpqYG+XB4cOHu1Srpbryu/B31/FzRQPfbt9PgHfbKaG7j1SjUUPVmSz6qEyoVLD2qwP0cdOQnNNIdYORqHB35o7vS3bGCQA0hjpq6lvYtecgKXmtY/bhgVpO5pSSlJREQ7OR0spGogY6d1ivNT/3HQZ8VlYWLS0tjBw5Ejc3N2JjY9myZQt9+/Zl7ty5QGsgOTl17ox7ZGQkWq3W4v2TkpKIjo7u1GvYs6u9PT+lH8bbs4HYX002B6v/gFq2HNqBi5OapQumdsvVpL7BtXz60w58AgYSPW4A/9m/h0HBWiZMmHDRvpdrk9Gk8I/vNoPWh+joyw9V/NLB3GSgivIGTbvHr2ts4cvtGdx63bA2v4MfDuYBpVw3/Zouratirc+cyaTw6U9bqTV4Eh09/qLHm1sMvLX5B0YP9uW2+Knd2lnrapuCwxrYcugH/vZNCd6eLowe7Mvvl0bj7KThywN7iQhxYvKk1s/EDycSOZJehkoFYyL8eGLWEKJHBLY5nrN3GVsPJ9LHfxD12Xn492vh+pgI/rE+hUFDRpFXXAcU86uY0Ywdcukhq8u1R6fTXbJjfCkdpnJBQQFvvfWW+erLH3/8kYkTJ7Jy5UqmTJmCu7s7n3/+ebvzxYW4lJM5FYwK923zxx8a6EWAjzujw326bamAgH6tFziVnJ0ZUVBax+TRnT95q1GrGBjUp0snWrPOfm3PO1NLY7P+oouKdiTl8+WOTKrrdTyx+HyAZhXW4OqiIdjPs9OvaU1qtYoxQ/xIzihr99v2t4m5VNbqeOqOCXb7TTzIz4OXH4ohs6CG02dq2ZFUwOawHBbMiCCzoIrrLlgm5cEbx5CSVc6k0f3x9W7/rnPnpu/mFteSmlfFyEE+DA1pXTIjq6CGnOJzs5/sbIhm5syZJCcnc+ONN6LRaIiNjeXRRx+lX79+LFmyBIPBQGxsrCx5KyxWUdPEmYpG4qYNbrNdpVLxl8dnoHXpvvnR7q7OeLm7UFLZSG1DCzX1LYQEdC0ww4P6cODEmU4NKRqNprMnWL3IO1NHWl4V44a3nYl2bmz4x5/zmTo22HwiMLuwhvBg706fmO0O1wz1Z29yEQWl9YReMMZe36Tni+0ZjBvmT2SEnw0r7Ng1wwK4Zljr776mvoXPf0hn2MB+NOmMDB14fj2j0ECvNm1sj7enFp8+WpJSSyivbmLEoH6ED+iDWtU69p5bXMMAf48eX+PIopOsy5YtY8uWLWzatInHHnsMgKVLl7Jlyxa+//57fv/733drkcKxnMxuPa8wKtznose8PbWXXJ/EWgJ93SmpaCC/pPUEa0d/vJcyKLgPtQ0tVNVdeg2dI2mlPPKn7dQ3tY7Z55fW02IwccPUcFSq1mUTLqTTGzmeVcG8mEEMCurD2+uOUlXXzIGU4tYZNN18gtVS52bGHMsoa7P9oy0nqW9sMS8K1lvcEz+KxmY9az5tPV8wbGDne9qDgrxJPrv65chBPri6OBEa6EVmQTWZ+dUMCenaInhXQuYkih53MqcCVxfNFV2scyUC+7lTWtVIQWnrDJiu9+Bb6889e3Vpe3YfLeT0mToOnmidMphV0Dp9buwQPwYGepnnRp9zIquCFr2RyaP788TicdQ2tHDPS9/zyvsH0bpomD3BPu4b0N/XnYB+biRnlpu3peVVsnVfLvHXDiaih2eLXKnwYG+umzCQkspGPFydujQMFnZ26QMXZ415fZ+IkL6cyC6nvKbZfJVrT5KAFz3uRE4FI8J8LnthUXcK9HGnpLKJ0yW1uDipu3zhyaDgc+Oulx6HTzm7ymDisSKg9cIXVxcNwf6ejBjkQ1peJSbT+WnBSWkluDipGTPEj4iQvjx881hmXDOAF+6bzAcvzGXYwJ7vBbZHpVIRNdSfYxll6PRGjEYT73yRjE8fV5bOHWHr8rrkjhtG4OKsYWhovy4Ng51b22ZoaF/zlbtDQ/vSpDOa/93TZC0a0aPqm/TkFteyJNZ2IRDg447BaOJ4ZjkDAjy7PKbt5e6Cn7ereX2YX6qoaaK4vAEPN2cOp5XS2Kwnq6B1HF2jVjFykA/f7c8jv7TOfJLucGopkRF+5kWr5sUM6tISDT1h3PAAth08zeI/fENAP3eKyht49u6JVluJsqf5ervxx/undPlCpPCz/+GPHHR+6PFcr12lan/9oO4mPXjRo1JzK1GU9sffe0rg2WV/c4pqCb3C27gNCvY2X+DyS+fWCL89djh6g4n9KWfILqox/9GPOBsE54ZpSipbh41sscpiV0wdG8yK/zeJhTMi6NfHlblTwpg65sqXk7ClMUP8zEMtnTWwfx8WXDuYOZPOz8A5d1I8JMDLKou3dZb04EWPKCqrR6c3cvDEGTRqFcPDbDfUEHjBuu4hV3iVZXhwH46klba7ZEFKVgVuWifmTwvnyx2ZfLE9A12L0XzuIdjPAy93F07lVjJ3yiAOp7Ze9DR+eO8IeI1axeTIICZH9u5QtxaNWsUDN45ps03rrGH88ACLlsLoDhLwotv958d0Ptpy/n6p52YY2MqFN+4IDbyyOeXhQa1LFmQX1jA8rO23kpTsckaF++CkUTN1bBCb9+QAmC9XV6lah2mSUkv593epJKWWEODj3uWTvsI+vdjO7RJ7igS86Fa5xbX8+7tUJo/u33qTCgWbzCa4kNZZQz8vLVV1uiseohk33B+ti4bv9ue1CfjqOh35JfXm+wpPGxvM5j05uDhr2gT47AmhZBZU89m2NBQFFswYbLcXB4neRwJedBuj0cSbnx3Gw82Zx267Bm9Py5em6G4BPu7U1OsI9r+yRcw83V2YFR3K9p9Pc3fcKHMbz42/R0b4AjAy3Je+XloCfdzbzB6aFhXMtKhg9AYTVXXN3bYMsLg6yUlW0W2+2plJZkEND9881q7CHVpPfg0K9rbKEsTx08NpMZj4/kCeeVtKdjlaF415OEajVvHMnRN48BdjtOc4n52u6WSjqaPCMUkPXnQLvcHEZ9vSiRkTdMkbadvS/QsjMVh4V6KOhPXvQ9RQP7bszeGmXw3BSaMmJauCkWE+bQLb3i/dF45HuguiW5RVNZqvyLRHWmeNVdcFSZg+mPKaZj7blsaKv+8lt7j2ojVmhOhp0oMX3aK4onXN856+KbWtTBjVn0Afdz7flo63pwv3L4wkblr7N2sWoqdIwItucab8XMD33P0nbUmjVvHE4nFkF9YwZ3KYTS5qEeKX5FMousWZykZcnDVX1ayQyAg/GWcXdkXG4EW3KC5voL+vu8zpFsKGJOBFtzhT0UB/n6tj/F0IeyUBL6xOURTOVDbS3+/qGH8Xwl5JwAurq67ToWsxSg9eCBuzKODffPNN5s+fT1xcHO+//z4AiYmJJCQkEBsby5o1a7q1SNG7nKlovaF1kJ8EvBC21OEsmoMHD7J//342btyIwWBg/vz5xMTEsHz5cv71r38RFBTEQw89xK5du5g5c2ZP1Czs3Pk58DJEI4QtddiDnzRpEh999BFOTk5UVFRgNBqpra0lLCyM0NBQnJycSEhIYOvWrT1Rr+gFzlQ0oFK1XXddCNHzLBqicXZ25q233iIuLo6YmBhKS0vx9/c3Px4QEEBJSUm3FSl6lzMVDfh6u1llIS8hRNepFEVROt6tVVNTEw8//DATJ04kNzeX1atXA63j8WvXrmXt2rUdHkOn05GSktL1ioXdW/t9KRq1inuu9+94ZyFEp0RGRqLVWrY6a4dj8FlZWbS0tDBy5Ejc3NyIjY1l69ataDTne2elpaUEBHRuYaXOFAmQlJREdHR0p17Dnjlye97ctJUJIwOJjh5n46qujCO/R47C0dp0ufZ0pXPc4RBNQUEBK1asoKWlhZaWFn788UcWL15MTk4OeXl5GI1GNm/ezIwZMzr1wsIxNesMVNXprppFxoSwZx324GfOnElycjI33ngjGo2G2NhY4uLi8PHx4bHHHkOn0zFz5kzmzZvXE/UKO3em8uwUSQl4IWzOosXGli1bxrJly9psi4mJYePGjd1SlOi9zpydIhkoUySFsDm5klVY1bmAl4uchLA9CXhhVcXlDXi4OePl7mLrUoS46knAC6sqKmuQ3rsQdkICXlhVQVk9oQGeti5DCIEEvLCiJp2B8uomBkjAC2EXJOCF1RSW1QMQEuBl40qEECABL6yooPRcwEsPXgh7IAEvrKagtA61CoLlJKsQdkECXlhNQWk9gb4esoqkEHZCAl5YTWFpvQzPCGFHJOCFVZhMCkVl9Qzwl4AXwl5IwAurqGk00mIwyQwaIeyIBLywivJaPSAzaISwJxLwwirKaw2ABLwQ9kQCXlhFea0BL3cXvD0tv0uXEKJ7ScALqyiv1UvvXQg7IwEvrKK81iABL4SdkYAXV6y+sYWGZpMEvBB2xqJb9r399tt8++23QOs9Wp9++mmee+45kpKScHNzA+DRRx9lzpw53VepsFu5xbUAMgdeCDvTYcAnJiayZ88evv76a1QqFffffz/btm0jJSWFjz/+mICAgJ6oU9ix7w7k4eKkIjLCz9alCCEu0OEQjb+/P88++ywuLi44OzsTERFBUVERRUVFPP/88yQkJPDWW29hMpl6ol5hZypqmth9pJBxgz3wcHO2dTlCiAuoFEVRLN05NzeXxYsX8+9//5s///nPvPTSS7i7u/PQQw8RHx/Pbbfd1uExdDodKSkpV1S0sB8/Jtew+0QdyxL64+Nl0YifEOIKREZGotVaOB1ZsVB6eroya9Ys5auvvrrose+//1757W9/a9FxmpublUOHDinNzc2WvrSiKIpy6NChTu1v7xyhPU06vbJkxTfKK+/td4j2/JKjtcnR2qMojtemy7WnK9kC2YwfAAAdRElEQVRp0SyapKQk7rnnHv7rv/6Lm266ibS0NL777rsL/5PAyUl6b1ebHUkF1DXqWTgjwtalCCHa0WHAFxcX88gjj7B69Wri4uKA1kBfuXIlNTU16PV6Pv/8c5lBc5UwmRSSM8p454tkPtx8gogQb0YP9rV1WUKIdnTY7V67di06nY5Vq1aZty1evJgHH3yQJUuWYDAYiI2NJT4+vlsLFfZh4+5s1m5MwdVFw8RR/fn1nGGoVCpblyWEaEeHAb9ixQpWrFjR7mNLly61ekHCvuUU1dDPS8v/Lb8eVxcZlhPCnsmVrKJTSiobCfLzkHAXoheQgBedUlLZSKCPu63LEEJYQAJeWExvMFFR00Sgj4etSxFCWEACXlisrLoRRUF68EL0EhLwwmIlFY0ABPpKwAvRG0jAC4uVVJ4NeOnBC9ErSMALi5VUNuKkUeHr7WbrUoQQFpCAFxYrqWzEv687GrVc2CREbyABLyxWUtkgwzNC9CIS8MJiJZWNcoJViF5EAl5YpElnoKa+RXrwQvQiEvDCIqUyg0aIXkcCXlhEpkgK0ftIwAuLnKlsAJBlCoToRSTghUVKKhvRumjw9nSxdSlCCAtJwAuLlFS0riIpN/cQoveQgBcWkWWCheh9LAr4t99+m7i4OOLi4njjjTcASExMJCEhgdjYWNasWdOtRQrbUhRFAl6IXqjDgE9MTGTPnj18/fXXrF+/nhMnTrB582aWL1/O3/72N7Zs2UJKSgq7du3qiXpFD1MUhb3HimjSGeQEqxC9TIf3XfP39+fZZ5/FxaX15FpERAS5ubmEhYURGhoKQEJCAlu3bmXmzJndW63oUemnq/jfr4+RfrqaAf6eTBsbbOuShBCd0GHADx061Pzv3NxctmzZwp133om/v795e0BAACUlJd1TobCJJp2Bl987gFqlYtlt1zB7QigajZyyEaI3sfjOyRkZGTz00EM888wzODk5kZOT0+bxzs6uSElJ6dT+AElJSZ1+jj2z5/bsPF5LdZ2O+2L98XEq5+jR8g6fY8/t6SpHa5OjtQccr03WbI9FAZ+UlMSyZctYvnw5cXFxHDx4kPLy83/wpaWlBAQEdOqFIyMj0Wq1Fu+flJREdHR0p17Dntlze6rqmln1xQ9MGxvMjXMnWvQce25PVzlamxytPeB4bbpce3Q6Xac7xh1+5y4uLuaRRx5h9erVxMXFARAVFUVOTg55eXkYjUY2b97MjBkzOvXCwn59+n0aeoOJu+aPtHUpQogr0GEPfu3ateh0OlatWmXetnjxYlatWsVjjz2GTqdj5syZzJs3r1sLFT0jv6SO7/bnMT9mEMH+nrYuRwhxBToM+BUrVrBixYp2H9u4caPVCxK2ozcYWf1JEu5aJ349Z7ityxFCXCGZFiHM3tt4guzCGp5cMp6+XpafHxFC2CcJ+KtMTlEN/++l78gvqWuzfe+xIjbvzWHhjAgmje5vo+qEENYkAX+V+WpHJuU1zRw4cca8TW8w8va6owwN7cvdcaNsWJ0Qwpok4K8iFTVN7D5aCMDxzPPTXE/mVFLfpGfxnOE4O8lHQghHIX/NV5Fv9uZgUhSiRwRwMqcCg9EEwJG0UjRqFZERvjauUAhhTRLwV4nmFgNb9+UyJTKIOZPCaG4xkplfDcCRtDJGhvvg7ups2yKFEFYlAX+V2JFUQF2jngXXDjb31I9nlVNV10x2UQ3jh3fuSmQhhP2zeC0a0XvpDUbW78wkIsSb0YN9UalUhPX34lhmOX593QAYN0wCXghHIz34q8CXOzIpKm/gjnkjzYvCjYnw41RuJT+fLKGPhwuDB3jbuEohhLVJwDu4orJ61v2QzvSoYCaMDDRvHzPED12Lkb3JhVwzzB+1Wu61KoSjkYB3YIqi8Lcvk3F2UvPAjWPaPDZ6cOs4vElBxt+FcFAS8A4s8VgxyRnl3HXDSHz6uLZ5zNtTy6CgPgBcM8y/vacLIXo5OcnqwHYdKcCvrxvzpoa3+/icyQNJy63C19uthysTQvQECXgHpTeYOJpexoxxA9BcYnx9wbURcG0PFyaE6DEyROOgUvMqadIZiB4R2PHOQgiHJAHvoJJOlaBRq4ga6mfrUoQQNiIB76AOp5UyKtxXlh8Q4iomAe+AKmqayCmqZfwImf4oxNXM4oCvr68nPj6egoICAJ577jliY2NZuHAhCxcuZNu2bd1WpOicI2mlAERLwAtxVbNoFk1ycjIrVqwgNzfXvC0lJYWPP/6YgAAJEXtzKLUUnz6u5nnuQoirk0U9+HXr1vHiiy+aw7yxsZGioiKef/55EhISeOuttzCZTN1aqLCM0dg6PTJ6RIB53RkhxNXJooB/9dVXmTBhgvnniooKpkyZwsqVK1m3bh2HDh3iiy++6LYiheXOVDbS0KRnVLjcvEOIq51KURTF0p1nz57NRx99REhISJvt27ZtY/369bzzzjsdHkOn05GSktL5SoVF8kp1vP9DGXfM8mNIkGvHTxBC9CqRkZFotVqL9u3SlaxpaWnk5uYyd+5coHVRKyenzh2qM0UCJCUlER0d3anXsGfd1Z7mY0VAGZPGRxIe3HNLADva+wOO1yZHaw84Xpsu156udI67NE1SURRWrlxJTU0Ner2ezz//nDlz5nTlUMLKaup1QOtiYkKIq1uXevAjRozgwQcfZMmSJRgMBmJjY4mPj7d2baILquvOBryHi40rEULYWqcCfvv27eZ/L126lKVLl1q9IHFlqut1eLm7oNHINWxCXO0kBRxMdZ2Ovl4yPCOEkIB3ODX1OvrK+LsQAgl4h1NTr8PbU8bfhRAS8A5HhmiEEOdIwDsQvcFIQ7NBhmiEEIAEvEOprmsBkB68EAKQgHcocpGTEOJCEvAOpPpswEsPXggBEvAO5dxVrDIGL4QACXiHYu7BS8ALIZCAdyg19Tq0LhpctV1aYkgI4WAk4B1IdZ1cxSqEOE8C3oFUyzIFQogLSMA7ELmKVQhxIQl4B9K6Do0EvBCilQS8gzCZFGoaWqQHL4Qwk4B3EHWNLZhMiqwkKYQwk4B3EDUyB14I8QsWBXx9fT3x8fEUFBQAkJiYSEJCArGxsaxZs6ZbCxSWkWUKhBC/1GHAJycns2TJEnJzcwFobm5m+fLl/O1vf2PLli2kpKSwa9eu7q5TdKDm7EqScpJVCHFOhwG/bt06XnzxRQICAgA4duwYYWFhhIaG4uTkREJCAlu3bu32QsXlVdU3AzJEI4Q4r8Nr2l999dU2P5eWluLv72/+OSAggJKSkk6/cEpKSqefk5SU1Onn2DNrtic1owaVCtJTj6NWqax23M5wtPcHHK9NjtYecLw2WbM9nV60RFGUi7apuhAokZGRaLWW9zaTkpKIjo7u9OvYK2u3JzHrKH09W5g4YYLVjtkZjvb+gOO1ydHaA47Xpsu1R6fTdbpj3OlZNIGBgZSXl5t/Li0tNQ/fCNuRi5yEEL/U6YCPiooiJyeHvLw8jEYjmzdvZsaMGd1Rm7CQoihkFdYQ5Odh61KEEHak00M0Wq2WVatW8dhjj6HT6Zg5cybz5s3rjtqEhQpK6ymvbuLX1w+zdSlCCDticcBv377d/O+YmBg2btzYLQWJzjucVgrA+OEyVCaEOE+uZHUAh1NLGeDvSYCPu61LEULYEQn4Xk6nN5KSVU70COm9CyHakoDv5U5kV9BiMDFOhmeEEL8gAd/LHU4txdlJTWSEr61LEULYGQn4Xu5wWimjB/vi6iI32hZCtCUB30Na9EZ+Pnmm3SuBu6qsqon8kjqZPSOEaJcEfA/ZkZTPS2sPsPdYkdWOeeBEMQDj5QSrEKIdEvA9JC2vCoCPtpzCYDRd8fEURWHrvlyGhHgT1r/PFR9PCOF4JOB7SEZ+Nd6eLhSXN7DtQN4VH+9UbiV5Z+qYFxNuheqEEI5IAr4HNLcYOF1Sx7wpgxgV7sOn36fRYrh0L377odN8tSOTsqqmS+7z7b5c3F2dmDluQDdULIRwBBLwPSC7sAaTSWFoaF/uiRtNVZ2Ofan17e7b2Kzn7f8k8/7mE9z7yvcs/9teqmqb2+xTU69jb3IRs6NDcdXK7BkhRPsk4HtARn41AEMH9mNkuA9TxwbxU0ot2YU1F+2blFqK3mDiySXjuWPeCE7mVLDuh/Q2+2w/lI/eYGJezKCeKF8I0UtJwPeAjNPV+Hq74tPHFYDfLorCXavm9Y9+prFZ32bffceL6eupZeb4EH49ZzizJ4Ty3YE8Ks/24vUGI98m5jIq3IewIDm5KoS4NAn4HpCRX8XQ0L7mn709tSya5suZigbe+SLZPDe+RW/k0KkzTI7sj0bdepesW68bhtGk8PXOTAD+sSGF4ooGbr1OlgYWQlyeBHw3q2/SU1TewNDQfm22DwrQcvu8Efx0pJDvD5wG4GhGGU06I1PHBJv3C/LzYOa4AXy7L5eNu7P4NjGXm341hAkjA3uyGUKIXkgCvptl5rfOf7+wB3/OrbOHcc1Qf/5v/XHyztSy71gxHq5OjBni13a/64bRojfyj/UpjAr34a75I3ukdiFE7yYB383MJ1jbCXi1WsXvbh+Pu9aJ1z86xIETxUwc3R9np7ZvS2igF7OiQ+nnpeXpOyfgpJG3TQjRsSuaY3fXXXdRUVGBk1PrYV566SWioqKsUpijyMivJsjPA093l3Yf79fHlSdvH8+L/7cPgKljgtrdb9mvx6E3GGVRMSGExbqcFoqikJ2dzc6dO80BL9pSFIX001WMHnz5pXzHDw9g8ZzhbN2fe8l13TVqFRoJdyFEJ3Q5MbKzs1GpVDzwwANUVFRw2223cccdd1iztl4vOaOMippmxg3reDGwpfNGsDh2uHn2jBBCXKkuB3xtbS0xMTH88Y9/pLm5mbvuuovw8HCmTZtmzfp6ta93ZtHPS8vM8ZYtJyDhLoSwJpVipQXKP/jgA4qKili+fPll99PpdKSkpFjjJe3amaoW/v5tKbOj+jBjtFyQJISwjsjISLRarUX7drkHf+jQIfR6PTExMUDreHNnxuI7UyRAUlIS0dHRna7TVtZ8ehiti4b7b5mGVzsnWHtbezriaO0Bx2uTo7UHHK9Nl2tPVzrHXZ5vV1dXxxtvvIFOp6O+vp6vv/6aOXPmdPVwDqWipomfjhQwZ9LAdsNdCCF6Qpd78LNmzSI5OZkbb7wRk8nE7bffzrhx46xZW6+18adsTCaFhTMibF2KEOIqdkXz7p544gmeeOIJa9XiEKpqm9m8N4cZ40Lo7+th63KEEFcxuSTSytb9kI7RaGLJ3OG2LkUIcZWTgLei0spGtu7P5fpJAwn287R1OUKIq5wEvBV9+n0aKpWKxXOk9y6EsD0JeCvJL6lj+6HTzJ8ajl9fN1uXI4QQEvDW8t6mE7hqnbj1uqG2LkUIIQAJeKtISi3h0KkSFs8Zjren5RdvCSFEd5KAv0JGo4m1G1MI8vMgfvpgW5cjhBBmEvBXaOu+XPJL6rk3YfRFN+oQQghbkkS6ArUNLXzyXSpjh/gxeXR/W5cjhBBtSMBfgY+/PUVDs4EHbhyDSiVL/Qoh7IvcIugCpZWNeLo74+7qDIDBaGLznmyOpJVRVt1EXUML86eF8+vrh5FdVMPW/bkkTB/MoCBZDlgIYX8k4M86kV3Bir8nonXREDctnNHhvry/+QS5xbWEB/chJMCTFr2Rf3+XSlZBNdX1Orw9tCyZO8LWpQshRLsk4IGisnpeff8AgT5uhAX14T8/prNOAT9vV/7w/yYxJbL1RtiKorBpTzZrN57AZFJ4/Nfj8HRztnH1QgjRvqs24E0mhYZmPRU1zaz84CAqlYoX748hyM+DgtI6UnMrmTo22DxcA6BSqVhwbQQRA/qSkl3O7AmhNmyBEEJc3lUX8CaTwua92Xz87SmadEYAnJ3UrPzNNIL8Wpf3DQnwIiTA65LHGD3Yl9GDfXukXiGE6CqHDvjGZj17k4vIKa5lgJ8HQX6efLUzg+SMcsaPCGD88AC8PbUMCfG+bKALIURv1CsDPv10FZ98l0p5dROKAhq1Cv9+bgT5euCqdaK+sYWKmmaOpJXSYjDh7KRGbzABoHXR8MgtUcydEiZTG4UQDq1XBXxNvY5/bkxhZ1IBfT21jAz3Qa1SYTCaKKls5FhmOXq9EU93F7zcXbh+0kBmTQhl+MB+VNY2c/pMHSEBXvj3k9UehRCO74oCftOmTbz77rvo9Xruueceli5daq262rXzcAF7k4u49bqh3DJ7aJsToNA6y0VRQK2+uGfu6+2Gr7cEuxDi6tHlgC8pKWHNmjV89dVXuLi4sHjxYiZPnsyQIUOsWV8b8dPCmTs5DFdt+2WrVCpk1EUIIVp1eamCxMREpkyZQt++fXF3d2fu3Lls3brVmrVdRKNRXzLchRBCtNXlgC8tLcXf39/8c0BAACUlJVYpSgghxJXrcndYUZSLtnVmVkpKSkqnXzMpKanTz7Fn0h7752htcrT2gOO1yZrt6XLABwYGcujQIfPPpaWlBAQEWPz8yMhItFrL736UlJREdHR0p2q0Z9Ie++dobXK09oDjtely7dHpdJ3uGHd5iGbq1Kns27ePyspKmpqa+P7775kxY0ZXDyeEEMLKrqgH/+STT3LXXXeh1+u55ZZbGDt2rDVrE0IIcQWuaEpKQkICCQkJ1qpFCCGEFfX4nMNzJ2dbWlo6/VydTmftcmxK2mP/HK1NjtYecLw2Xao95zKzvQkul6JSOrO3FdTV1ZGent6TLymEEA5j2LBheHlZtjhijwe8yWSioaEBZ2dnWexLCCEspCgKer0eDw8P1GrL5sf0eMALIYToGV2eJimEEMK+ScALIYSDkoAXQggHJQEvhBAOSgJeCCEclAS8EEI4KAl4IYRwUHYf8Js2bWL+/PnMmTOHTz75xNbldMnbb79NXFwccXFxvPHGG0DrHbESEhKIjY1lzZo1Nq6wa15//XWeffZZAE6dOsWiRYuYO3cuf/jDHzAYDDaurnO2b9/OzTffzLx583jllVeA3v8ebdiwwfy5e/3114He+T7V19cTHx9PQUEBcOn3pbe07Zft+fzzz4mPjychIYHnnnvOvCSBVdqj2LEzZ84os2bNUqqqqpSGhgYlISFBycjIsHVZnbJ3717l17/+taLT6ZSWlhblrrvuUjZt2qTMnDlTOX36tKLX65V7771X2blzp61L7ZTExERl8uTJyjPPPKMoiqLExcUpR44cURRFUZ577jnlk08+sWV5nXL69Gll+vTpSnFxsdLS0qIsWbJE2blzZ69+jxobG5WJEycqFRUVil6vV2655RZl7969ve59Onr0qBIfH6+MHj1ayc/PV5qami75vvSGtv2yPdnZ2cqcOXOUuro6xWQyKU8//bTy/vvvK4pinfbYdQ/eFvd9tTZ/f3+effZZXFxccHZ2JiIigtzcXMLCwggNDcXJyYmEhIRe1a7q6mrWrFnDww8/DEBhYSHNzc1cc801ANx88829qj3btm1j/vz59O/fH2dnZ9asWYObm1uvfo+MRiMmk4mmpiYMBgMGgwEnJ6de9z6tW7eOF1980XwzoWPHjrX7vvSWz+Av2+Pi4sIf//hHPD09UalUDBs2jKKiIqu1x67vYN3efV+PHTtmw4o6b+jQoeZ/5+bmsmXLFu68885efT/bF154gSeffJLi4mLg4vfJ39+/V7UnLy8PZ2dn7rvvPsrKypg1axZDhw7t1e+Rp6cnjz/+ODfccAOurq5MmjQJZ2fnXvc+vfrqq21+vtS9oHvLZ/CX7RkwYAADBgwAoLKykk8++YTXXnvNau2x6x68coX3fbUnGRkZ3HvvvTzzzDMMHDjwosd7S7v+85//EBQURExMjHlbb3+fjEYj+/bt409/+hPr1q3j+PHj5vHRC/WmNqWmpvLll1+yY8cO9uzZg1qtZu/evRft15vaBJf+rPX2z2BJSQl33303ixYtYvLkyVZrj1334K/0vq/2IikpiWXLlrF8+XLi4uI4ePAg5eXl5sd7U7u2bNlCWVkZCxcupKamhsbGRlQqVZv2lJWV9Zr2APj5+RETE4OPjw8A1113HVu3bkWj0Zj36U3vEcCePXuIiYnB19cXaP2Kv3bt2l79PkFrJrT3t/PL7b2pbVlZWTzwwAPccccd3HvvvcDF7exqe+y6B+8I930tLi7mkUceYfXq1cTFxQEQFRVFTk4OeXl5GI1GNm/e3Gva9f7777N582Y2bNjAsmXLmD17Nq+99hpardZ8N/j169f3mvYAzJo1iz179lBbW4vRaGT37t3Mmzev175HACNGjCAxMZHGxkYURWH79u1MmjSpV79PcOm/nQEDBvTKttXX13Pffffx+OOPm8MdsFp77L4H39vv+7p27Vp0Oh2rVq0yb1u8eDGrVq3iscceQ6fTMXPmTObNm2fDKq/c6tWrWbFiBQ0NDYwaNYq77rrL1iVZLCoqivvvv5/bb78dvV7PtGnTWLJkCYMHD+6179H06dM5efIkN998M87OzowZM4YHH3yQOXPm9Nr3CUCr1V7yb6c3fga/+OILysvLee+993jvvfcAmD17No8//rhV2iPrwQshhIOy6yEaIYQQXScBL4QQDkoCXgghHJQEvBBCOCgJeCGEcFB2PU1SiAsNHz6cYcOGoVa37Ze88847hISEXPJ5b775JmFhYdx4441XXMPatWvJyMhoM+3VEmVlZTz66KN8/vnnrF+/npycHJ588skrrkeIy5GAF73Khx9+aL7i1FKPP/54N1VjuYMHDzJx4kQADhw4QEJCgo0rElcDCXjhEA4cOMAbb7xBYGAg+fn5uLq6smrVKiIiInj22WcZOnQo9913H2+99Rbbtm3D2dmZfv368dprrxEQEMChQ4d44403aGpqwtnZmSeeeIIZM2ag1+t55ZVXSExMxNfXF19fX7y8vACoq6vj1VdfJT09Hb1eT0xMDE8//TROTuf/rI4fP86KFSsoLCzE19eX3bt3k5WVxcmTJ/n73/9OUFCQrX5l4iogAS96lbvvvrvNEE1ISAjvvPMOACdPnuS5555jwoQJfPrppzz11FN89dVX5n2Li4v58MMP2bdvHy4uLrz33nscO3aM6Oholi1bxrvvvktUVBQZGRnccccdfPHFF2zfvp3c3Fy++eYbDAYDd9xxhzngV65cyejRo1m1ahVGo5Fnn32W999/nwceeMD8mmPGjGHDhg3ceuutvPPOO7i5uXHHHXewYcOGHvqNiauZBLzoVS43RDNixAgmTJgAwKJFi3jppZeoqqoyPx4YGMiIESO46aabmDFjBjNmzCAmJoZdu3YxcOBAoqKigNYlnsePH8/BgwfZt28f8fHxuLi44OLiQkJCAmlpaQDs3LmT48eP88UXXwDQ3Nzcbl0mk4nKykoCAgI4ePAgI0eOtNrvQ4jLkYAXDuPC1R+hdWnZC7ep1Wo+/vhjjh8/zr59+1i5ciWTJ09m2rRpFx1LUZR2b5F24fFMJhNvvvkmERERANTW1l60pOvx48d56qmnqKqqYuHChVRWVqJSqVi4cKEM0YhuJ9MkhcNITU0lNTUVaL3P5fjx4+nTp0+bx+Pj44mIiOChhx7innvuIS0tzbxC4bmbyWRkZPDzzz8zadIkrr32WtavX49Op0On07Flyxbz8aZPn84HH3yAoii0tLTwm9/8ho8//rhNTWPGjOHuu+/mwQcfZMOGDYwZM4Z3332XDRs2SLiLbic9eNGr/HIMHuB3v/sdrq6u+Pn58de//pXCwkJ8fHzMNzg/Z8SIEdxwww0sWrQId3d3XF1dWbFiBT4+Prz55pu8/PLLNDc3o1KpeO211wgPD2fgwIGcPn2a+Ph4+vbtS1hYmPl4f/jDH3j11VdJSEhAr9czdepU7r///otqPnToELfffjuKopCRkSFDNKLHyGqSwiEcOHCAl19+mc2bN9u6FCHshgzRCCGEg5IevBBCOCjpwQshhIOSgBdCCAclAS+EEA5KAl4IIRyUBLwQQjgoCXghhHBQ/x9NjYrML52x7QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_scores(scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   0% |                                          | ETA:  --:--:--\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 19/500            Average Score: 0.89            Score: 3.35            Max Score: 3.35"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   4% |#                                          | ETA:  2:06:30\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 39/500            Average Score: 8.99            Score: 27.05            Max Score: 28.79"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   8% |###                                        | ETA:  2:00:11\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 59/500            Average Score: 17.54            Score: 33.40            Max Score: 37.67"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  12% |#####                                      | ETA:  1:58:25\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 79/500            Average Score: 21.84            Score: 35.69            Max Score: 37.67"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  16% |######                                     | ETA:  1:51:59\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 99/500            Average Score: 24.53            Score: 32.78            Max Score: 37.67"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  20% |########                                   | ETA:  1:46:34\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 116/500            Average Score: 29.93            Score: 33.93            Max Score: 37.67"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop: 100% |###########################################| Time: 0:31:17\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Episode 117/500            Average Score: 30.26            Score: 34.19            Max Score: 37.67\n",
      "Environment solved in 17 episodes!\t Average Score: 30.26\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "with_batch_normalization_and_add_action_in_second_hidden_layer = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   0% |                                          | ETA:  --:--:--\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 9/100            Average Score: 0.58            Score: 1.14            Max Score: 1.14"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   9% |###                                        | ETA:  0:48:02\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 19/100            Average Score: 1.08            Score: 2.53            Max Score: 2.53"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  19% |########                                   | ETA:  0:41:45\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 29/100            Average Score: 2.49            Score: 10.17            Max Score: 10.17"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  29% |############                               | ETA:  0:36:53\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 39/100            Average Score: 4.94            Score: 14.93            Max Score: 16.09"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  39% |################                           | ETA:  0:31:36\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 49/100            Average Score: 8.29            Score: 23.98            Max Score: 24.44"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  49% |#####################                      | ETA:  0:27:04\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 59/100            Average Score: 12.15            Score: 32.73            Max Score: 33.01"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  59% |#########################                  | ETA:  0:21:50\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 69/100            Average Score: 15.42            Score: 37.16            Max Score: 37.16"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  69% |#############################              | ETA:  0:16:27\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 79/100            Average Score: 18.10            Score: 36.91            Max Score: 37.58"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  79% |#################################          | ETA:  0:11:10\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 89/100            Average Score: 20.01            Score: 33.21            Max Score: 37.58"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  89% |######################################     | ETA:  0:05:51\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 99/100            Average Score: 21.52            Score: 36.61            Max Score: 37.58"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop: 100% |###########################################| Time: 0:53:46\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Episode 100/100            Average Score: 21.65            Score: 35.39            Max Score: 37.58"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "with_batch_normalization = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   0% |                                          | ETA:  --:--:--\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 9/100            Average Score: 1.05            Score: 1.65            Max Score: 1.65"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  10% |####                                       | ETA:  0:44:54\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 19/100            Average Score: 1.73            Score: 3.76            Max Score: 3.76"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  20% |########                                   | ETA:  0:42:11\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 29/100            Average Score: 2.97            Score: 6.51            Max Score: 6.79"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  30% |############                               | ETA:  0:37:18\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 39/100            Average Score: 3.74            Score: 6.55            Max Score: 6.79"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  40% |#################                          | ETA:  0:32:24\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 49/100            Average Score: 4.47            Score: 7.38            Max Score: 9.23"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  50% |#####################                      | ETA:  0:26:29\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 59/100            Average Score: 5.18            Score: 7.41            Max Score: 9.89"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  60% |#########################                  | ETA:  0:20:53\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 69/100            Average Score: 5.44            Score: 5.84            Max Score: 9.89"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  70% |##############################             | ETA:  0:15:34\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 79/100            Average Score: 5.52            Score: 6.32            Max Score: 9.89"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  80% |##################################         | ETA:  0:10:20\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 89/100            Average Score: 5.60            Score: 7.02            Max Score: 9.89"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  90% |######################################     | ETA:  0:05:09\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 99/100            Average Score: 5.81            Score: 7.36            Max Score: 9.89"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop: 100% |###########################################| Time: 0:51:28\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Episode 100/100            Average Score: 5.82            Score: 7.37            Max Score: 9.89"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "without_batch_normalization = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Watch Smart Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_size -> 5.0\n",
      "\t\tgoal_speed -> 1.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from ddpg_agent import Agent\n",
    "from network import Actor\n",
    "from network import Critic\n",
    "from replay_buffer import ReplayBuffer\n",
    "from noise import OUNoise\n",
    "from unityagents import UnityEnvironment\n",
    "from unity_env_wrapper import EnvMultipleWrapper\n",
    "unity_env = UnityEnvironment(file_name='Reacher20.app')\n",
    "\n",
    "buffer_size = int(1e5)\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "learning_rate_actor = 1e-4 \n",
    "learning_rate_critic = 1e-3\n",
    "batch_size = 128\n",
    "discount = 0.99\n",
    "seed = 2\n",
    "\n",
    "action_size = env.action_size\n",
    "state_size = env.state_size\n",
    "num_agents = env.num_agents\n",
    "\n",
    "def create_actor(state_dim, action_dim):\n",
    "    return Actor(\n",
    "        state_dim = state_dim, \n",
    "        action_dim = action_dim, \n",
    "        fc1_units = 256, \n",
    "        fc2_units = 128, \n",
    "        seed = seed)\n",
    "\n",
    "def create_critic(state_dim, action_dim):\n",
    "    return Critic(\n",
    "        state_dim = state_dim, \n",
    "        action_dim = action_dim, \n",
    "        fc1_units = 256,\n",
    "        fc2_units = 128,\n",
    "        seed = seed)\n",
    "\n",
    "agent = Agent(\n",
    "    create_actor = create_actor, \n",
    "    create_critic = create_critic, \n",
    "    replay_buffer = ReplayBuffer(buffer_size = buffer_size, seed = seed),\n",
    "    noise = OUNoise(size = (num_agents, action_size), seed = seed),\n",
    "    state_dim = state_size, \n",
    "    action_dim = action_size,\n",
    "    seed = seed,\n",
    "    lr_actor = learning_rate_actor,\n",
    "    lr_critic = learning_rate_critic,\n",
    "    batch_size = 128,\n",
    "    discount = 0.99)\n",
    "\n",
    "agent.actor_local.load_state_dict(torch.load('checkpoint_actor.pth'))\n",
    "agent.critic_local.load_state_dict(torch.load('checkpoint_critic.pth'))\n",
    "\n",
    "env = EnvMultipleWrapper(env=unity_env, train_mode=False)\n",
    "states = env.reset()\n",
    "for t in range(5000):\n",
    "    actions = agent.act(states, add_noise=False)\n",
    "    next_states, rewards, dones = env.step(actions)\n",
    "    states = next_states\n",
    "    if np.any(dones):\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "       BatchNorm1d-1                   [-1, 33]              66\n",
      "            Linear-2                  [-1, 256]           8,704\n",
      "            Linear-3                  [-1, 128]          32,896\n",
      "            Linear-4                    [-1, 4]             516\n",
      "================================================================\n",
      "Total params: 42,182\n",
      "Trainable params: 42,182\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.00\n",
      "Forward/backward pass size (MB): 0.00\n",
      "Params size (MB): 0.16\n",
      "Estimated Total Size (MB): 0.16\n",
      "----------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/drlnd/lib/python3.6/site-packages/torch/nn/functional.py:995: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.\n",
      "  warnings.warn(\"nn.functional.tanh is deprecated. Use torch.tanh instead.\")\n"
     ]
    }
   ],
   "source": [
    "from torchsummary import summary\n",
    "summary(agent.actor_local, (state_size,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "       BatchNorm1d-1                   [-1, 33]              66\n",
      "            Linear-2                  [-1, 256]           8,704\n",
      "            Linear-3                  [-1, 128]          33,408\n",
      "            Linear-4                    [-1, 1]             129\n",
      "================================================================\n",
      "Total params: 42,307\n",
      "Trainable params: 42,307\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.00\n",
      "Forward/backward pass size (MB): 0.00\n",
      "Params size (MB): 0.16\n",
      "Estimated Total Size (MB): 0.17\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "summary(agent.critic_local, [(state_size,), (action_size, )])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}