Tennis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collaboration and Competition\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the third project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Tennis.app\"`\n",
    "- **Windows** (x86): `\"path/to/Tennis_Windows_x86/Tennis.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Tennis_Windows_x86_64/Tennis.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Tennis_Linux/Tennis.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Tennis_Linux/Tennis.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Tennis_Linux_NoVis/Tennis.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Tennis.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Tennis.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name=\"Tennis.app\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, two agents control rackets to bounce a ball over a net. If an agent hits the ball over the net, it receives a reward of +0.1.  If an agent lets a ball hit the ground or hits the ball out of bounds, it receives a reward of -0.01.  Thus, the goal of each agent is to keep the ball in play.\n",
    "\n",
    "The observation space consists of 8 variables corresponding to the position and velocity of the ball and racket. Two continuous actions are available, corresponding to movement toward (or away from) the net, and jumping. \n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 2\n",
      "Size of each action: 2\n",
      "There are 2 agents. Each observes a state with length: 24\n",
      "The state for the first agent looks like: [ 0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.          0.          0.\n",
      "  0.          0.          0.          0.         -6.65278625 -1.5\n",
      " -0.          0.          6.83172083  6.         -0.          0.        ]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents \n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agents and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agents' performance, if they select actions at random with each time step.  A window should pop up that allows you to observe the agents.\n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agents are able to use their experiences to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for i in range(1, 6):                                      # play game for 5 episodes\n",
    "#     env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "#     states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "#     scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "#     while True:\n",
    "#         actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "#         actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "#         env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "#         next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "#         rewards = env_info.rewards                         # get reward (for each agent)\n",
    "#         dones = env_info.local_done                        # see if episode finished\n",
    "#         scores += env_info.rewards                         # update the score (for each agent)\n",
    "#         states = next_states                               # roll over states to next time step\n",
    "#         if np.any(dones):                                  # exit loop if episode finished\n",
    "#             break\n",
    "#     print('Score (max over agents) from episode {}: {}'.format(i, np.max(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "%matplotlib inline\n",
    "\n",
    "unity_env = env"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env.action_size: 2\n",
      "env.state_size: 24\n",
      "env.num_agents: 2\n"
     ]
    }
   ],
   "source": [
    "from unity_env_wrapper import EnvMultipleWrapper\n",
    "\n",
    "env = EnvMultipleWrapper(env=unity_env, train_mode=True)\n",
    "print(f\"env.action_size: {env.action_size}\")\n",
    "print(f\"env.state_size: {env.state_size}\")\n",
    "print(f\"env.num_agents: {env.num_agents}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from maddpg_agent import MultiAgent\n",
    "from model import Actor\n",
    "from model import Critic\n",
    "from replay_buffer import ReplayBuffer\n",
    "from noise import OUNoise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "buffer_size = int(1e5)\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "learning_rate_actor = 1e-4\n",
    "learning_rate_critic = 1e-3\n",
    "seed = 2\n",
    "episodes_before_train = 300\n",
    "batch_size = 256\n",
    "\n",
    "action_dim = env.action_size\n",
    "state_dim = env.state_size\n",
    "num_agents = env.num_agents\n",
    "\n",
    "def create_actor():\n",
    "    return Actor(\n",
    "        state_dim=state_dim,\n",
    "        action_dim=action_dim,\n",
    "        fc1_units=400,\n",
    "        fc2_units=300,\n",
    "        seed=seed)\n",
    "\n",
    "def create_critic():\n",
    "    return Critic(\n",
    "        state_dim=state_dim * num_agents,\n",
    "        action_dim=action_dim * num_agents,\n",
    "        fc1_units=400,\n",
    "        fc2_units=300,\n",
    "        seed=seed)\n",
    "\n",
    "def create_noise():\n",
    "    return OUNoise(size=action_dim, seed=seed)\n",
    "\n",
    "replay_buffer = ReplayBuffer(buffer_size=buffer_size, seed=seed)\n",
    "agent = MultiAgent(\n",
    "    num_agents=num_agents,\n",
    "    create_actor=create_actor,\n",
    "    create_critic=create_critic,\n",
    "    replay_buffer=replay_buffer,\n",
    "    create_noise=create_noise,\n",
    "    state_dim=state_dim,\n",
    "    action_dim=action_dim,\n",
    "    episodes_before_train=episodes_before_train,\n",
    "    device=device,\n",
    "    lr_actor=learning_rate_actor,\n",
    "    lr_critic=learning_rate_critic,\n",
    "    batch_size=batch_size,\n",
    "    discount=0.99,\n",
    "    tau=1e-3,\n",
    "    initial_noise_scale=1.0,\n",
    "    noise_reduction=0.99,\n",
    "    seed=seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from train import train, plot_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   0% |                                          | ETA:  --:--:--\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 47/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 16.89"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   1% |                                           | ETA:  0:08:50\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 50/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 16.72\n",
      "Episode 99/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 17.31"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   2% |                                           | ETA:  0:06:45\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 17.28\n",
      "Episode 145/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 17.14"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   3% |#                                          | ETA:  0:05:51\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 150/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 17.15\n",
      "Episode 197/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 16.81"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   4% |#                                          | ETA:  0:05:26\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 200/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 16.82\n",
      "Episode 248/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 16.79"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   5% |##                                         | ETA:  0:05:13\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 250/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 16.94\n",
      "Episode 299/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 16.78"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   6% |##                                         | ETA:  0:05:11\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Start training...\n",
      "Episode 300/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 16.77\n",
      "Episode 349/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 15.52"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   7% |###                                        | ETA:  0:12:17\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 350/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.10            Average steps: 15.52\n",
      "Episode 399/5000            Score: 0.00            Average Score: 0.00            Max Score: 0.00            Average steps: 14.20"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   8% |###                                        | ETA:  0:17:41\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 400/5000            Score: 0.00            Average Score: 0.00            Max Score: 0.00            Average steps: 14.20\n",
      "Episode 449/5000            Score: 0.00            Average Score: 0.00            Max Score: 0.00            Average steps: 14.20"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:   9% |###                                        | ETA:  0:21:41\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 450/5000            Score: 0.00            Average Score: 0.00            Max Score: 0.00            Average steps: 14.20\n",
      "Episode 499/5000            Score: 0.00            Average Score: 0.00            Max Score: 0.00            Average steps: 14.20"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  10% |####                                       | ETA:  0:24:31\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 500/5000            Score: 0.00            Average Score: 0.00            Max Score: 0.00            Average steps: 14.20\n",
      "Episode 549/5000            Score: 0.00            Average Score: 0.00            Max Score: 0.00            Average steps: 14.20"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  11% |####                                       | ETA:  0:26:44\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 550/5000            Score: 0.00            Average Score: 0.00            Max Score: 0.00            Average steps: 14.22\n",
      "Episode 599/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.09            Average steps: 15.50"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  12% |#####                                      | ETA:  0:29:14\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 600/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.09            Average steps: 15.50\n",
      "Episode 649/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.09            Average steps: 15.92"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  13% |#####                                      | ETA:  0:30:45\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 650/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.09            Average steps: 15.90\n",
      "Episode 699/5000            Score: 0.00            Average Score: 0.01            Max Score: 0.09            Average steps: 15.73"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  14% |######                                     | ETA:  0:32:23\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 700/5000            Score: 0.09            Average Score: 0.01            Max Score: 0.09            Average steps: 15.90\n",
      "Episode 749/5000            Score: 0.00            Average Score: 0.03            Max Score: 0.10            Average steps: 18.61"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  15% |######                                     | ETA:  0:34:39\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 750/5000            Score: 0.09            Average Score: 0.03            Max Score: 0.10            Average steps: 18.78\n",
      "Episode 799/5000            Score: 0.10            Average Score: 0.05            Max Score: 0.20            Average steps: 24.25"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  16% |######                                     | ETA:  0:37:59\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 800/5000            Score: 0.10            Average Score: 0.05            Max Score: 0.20            Average steps: 24.34\n",
      "Episode 849/5000            Score: 0.09            Average Score: 0.05            Max Score: 0.30            Average steps: 24.76"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  17% |#######                                    | ETA:  0:39:31\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 850/5000            Score: 0.00            Average Score: 0.04            Max Score: 0.30            Average steps: 24.59\n",
      "Episode 899/5000            Score: 0.09            Average Score: 0.05            Max Score: 0.30            Average steps: 23.62"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  18% |#######                                    | ETA:  0:41:34\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 900/5000            Score: 0.09            Average Score: 0.05            Max Score: 0.30            Average steps: 23.53\n",
      "Episode 949/5000            Score: 0.10            Average Score: 0.08            Max Score: 0.10            Average steps: 28.88"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  19% |########                                   | ETA:  0:44:27\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 950/5000            Score: 0.09            Average Score: 0.08            Max Score: 0.10            Average steps: 29.06\n",
      "Episode 999/5000            Score: 0.10            Average Score: 0.10            Max Score: 0.20            Average steps: 34.21"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  20% |########                                   | ETA:  0:47:59\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1000/5000            Score: 0.50            Average Score: 0.10            Max Score: 0.50            Average steps: 35.74\n",
      "Episode 1049/5000            Score: 0.10            Average Score: 0.11            Max Score: 0.50            Average steps: 39.14"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  21% |#########                                  | ETA:  0:51:09\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1050/5000            Score: 0.20            Average Score: 0.11            Max Score: 0.50            Average steps: 39.51\n",
      "Episode 1099/5000            Score: 0.09            Average Score: 0.11            Max Score: 0.50            Average steps: 40.28"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  22% |#########                                  | ETA:  0:53:33\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1100/5000            Score: 0.09            Average Score: 0.11            Max Score: 0.40            Average steps: 38.74\n",
      "Episode 1149/5000            Score: 0.09            Average Score: 0.13            Max Score: 0.60            Average steps: 46.33"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  23% |#########                                  | ETA:  0:57:52\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1150/5000            Score: 0.10            Average Score: 0.13            Max Score: 0.60            Average steps: 45.95\n",
      "Episode 1199/5000            Score: 0.09            Average Score: 0.13            Max Score: 0.60            Average steps: 48.29"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  24% |##########                                 | ETA:  1:00:07\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1200/5000            Score: 0.09            Average Score: 0.13            Max Score: 0.60            Average steps: 48.29\n",
      "Episode 1249/5000            Score: 0.09            Average Score: 0.14            Max Score: 0.70            Average steps: 54.20"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  25% |##########                                 | ETA:  1:05:01\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1250/5000            Score: 0.10            Average Score: 0.14            Max Score: 0.70            Average steps: 54.41\n",
      "Episode 1299/5000            Score: 0.19            Average Score: 0.17            Max Score: 0.70            Average steps: 69.29"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  26% |###########                                | ETA:  1:10:11\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1300/5000            Score: 0.30            Average Score: 0.18            Max Score: 0.70            Average steps: 70.27\n",
      "Episode 1349/5000            Score: 1.30            Average Score: 0.41            Max Score: 1.30            Average steps: 164.16"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop:  27% |###########                                | ETA:  1:35:51\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1350/5000            Score: 1.30            Average Score: 0.43            Max Score: 1.30            Average steps: 168.64\n",
      "Episode 1359/5000            Score: 1.30            Average Score: 0.50            Max Score: 1.40            Average steps: 196.64"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "training loop: 100% |###########################################| Time: 0:38:43\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Episode 1360/5000            Score: 0.70            Average Score: 0.51            Max Score: 1.40            Average steps: 198.92Environment solved in 1260 episodes!\t Average Score: 0.51\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "scores = train(env=env, agent=agent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEPCAYAAABBUX+lAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XtcVHX+P/DXcAdveIHRJM11VUwQC9dbRatleGE0w++2mtpuLmWuS9l+7bK6aRev6Vfb6lvpmtWK/SQ1ESs0M0uFNPnmBUUFFRUEBgSRywzM5fz+GDkwMMPMgbkyr+c+9hFzzuec854Dvuczn/M57yMTBEEAERG1e17ODoCIiByDCZ+IyEMw4RMReQgmfCIiD8GET0TkIZjwiYg8BBM+EZGHYMInIvIQTPhERB6CCZ+IyEMw4RMReQgfZx5cr9ejuroavr6+kMlkzgyFiMhtCIIAjUaDDh06wMvL+n67UxN+dXU1Ll686MwQiIjc1sCBA9GpUyer2zs14fv6+gIwBO3n5yd5+6ysLERERNg6LLtizI7hbjG7W7wAY3YUUzHX1dXh4sWLYg61llMTfv0wjp+fH/z9/Vu1j9Zu50yM2THcLWZ3ixdgzI5iLmapQ+G8aEtE5CGY8ImIPIRTh3RaotfrkZ+fj+rqarNtfHx8kJ2d7cCo2q41MXfo0AFhYWGSrsYTETVlVcKvqqrCH//4R3z00UcICwsz2ebQoUN48803cfDgQZsEVlpaCplMhkGDBplNdNXV1ejQoYNNjucoUmPW6/UoKChAaWkpQkND7RgZEbV3FruMp06dwowZM5CXl2e2TWlpKVavXm3LuHDr1i3I5XKP79V6eXlBLpejoqLC2aEQkZuzmE2Tk5OxdOnSFnuXS5YswYIFC2wamE6nkzzlqL3y9fWFVqt1dhhE5OYsDuksX768xfWff/457r33XkRFRdksqHq8+9aA54Go/Xj5vcPw8/XC2/MecPix23TR9uLFi9i/fz8+/fRTFBUVtXo/WVlZzQPz8Wnxgm09a9rYwoEDB/DJJ59Ap9NBEARMnjwZTz/9dKv21ZqY6+rqkJmZ2arj2YIzj91a7hazu8ULMObWyM4rkxyHrWJuU8JPS0tDSUkJ4uPjodFooFQqMXPmTGzbtk3SfiIiIprdWJCdnW3x4qajLtoWFxdjw4YN2LVrF7p27Yrq6mrMnj0bgwYNwiOPPCJpX62N2c/Pzy7foqyRmZmJ6Ohopxy7tdwtZneLF2DMrbYtHwCsjsNUzLW1tSY7ypa0KeEnJiYiMTERAJCfn485c+ZITvbuoLy8HBqNBmq1GoBhmuSqVavg7++P9PR0rFq1CoIg4K677sK6desQFBSEFStWICMjAzKZDFOmTMGzzz6LY8eOiRe3BwwYgNdffx1vvvkmcnJyoNPpkJCQgLi4OGe+VSJqx1qV8BMSEpCYmIjIyEhbx2PSwRPX8N3xa82W63Q6eHt7t2nf40f0wbjhfVpsEx4ejkceeQSPPvooBg8ejJEjR0KhUKBXr16YMWMGNm/ejMGDB+N//ud/8NVXX8HLywuFhYXYs2cP6urqMHv2bAwcOBCBgYG4evUqDh06hE6dOmHt2rUYMmQIVq9eLU59jYqKwt13392m90REZIrVCb/x/PpNmzY1Wx8WFmazOfiu6I033sD8+fNx5MgRHDlyBH/4wx/wt7/9DXK5HIMHDwYAvPTSSwAM33ymTZsGb29vBAYGQqFQICMjA+PGjcM999wjVrdLT0+HWq3Gzp07AQA1NTXIyclhwiciu3DZO20bGzfcdC/cUWP4hw4dQk1NDSZNmoT4+HjEx8cjOTkZqampRu0qKytRXV0NvV5vtFwQBOh0OgDGRZD0ej3eeecdDBkyBIDhfoYuXbrY+d0Qkafy7LuarBQQEIB169YhP99wsUUQBOTm5iIiIgJlZWXIzc0FAPz73//GF198gVGjRmH37t3Q6XRQqVRITU3FyJEjm+131KhR+OKLLwAASqUSU6ZMQWFhoePeGBF5FLfo4TvbqFGjsGDBAsybNw8ajQYA8NBDD+G///u/MXbsWLz88svQaDTo06cP1qxZAz8/P+Tl5WHq1KnQaDSYMmUKxo8fj2PHjhntd8GCBVi2bBni4uKg0+mwaNEi9OnT8vUEIqLWYsK30rRp0zBt2rRmy0eMGIFdu3Y1W75kyZJmy0aOHGl0/aNjx45Yu3atbQMlIjKDQzpERB6CCZ+IyEMw4RMReQiXTviCIDg7BJfA80BEtuCyF20DAgJw8+ZNdO/e3aOrRQqCgJs3byIgIMDZoRCRBUU3q5Gw4oBVbZ9dcQCv/2UkwkI72TmqBi6b8MPCwpCfn4+SkhKzberq6uDn5+fAqNquNTEHBASYfdIYEbmOSwXWP6io8GY18pVVTPiA4aEf/fr1a7FNZmam0ypItpY7xkxEVpI4+uro0VqXHsMnInIngtSML7l92zDhExF5CCZ8IiIbkTpEwyEdIiJ3JTXh2ycKs5jwiYg8BBM+EZGNSL5oyyEdIiL3JHkMn7N0iIjIHqxO+FVVVYiLixOf+tTYgQMHMHXqVEyZMgXz589HRYX1d5sREbUXkmfhu+KQzqlTpzBjxgzk5eU1W1dVVYVly5Zh48aN2LNnDwYNGoT33nvP1nESEbk+6WM6DmVVwk9OTsbSpUsRGhrabJ1Go8GyZcsgl8sBAIMGDeJzWYmIXJBVtXSWL19udl3Xrl3x6KOPAgDUajU2btyI2bNn2yY6IiI3Ir2wgmO7+DJBQrH1cePG4fPPPzdZubGyshLz58/H3XffjRUrVli1v9raWmRlZVkfLRGRCzt1pRpfZZRb3T5+TDdE3hPU6uNFRETA39/f6vY2qZapVCoxd+5cjBo1Cv/4xz8kby816HqZmZmIjo6WvJ0zMWbHcLeY3S1egDGbUiFcByQk/H79+iH6/pZLn5uKubWd5TYnfJ1Oh3nz5mHixImYP39+W3dHROTGpA3ROLq0QqsTfkJCAhITE1FUVIRz585Bp9Nh3759AAw99pbG/YmI2iPJ0ywdPC9TUsI/ePCg+POmTZsAAJGRkTh//rxtoyIickOu/vhp3mlLROQkrJZJROS2JI7hu+KNV0REZBmHdIiIyAxWyyQicksuPkmHCZ+IyFY4pENERCaxh09E5Lakl09zJCZ8IiIb4ZAOERGZxCEdIiI35doDOkz4RES24+JjOkz4REQ2wnn4REQeQnoC5ywdIiL3xCEdIiLPwCEdIiIyibN0iIjclIuP6DDhExHZiiC1z+7gTwgmfCIiJ3HZIZ2qqirExcUhPz+/2brs7GzEx8cjNjYWixcvhlartWmQRERuwbU7+NYl/FOnTmHGjBnIy8szuX7RokX45z//iX379kEQBCQnJ9syRiIit+DiQ/jWJfzk5GQsXboUoaGhzdYVFBRArVZj2LBhAIAnnngCaWlpto2SiMgNSO6xO7iL72NNo+XLl5tdp1QqERISIr4OCQlBcXGxpCCysrIktW8sMzOz1ds6C2N2DHeL2d3iBRhzU9evV0pqf/XadWRmlltsZ6uYrUr4LRFMfELJZDJJ+4iIiIC/v7/kY2dmZiI6Olryds7EmB3D3WJ2t3gBxmzKtcpc4NcKq9v36XM3oqN/02IbUzHX1ta2qqPc5lk6crkcpaWl4uuSkhKTQz9ERO2d1BEal7xo25LevXvD399f/Mqxe/duxMTEtDkwIiL3Iy2DS56330atTvgJCQk4c+YMAGDt2rVYuXIlJk6cCJVKhTlz5tgsQCIid+Hqd9pKGsM/ePCg+POmTZvEn8PDw7Fjxw7bRUVE5IZcvDoy77QlIrIVU5NYWmxvpzjMYcInIvIQTPhERDbS7mfpEBGRgfRZN24yS4eIiJpw8Vk6TPhERDbi4qV0mPCJiGyFY/hERJ7Cxe+8YsInIrIR6emeF22JiDwCh3SIiNyUi4/oMOETETkLSysQEbkpqTdeSa2901ZM+EREHoIJn4jIVjiGT0REpnCWDhGRm3LxDj4TPhGRs7jNM22JiMiY5Fk3rjikk5qaikmTJmH8+PFISkpqtv7s2bOIj4/HlClT8Nxzz+H27ds2D5SIiNrGYsIvLi7G+vXrsW3bNqSkpGD79u3Izc01arN8+XIkJiZiz5496NevHzZv3my3gImI2guXu/EqPT0do0aNQnBwMIKCghAbG4u0tDSjNnq9HtXV1QAAlUqFgIAA+0RLRNSOOHqWjkywMOj08ccfo6amBgsXLgQAfPnllzh9+jTeeustsc3Jkyfx5z//GR06dEBgYCCSk5PRtWtXiwevra1FVlZWG98CEZFrOHCyAkfOVVrd/veRnfH7yM6tPl5ERAT8/f2tbu9jqYGpzwOZTCb+rFarsXjxYnz22WcYOnQotmzZgldeeQUbN260OgipQdfLzMxEdHS05O2ciTE7hrvF7G7xAozZlKyic4CEhH9Xr16Ijg5vsY2pmFvbWbY4pCOXy1FaWiq+ViqVCA0NFV9fvHgR/v7+GDp0KADgySefxPHjxyUHQkTk7qTO0nG5MfwxY8YgIyMDZWVlUKlU2L9/P2JiYsT1ffv2RVFRES5fvgwA+P777xEZGWm/iImIqFUsDunI5XIsXLgQc+bMgUajwfTp0zF06FAkJCQgMTERkZGRWLlyJV588UUIgoDu3btjxYoVjoidiMitOfqircWEDwAKhQIKhcJo2aZNm8SfH374YTz88MO2jYyIqJ3jnbZERG6KT7wiIiLTXLG0AhERWSY1f7vcLB0iImofmPCJiGxE8jx8PtOWiIjsgQmfiMhDMOETkceo0+qh1emdHYaIz7QlIrKTFck38PrHGXbbv9QEHuDvbZ9AzGDCJyKPcuZSqeVGDvDbu4OhePA3Dj0mEz4RkY1IKZXQu0dHBAX42jGa5pjwiYicwNF1dAAmfCIi22EtHSIiasYJHw5M+ERENuLiHXwmfCIiZ3DGhwMTPhGRjUipjePoOjoAEz4Rke24+JgOEz4RkRO47JBOamoqJk2ahPHjxyMpKanZ+suXL2P27NmYMmUK5s6di4qKCpsHSkTk6iQlcVecpVNcXIz169dj27ZtSElJwfbt25GbmyuuFwQBzz//PBISErBnzx4MHjwYGzdutGvQRERSOWPM3NVYTPjp6ekYNWoUgoODERQUhNjYWKSlpYnrz549i6CgIMTExAAA5s2bh6eeesp+ERMRtYLeAfle0kVbV7zTVqlUIiQkRHwdGhqK4uJi8fW1a9fQo0cPvPLKK1AoFFi6dCmCgoLsEy0RtXtbUs9C8fcUxL+SatP92rOH/59vs6H4e4rd9m8rPpYamDpJMplM/Fmr1eL48ePYunUrIiMjsWHDBqxatQqrVq2yOoisrCyr2zaVmZnZ6m2dhTE7hrvF7G7xAvaJedehfACG2vW23L9W15DLbB138gFDzMqSEqu3KS+/ZXUctorXYsKXy+U4ceKE+FqpVCI0NFR8HRISgr59+yIyMhIAEBcXh8TERElBREREwN/fX9I2gOEkREdHS97OmRizY7hbzO4WL2DHmLfliz/acv+1Gh2wvcDm+wUgxty9ew8gp9qqTYKDg62Kw9R5rq2tbVVH2eKQzpgxY5CRkYGysjKoVCrs379fHK8HgPvuuw9lZWU4f/48AODgwYMYMmSI5ECIiOxJcMAgvpSnaTnjIrJVPfyFCxdizpw50Gg0mD59OoYOHYqEhAQkJiYiMjISH3zwAZYsWQKVSoWePXtizZo1joidiMhqejsmWC8vGfR6AToJHyrOmDRkMeEDgEKhgEKhMFq2adMm8eeoqCjs2LHDtpEREdmQPROsl0wGPQTodK499ZN32hKRR7DnEIqXl2Eii07vOg9IN4UJn4g8gj2H8L3vZFIpPXxnDOkw4RORR7BvD9+QSqVctHUGJnwi8gh2vWh7594kST18V7zTloioPbDnEIq3OIbPIR0iIqez75CO4b8c0iEicgH2nEBTP6Sjd0SFtjZgwicij+CIaZlaCZ8qfMQhEbmF7Qcu4OO0YssNXYjUi7bXiyvx+KI9KCy1XBtHnIfPG6+IqL3Z+u15FJZpnB2GJFI71D9kXodOL+DwyQKLbesrCEv5UHHZRxwSEbk7qdMg65O4pIeaSPlU4SwdIiI7kZhg6x/7IeU6rItXVmDCJyLPILVDLWXmTf0joez2bcBGmPCJyCNITbCtGtKRdATHY8InIo8gtUNdfzOVpAuxLv7hwIRPRGSCl9jDt9y2NeP9vGhLRGQnDhnScUaBHAmY8InII0i/aGv4r5Reu5Tn5rJaJhGRvUielimlh19/45WEcFx1SCc1NRWTJk3C+PHjkZSUZLbdoUOHMG7cOJsFR0RkK1Lza6vunnXxIR2LDzEvLi7G+vXrsWvXLvj5+eGPf/wjRo4cid/+9rdG7UpLS7F69Wq7BUpE1BZSk3H9kI6Ui7Yunu8t9/DT09MxatQoBAcHIygoCLGxsUhLS2vWbsmSJViwYIFdgiQicjRxSMfFx+WlsJjwlUolQkJCxNehoaEoLjaukvf555/j3nvvRVRUlO0jJKJ27fDJAij+noIrNyqarVv84VGs/Ow4fs4qtHp/NWoNFn94FEU3jatcWup9/995JdZtyxRfN1y0tT6JV1TVWd3WGd8GLA7pmPoaVP/JBwAXL17E/v378emnn6KoqKhVQWRlZbVqOwDIzMy03MjFMGbHcLeY3S1ewDYxr9mWb/jvp+n4S2yo0brTuaUAgPTThVg2M8yq/Z2+UoPTuWV4N+kopj/QXVxeWNaQjE3FvexOHL8fZHh9Pb8KAKBUllh8n2q12qrYGrtdedvq82ervw2LCV8ul+PEiRPia6VSidDQhl9KWloaSkpKEB8fD41GA6VSiZkzZ2Lbtm1WBxEREQF/f3+JoRtOQnR0tOTtnIkxO4a7xexu8eJOcrRJzHf21aFjB8P+7rxuytpjVcnygYwydOvazWib3PxbQJrS/L6avCdlXR7wyy1079ED0dHDWjxm4PffA7erTK4bcW9PHD/XvDPcqVMnq96Tqb+N2traVnWULQ7pjBkzBhkZGSgrK4NKpcL+/fsRExMjrk9MTMS+ffuQkpKCjRs3IjQ0VFKyJyKyJfECatMVVg6h1I9q1I9j2OuxhS45LVMul2PhwoWYM2cOHn/8ccTFxWHo0KFISEjAmTNnHBEjEZHVZDA9f97aC6r1+b2+tXWJWWa5iQuwOKQDAAqFAgqFwmjZpk2bmrULCwvDwYMHbRMZEVEryO50Y5smamt71IYPCpn4gSH10YiujHfaElG7IvbwWzlFsj7R10/HtNfNVKyHT0TUVmZugrI2wdYP6YhDOxJuvHJ1TPhE1K6Ye/qUtf3ppj17ew3puORFWyIid2K2zIGVCVYvJnoz+3FjTPhE1K7IzIyvWH/Rtv6/1vfw3WREhwmfiFrPFatDNgzpGC+39iKueNFW4EVbIiKRC+b7hqJnTRK8tbG25qJta/CZtkTkVmyatGy0s7aWKm7as7fmTltzw0iuxqobr4jIsU5kF+N2dR3GDb/b2aG0TBCQr6zCT78WYMZjg9qU+KydDbP3yGVcL65EdLgcI4b0FJcX3azGrkO5OHmx5E5oTXv4Da//33cX8Mfxg8TXBSUNdXC+OpSLSwUV4n5UtVrMWPINAvx9sDbxIXTvEijGse/nq/jbH4Yhr/C2xHcLp3TxmfCJXNAb//4ZAFw+4QsA/vlxBkpvqTBpTD8Ed5JeBLHxvqzx8VeGki7fpOchdd1UcfmKT4/jyo2GxNvSJJ2ktPNGCX/vkcvizzt/yDXarr5iZ5VKg9Wfn8CiWcMR0jVQjOPv7/5kZeTOxyEdImo1QQC0Ov2dn9vYZW3j9hqt3nh3TYdiWth9nUZvfmUj2XlleObt/VJDM4kPMSciNyO06kEhpti6KGXzHr5zrjCbG+XijVdE5FYEodGsmLYmsDZu3yyxNu3gW9eJtzlXmsnEhE9ErSagIeG3tYff1h64pXn3TV/bq869tTgtk4jciiA0GtJpYwK1dU/YUnlknYMSvtmJS7zxiojcSqMhnbYnfNsmwOY9fmM6vZPGeJyICZ+IWk0A4HUn4be1x2zr/m6zIaImLzmkQ0QkgSAI8LqTRdqc8O3ewzde0JZ4pcTKi7ZE1G7UD+nodG0bIrH9GH7LtXR0urYk/FZvatN9SMWET0St1nhaZtt7+LaIyPr9tWUMX0oP3/xF21YfvtWsSvipqamYNGkSxo8fj6SkpGbrDxw4gKlTp2LKlCmYP38+KioqbB4oEbkewxi+4WdHXLRty7BP020bf0BJ3a+Th/9bzWLCLy4uxvr167Ft2zakpKRg+/btyM1tqDVRVVWFZcuWYePGjdizZw8GDRqE9957z65BE5GLEATb9fCtaCPlEJYecdiWDyhbXG9wydIK6enpGDVqFIKDgxEUFITY2FikpaWJ6zUaDZYtWwa5XA4AGDRoEAoLC+0XMZGbKL2lwoHjV6HT6bH7x1xotDoAwDfpV1BRVQvAUOHxh8zrZveRfaXMIbFa8m1GHt75zwl8m34Fe366JC7fc/iyWCmyccXJxiqqavH10StGZYdTD19GtUpj3NBC/jv0f/n47OtzRssyzxebbd9sd00WrEvKxD/+96j4u5Bi4YYfJW/TLBxXrJapVCoREhIivg4NDcXp06fF1127dsWjjz4KAFCr1di4cSNmz54tKYisrCxJ7RvLzMxs9bbOwpgdw9kxv7+3CKW3tTh74TIOnLyNy1euY3CfQHz4dTHSjlzA04+EYM3OG6ip1WPZzDCT8b78/mEsmxnmhOgblFdp8b97igAAP50sMFr3xf4L4s+nz11Cd5/SZttv/aEUuYVqQFWEnl39cKVYjc++L0XGr7mIf6C72E6lVrf4O1uX1Hzdsk0/i+dHrVYbrauqqjLaX06+ymj9+avlAIClH/6Azh28zR7XlGtFlVa3HXKXDsfOGi8L8JVhzEBfq/9GbfW3bDHhm/rqYqrmdWVlJebPn4/w8HBMmzZNUhARERHw95deVjUzMxPR0dGSt3MmxuwYrhCz6qtvAABdu8sB3EaXbiEYMDAM+LoYMp8AREdHo2ZbPgDDv7Phw4c3bHxnOQCnv498ZSVwJ+G3pGevXoiOHtxs+dbDPwJQY8DAcAzs0xVCdjGAUvgGdjK8tzvv1c/f3+i1terPT8CB74HbDd8ygoI6GJ07jX8h8NPN5jvwCUCP7l2AS9ckHdda0yaMwSffpRgt+3LVFKu3N/W3XFtb26qOssUhHblcjtLShk9tpVKJ0NBQozZKpRIzZ85EeHg4li9fLjkIovascZ+pvgPl1aTP5KbXAI1IHRNv1m209Tx8++5e9NLM+/H+orH22bmNWUz4Y8aMQUZGBsrKyqBSqbB//37ExMSI63U6HebNm4eJEydi8eLFbvOoLyJH0RvNBjH8t+m/E1e6Oae1rE345i542nzmS7PjmD5AW6to/jYsGGGhndq2EwexOKQjl8uxcOFCzJkzBxqNBtOnT8fQoUORkJCAxMREFBUV4dy5c9DpdNi3bx8AwxANe/pEBo1nr+jFHn77S/htLkZm45PQ7PknZnbf1iqfMlnzb2yuyqpHHCoUCigUCqNlmzZtAgBERkbi/Pnzto+MqJ1onFDqe5NNvwi7cr63Nh+a7eGb2UGzbzlSgrKGlQ+8amsRNS+ZzG1GNninLZGdNS45UJ/8mw/puG7Kt7YH7HK1dKzM+G0psQCYnsTiqpjwieysPhEKQqOLtl7uM6Rj7di8pYRf/8FhrpXd6+GbOXJbP6jcKN8z4RPZm+mLtsZtXDnhW5sQLRVPs9STbuspsJR47TWG3/R6jCtjwieylzuJROzhQxCTf7OLto6NTBJre/iWEqel/dh6SKdZPBzSYcInsjcx0QmNx/CN27jyGL61CdHSN4H6i6OCmXbOGtLR64U2HdvLjbKoG4VK5GZMPPqvPrG407RMa4c89BY+GOo/EOr/a6thLfPfHFquh98Ql75NhczYwycikal5+DI3umhr7bRFiz38Ox8I5r4xtPZbjrnjWjmiA10be/hulO+tm4fvyspuq1FQUoV77+mGn88WYUi/7rhadBs+3l7o2tkf14sqMTKil9j+fF4ZIAPOXb6JB6J6o6KqFlmXSjElpj98vJt//tVpdMg8r8ToyF44n1eGbp0DENotyKiNulaLkzklGBXRC3q9gIwzhRgd2QtanV7cVmyr0eNEdjGGD5aLyzLPF6NTkB8C/X2g0erxm95dcOVGBXy8vXC33PgOvvJKNY6fLYa/nzd+f38YrhdXQqvTo99dXWx1SkkCQRCQfroQ3bsE4FpxJbp08EO+sgp39+wkVoM8nWsoTbLn8GUc+MVQr+V6cSWOn2uoT3MqrwYPP2j5OH17GfY7qG83AMC5Kzfx068FqKiqhbxbEEYM6Yk+PTvjgy9PQqvTY8zQu/B/55UIk3eERqNHXuFt6AUBdRoddHoB9w0MRe+QjrhwrRzeXjLUanTo37sLSm6pUFWjgY+3TCwyZsmtqlrsPJiDn7MK7/zNywAIyM03PB/j06/P4kZpFf7zreG+nV/OFSP5wEVx+yqVBseypFfa3XUoB8U3a3C92LhaZ76yCm/8+2cE+ftAXacz24uvUWtx8IT5iqWWuNNFW7dP+C9t+BE3K9R4akI4ktJM3wC2YeHD6B8WDABY9N5hcfmWvQ2lVssrazF3SkSzbbeknsXeo1ewZsFDePl9w7ap66YatVmblIljZ4uwefF4/JJdjI92ncbf/jAMeYW3kXr4Mlb99UEM+Y2hKuBXGWW4kH8DmxePFz84lm362Wh/qeumInHdIZPHeuW9Iyi8WQ0AiBrQA/PXHDTZzhMIggC9AHh7yaDT6eFt4gPbXuqHEc5euYlVn//SYtvKmjrx5xq1FgBQUq7CW5uPicu/+7UCTylU6N4l0OQ+vv/lOt7d/qv4eseqOPj7euOV948Ytdv5Qy7u6dVZLFn8c1bLRc+yLpkoJtZKZy/fxNnLhv2Z+pC4XlyFzXuMy0b+59tso9dvbzku+bhbvzV/4+eJbPPlk20l0N98Gh18Tze7H18Kt0/4NysMJVFLylXm2c1tAAAUaUlEQVRm29T/I/t412mzbXb/eAm7f7yExx/ujzmTBuOJV/Yara9P9k2parU4dtbwj+pfyb/iVI6hN1deqUbRncT86gdHMGtCOJ4cPwiltw2xzF3+HYICfLB9+WSL7/Hb9Cv4352nkbxispjsAWDOsn0Wt20PSm+p8Oe39mPc8Ltx8MR1fLY0Ft06B2DFp8dxragSE0bfg09SDYnk7XljEDUgxMIejd0orcJzK78HAPztD8PwXvJJAMDWNyagS0d/fLjzFL5Jz7PpezLlT2/uB9D8w1vx95Rmbae/urfZsnr1yd6ZRkf2wuQH+mHJR+mt2v6lmffjvoGhmL0szXLjVmipg9hY6rqpUNdpEeDnA41WB61OgLeXDL4+XqjT6uHjJRM7GvW/t8a/r7fmjbFL/K3l9glfir1Hr1hss/vHS5g+boDV+7xV2fDwhPpkb8q2fefx5PhBRsvqP4gs2XUot9mxPElu/i0AEL92Xy28jW6dA8Te67cZeWLbn34tkJzwz11u6OV+1KhTUFBShS4d/R2S7O3pgai70KNLIMpuq3H4Tj37mbHhKCmvwXfHDUNMcQ/2Q22dDr16dMC1okro9QJGD+2FsttqnL18Ezcr1Lhwp9cec19vPDD0LtwquYazhT4Y+tse0OkFqGu16BEciOwrZZg9aTAC/Hzw1+lR0Or0qKiqQ1FZNX5zVxdcLboNL5kMNWotKmvq0CM4EFduVCCifw9UVNUivG83jBl6F/x9vbFoVjT2HL6M0K5B0On1GB15F/JuVKCkXIXf9O6CarUGvt5eCAzwQXBHf1SrNLjnri7Iu1EBLy8ZSivUCPDzRtltNeTdguCrKUHnHn0wZuhdGHFvT3h7y8Rva7er61Cj1qBPz86oVmnQIdAXABDgZ0iTvj7e8G2UMf19LdfQd7XBnnaT8F1tWpvszv+acrEwyQO8NON++N1JTvUJf8Zjhs5HfcJ/btpQs9tPeag/gIae66JZhrr9mZmFmPRI81r9Mfc1PLBlwuh72hR7zH1hRvsDANxv+YEw9UOoTWVmViB6WG8AwG96e951r3YzS0dr4S4/Kdp65x1gmPNr6iJR0ylkba4wSGSBI69tkGtrN38Jbb1brrX7kjp/t2l+10uo1OeMhx67g7Y8jNoTuEvp3vbI1SbwtJ+Eb8N/9FL2JTXZNOvhS/hwYWIzjd+SWuZONwaRfbWbhN/iMIzEv3dLRaCM27athy8lWdnyW0x7IuX35Q5sOTxJ1Fi7SfgtJUOpF3QlJeG29vDteCxPYa/z4qwL7HUanXMOTO1e+0n4LYyFS+0ZS+lhSe2NNQ3Fnsdqr5r+Nu3Vw2/rk5Baq07D33P74VrDae0o4ZtP6lJ7gI4cw5fSm/PUMfymPW1HzXRy1hAae/hkL1Yl/NTUVEyaNAnjx49HUlJSs/XZ2dmIj49HbGwsFi9eDK3WuhuKbKmlZCi1ByilveQefpPmUv5xe2oPv+nvtul50NopMTtrCK2WCZ/sxGLCLy4uxvr167Ft2zakpKRg+/btyM3NNWqzaNEi/POf/8S+ffsgCAKSk5PtFrA5zurhm2tr7sarpuq0Ei4Qe2gPv+nQStPzIGVqqylmy+Y66QOWPXyyF4t32qanp2PUqFEIDjYUH4uNjUVaWhoWLFgAACgoKIBarcawYcMAAE888QT+9a9/YebMmXYM29Dry7nRUD9HWV5jtu3ZKzcl3Ux16mJJi+uPnCoQf75yw3Tdkis3KnCjtKF6n14w3q5eponiTod/bWh35FQBlHfqBNVXXbQUk61dvlYDlY/99m9J9pUyo9fnLt80uhDfOP9fL67EkVMFkmK+cK2h0Jem0Qdw1uWb0Dgh6Z84b/+CX+QYrlYBwGLCVyqVCAlpqE0SGhqK06dPm10fEhKC4mJpf7BZWVmS2gPAjbI6JB1qqIHSUvG0PT9dxp6fLlu97237L7S4fvXnJyzu48ipG1Ztt9VEAac1WxvaNd6mcSnZ1sTUJkfKLLdxkD2HL2PPYdO/z+y8MmTn3Ym1jTHv/vFSm7ZvrZaqP1rjt738kVvYUHcpMzNT/NnfVwaNVhCXdQr0QqVKb9TGnABfGWobbdt03+7CkTGfPPkrvJvc+da7u5/kGGwVs8WEb+oTqvGNHJbWWyMiIgL+/v6StokGEOh3DAMGDb5Tddv4erhMJoNeEMR19bzuLPdqtN7fzwe1dYbrDvVtTb2D+n025evtBW9vL6jrtGJtbPFBF3fa1G917uw5REVFQKPVi7H5+XhDq9Mbxds07savvWQy+Pl6o7ZOa7KtrZ07ew73DrnXjkewzNR58ZLJ4O0tg0arR8Cd32Hj8ywl5sa/p6a/s/r1Af4+0Or00Gr18PH2QoC/D6pq6uDv5wO9XoCfrxfqNHr4+3lDVauFXi+gY6AvNFo9OnfwQ51Wjw4BPrhdXQe9IKBKpRH//s6cOYOBgwaLx/X18YJeL8DLSwadToBeEBDg5wN1nRaB/j7oEOCLiupa6PWC+CwFvSDgdnUdQoIDoarVwtfHG3pBMCry9UWU4Zu4r49h2WdR90EQBLHWTku2RQ2DIEBsm5mZiejo5rV0XJkjYt451HCOdXpBLJ38ZUQUqtWG5yN06ehv8tkb5piKuba2tlUdZYsJXy6X48SJht6jUqlEaGio0frS0oahhpKSEqP19tS1ow/69uzskGPZSmmBL+7q0dHZYUhSWuDrlufZETF36xxgdduAO32arne2aVz7Pr+jD/pIjLe+mqPRMe5UdgwKaL4OaEj0Da+tTzxNtyXTTH14Bvj7IKCFuvmOYvG3PWbMGGRkZKCsrAwqlQr79+9HTEyMuL53797w9/cXv3Ls3r3baD0REbkGiwlfLpdj4cKFmDNnDh5//HHExcVh6NChSEhIwJkzZwAAa9euxcqVKzFx4kSoVCrMmTPH7oETEZE0Vn3HUCgUUCgURss2bdok/hweHo4dO3bYNjIiIrKpdnOnLRERtYwJn4jIQzDhExF5CKfOE6qfw19XV9fqfdTWut+DvRmzY7hbzO4WL8CYHaVpzPU5U+qdvDLBiff+VlZW4uJF83ePEhGReQMHDkSnTp2sbu/UhK/X61FdXQ1fX18+ho2IyEqCIECj0aBDhw7w8rJ+ZN6pCZ+IiByHF22JiDwEEz4RkYdgwici8hBM+EREHoIJn4jIQzDhExF5CCZ8IiIP4bYJPzU1FZMmTcL48eORlJTk7HBE77//PiZPnozJkydjzZo1AAwPglcoFHjsscewfv16sW12djbi4+MRGxuLxYsXQ6vVOitsAMDq1avx6quvthjbjRs38NRTT2HChAl4/vnnUV1d7ZRYDx48iCeeeAITJkzA22+/DcD1z3NKSor4t7F69eoWY3Pmea6qqkJcXBzy8/MBSD+vzoi9aczbt29HXFwcFAoFXnvtNbEUgSvHXC8pKQmzZ88WX5uL7fbt23j22WcxceJEPPXUUygpKbF8UMENFRUVCWPHjhXKy8uF6upqQaFQCDk5Oc4OSzh69Kjw5JNPCrW1tUJdXZ0wZ84cITU1VXj44YeFa9euCRqNRnjmmWeEQ4cOCYIgCJMnTxZ+/fVXQRAE4bXXXhOSkpKcFnt6erowcuRI4ZVXXmkxtmeffVbYu3evIAiC8P777wtr1qxxeKzXrl0THnzwQaGwsFCoq6sTZsyYIRw6dMilz3NNTY3wu9/9Trh586ag0WiE6dOnC0ePHnW583zy5EkhLi5OGDJkiHD9+nVBpVJJPq+Ojr1pzJcvXxbGjx8vVFZWCnq9Xnj55ZeFLVu2uHTM9XJycoSHHnpImDVrlrjMXGxvvPGG8PHHHwuCIAhfffWV8MILL1g8rlv28NPT0zFq1CgEBwcjKCgIsbGxSEtLc3ZYCAkJwauvvgo/Pz/4+vqif//+yMvLQ9++fXH33XfDx8cHCoUCaWlpKCgogFqtxrBhhgceP/HEE057D7du3cL69esxb948ADAbm0ajwS+//ILY2Finxvzdd99h0qRJ6NmzJ3x9fbF+/XoEBga69HnW6XTQ6/VQqVTQarXQarXw8fFxufOcnJyMpUuXis+lPn36tKTz6ozYm8bs5+eHZcuWoWPHjpDJZBg4cCBu3Ljh0jEDhoJor7/+Ol544QVxWUuxHTp0SHwwVVxcHH766SdoNJoWj+v8p+q2glKpREhIiPg6NDQUp0+fdmJEBgMGDBB/zsvLwzfffIPZs2c3i7W4uLjZewgJCUFxcbFD4633+uuvY+HChSgsLATQ/PzWx1ZeXo6OHTvCx8fHqTFfvXoVvr6+mDt3LkpKSjB27FgMGDDApc9zx44d8cILL2DixIkICAjAiBEj4Ovr63Lnefny5UavTf1ba+m8OiP2pjH37t0bvXv3BgCUlZUhKSkJK1eudOmYAWDdunWIj49HWFiYuKyl2Bq/Hx8fH3Ts2BFlZWWQy+Vmj+uWPXzBRPkfVyq+lpOTg2eeeQavvPIK+vTp02y9TCZzmffw5ZdfolevXhg9erS4zFxsrhKzTqdDRkYG3nnnHSQnJ+PMmTPNxkHrY3OVmM+fP4+dO3fihx9+wJEjR+Dl5YWjR4+ajM1VYgak/y24UuzFxcV4+umnER8fj5EjR7p0zEePHkVhYSHi4+ONlkuNzVIhNbfs4cvlcpw4cUJ8rVQqjb4aOVNmZiYSExPxj3/8A5MnT8bx48dRWloqrq+PVS6XGy0vKSlxynv45ptvUFJSgqlTp6KiogI1NTWQyWQmY+vWrRuqqqqg0+ng7e3ttJh79OiB0aNHo1u3bgCARx55BGlpafD29hbbuNp5PnLkCEaPHo3u3bsDMHw137x5s0ufZwDNzp+l8+oqsV+6dAkJCQmYNWsWnnnmGZPvxZVi3rt3L3JycjB16lTU1NSgtLQUL774It555x2zsYWGhqK0tBQ9e/aEVqtFVVUVgoODWzyOW/bwx4wZg4yMDJSVlUGlUmH//v2IiYlxdlgoLCzEX//6V6xduxaTJ08GAERFReHKlSu4evUqdDod9u7di5iYGPTu3Rv+/v7IzMwEAOzevdsp72HLli3Yu3cvUlJSkJiYiHHjxmHlypUmY/P19cXw4cPxzTffODXmsWPH4siRI7h9+zZ0Oh0OHz6MCRMmuPR5Dg8PR3p6OmpqaiAIAg4ePIgRI0a49HkGpP/9ukLsVVVVmDt3Ll544QUx2QNw6ZhXrlyJb7/9FikpKXj77bcRERGBDRs2tBjbww8/jN27dwMwdNyGDx8OX1/flg/UxovNTrNnzx5h8uTJwmOPPSZs3LjR2eEIgiAIb731ljBs2DBhypQp4v+3bdsmpKenCwqFQnjssceE5cuXC3q9XhAEQcjOzhbi4+OFCRMmCC+99JJQW1vr1Ph37twpztIxF1t+fr4wa9YsYeLEicIzzzwj3Lp1yymxfvnll+Lv/4033hB0Op3Ln+ePP/5YiI2NFeLi4oTXXntNUKvVLnuex44dK84ekXpenRV7fcxbtmwRhgwZYvTvcMOGDS4dc2M///yz0Swdc7GVl5cLzz33nDBp0iThySefbLYfU1gPn4jIQ7jlkA4REUnHhE9E5CGY8ImIPAQTPhGRh2DCJyLyEG554xXRoEGDMHDgwGZ3Fn7wwQdGt6Y39e6776Jv3754/PHH2xzD5s2bkZOTg1WrVknarqSkBAsWLMD27duxe/duXLlyBQsXLmxzPESWMOGT2/rss8/Eu22t1bgwlbMcP34cv/vd7wAAx44dEwtgEdkbEz61O8eOHcOaNWsgl8tx/fp1BAQEYNWqVejfvz9effVVDBgwAHPnzsW//vUvfPfdd/D19UXXrl2xcuVKhIaG4sSJE1izZg1UKhV8fX3x4osvIiYmBhqNBm+//TbS09PRvXt3dO/eHZ06dQIAVFZWYvny5bh48SI0Gg1Gjx6Nl19+WSx6BQBnzpzBkiVLUFBQgO7du+Pw4cO4dOkSzp07h48++gi9evVy1ikjD8GET27r6aefNhrSCQsLwwcffAAAOHfuHF577TUMHz4cX3zxBRYtWoRdu3aJbQsLC/HZZ58hIyMDfn5++OSTT3D69GlER0cjMTERH374IaKiopCTk4NZs2Zhx44dOHjwIPLy8vD1119Dq9Vi1qxZYsJfsWIFhgwZglWrVkGn0+HVV1/Fli1bkJCQIB4zMjISKSkp+K//+i988MEHCAwMxKxZs5CSkuKgM0aejgmf3FZLQzrh4eEYPnw4ACA+Ph5vvvkmysvLxfVyuRzh4eGYNm0aYmJiEBMTg9GjR+PHH39Enz59EBUVBcBQ8vr+++/H8ePHkZGRgbi4OPj5+cHPzw8KhQIXLlwAYKhNfubMGezYsQMAoFarTcal1+tRVlaG0NBQHD9+HIMHD7bZ+SCyhAmf2qXGlTMBQ5nZxsu8vLywdetWnDlzBhkZGVixYgVGjhyJBx54oNm+BEEw+VjExvvT6/V499130b9/fwCGx881LWN75swZLFq0COXl5Zg6dSrKysogk8kwdepUDumQQ3BaJrVL58+fx/nz5wEYnm96//33o3Pnzkbr4+Li0L9/fzz33HP405/+hAsXLojVIesfqJOTk4NffvkFI0aMwEMPPYTdu3ejtrYWtbW1YgVDAHjwwQfx6aefQhAE1NXV4fnnn8fWrVuNYoqMjMTTTz+NZ599FikpKYiMjMSHH36IlJQUJntyCPbwyW01HcMHgJdeegkBAQHo0aMHNmzYgIKCAnTr1k18oHy98PBwTJw4EfHx8QgKCkJAQACWLFmCbt264d1338Vbb70FtVoNmUyGlStXol+/fujTpw+uXbuGuLg4BAcHo2/fvuL+Fi9ejOXLl0OhUECj0WDMmDH4y1/+0izmEydOYObMmRAEATk5ORzSIYditUxqd44dO4a33noLe/fudXYoRC6FQzpERB6CPXwiIg/BHj4RkYdgwici8hBM+EREHoIJn4jIQzDhExF5CCZ8IiIP8f8Bzzk5NftLxUQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_scores(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Watch Smart Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: TennisBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 8\n",
      "        Number of stacked Vector Observation: 3\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 2\n",
      "        Vector Action descriptions: , \n"
     ]
    }
   ],
   "source": [
    "from maddpg_agent import MultiAgent\n",
    "from model import Actor\n",
    "from model import Critic\n",
    "from replay_buffer import ReplayBuffer\n",
    "from noise import OUNoise\n",
    "from unity_env_wrapper import EnvMultipleWrapper\n",
    "from unityagents import UnityEnvironment\n",
    "import torch\n",
    "\n",
    "unity_env = UnityEnvironment(file_name=\"Tennis.app\")\n",
    "env = EnvMultipleWrapper(env=unity_env, train_mode=False)\n",
    "\n",
    "buffer_size = int(1e5)\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "learning_rate_actor = 1e-4\n",
    "learning_rate_critic = 1e-3\n",
    "seed = 2\n",
    "episodes_before_train = 300\n",
    "batch_size = 256\n",
    "\n",
    "action_dim = env.action_size\n",
    "state_dim = env.state_size\n",
    "num_agents = env.num_agents\n",
    "\n",
    "def create_actor():\n",
    "    return Actor(\n",
    "        state_dim=state_dim,\n",
    "        action_dim=action_dim,\n",
    "        fc1_units=400,\n",
    "        fc2_units=300,\n",
    "        seed=seed)\n",
    "\n",
    "def create_critic():\n",
    "    return Critic(\n",
    "        state_dim=state_dim * num_agents,\n",
    "        action_dim=action_dim * num_agents,\n",
    "        fc1_units=400,\n",
    "        fc2_units=300,\n",
    "        seed=seed)\n",
    "\n",
    "def create_noise():\n",
    "    return OUNoise(size=action_dim, seed=seed)\n",
    "\n",
    "replay_buffer = ReplayBuffer(buffer_size=buffer_size, seed=seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = MultiAgent(\n",
    "    num_agents=num_agents,\n",
    "    create_actor=create_actor,\n",
    "    create_critic=create_critic,\n",
    "    replay_buffer=replay_buffer,\n",
    "    create_noise=create_noise,\n",
    "    state_dim=state_dim,\n",
    "    action_dim=action_dim,\n",
    "    episodes_before_train=episodes_before_train,\n",
    "    device=device,\n",
    "    lr_actor=learning_rate_actor,\n",
    "    lr_critic=learning_rate_critic,\n",
    "    batch_size=batch_size,\n",
    "    discount=0.99,\n",
    "    tau=1e-3,\n",
    "    noise_reduction=0.99,\n",
    "    seed=seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "       BatchNorm1d-1                   [-1, 24]              48\n",
      "            Linear-2                  [-1, 400]          10,000\n",
      "            Linear-3                  [-1, 300]         120,300\n",
      "            Linear-4                    [-1, 2]             602\n",
      "================================================================\n",
      "Total params: 130,950\n",
      "Trainable params: 130,950\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.00\n",
      "Forward/backward pass size (MB): 0.01\n",
      "Params size (MB): 0.50\n",
      "Estimated Total Size (MB): 0.51\n",
      "----------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/drlnd/lib/python3.6/site-packages/torch/nn/functional.py:995: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.\n",
      "  warnings.warn(\"nn.functional.tanh is deprecated. Use torch.tanh instead.\")\n"
     ]
    }
   ],
   "source": [
    "from torchsummary import summary\n",
    "summary(agent.agents[0].actor, (state_dim,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "       BatchNorm1d-1                   [-1, 48]              96\n",
      "            Linear-2                  [-1, 400]          19,600\n",
      "            Linear-3                  [-1, 300]         121,500\n",
      "            Linear-4                    [-1, 1]             301\n",
      "================================================================\n",
      "Total params: 141,497\n",
      "Trainable params: 141,497\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.00\n",
      "Forward/backward pass size (MB): 0.01\n",
      "Params size (MB): 0.54\n",
      "Estimated Total Size (MB): 0.55\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "summary(agent.agents[0].critic, [(state_dim * num_agents,), (action_dim * num_agents, )])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, agent_i in enumerate(agent.agents):\n",
    "    agent_i.actor.load_state_dict(torch.load(f\"checkpoint_actor_{i}.pth\"))\n",
    "    agent_i.critic.load_state_dict(torch.load(f\"checkpoint_critic_{i}.pth\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.600000038743019\n"
     ]
    }
   ],
   "source": [
    "states = env.reset()\n",
    "score = np.zeros(env.num_agents)\n",
    "for t in range(5000):\n",
    "    actions = agent.act(states, add_noise=False)\n",
    "    next_states, rewards, dones = env.step(actions)\n",
    "    states = next_states\n",
    "    \n",
    "    score += rewards\n",
    "    if np.any(dones):\n",
    "        break\n",
    "        \n",
    "print(np.max(score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}