examples/guided_a3c.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Guided policy search \n",
    "\n",
    "_Speeding up training by applying auxillary imitation loss on expert actions:_\n",
    "\n",
    "`L_gps = lamda_a3c * L_a3c + guided_lambda * L_gps`\n",
    "\n",
    "##### Intuition:\n",
    "- This implementation is loosely reffered as 'guided policy search' after algorithm described in paper by S. Levine and P. Abbeel [Learning Neural Network Policies with Guided PolicySearch under Unknown Dynamics](https://people.eecs.berkeley.edu/~svlevine/papers/mfcgps.pdf) in a sense that we utilize same idea of fitting 'local' (here - single episode) oracle for environment with unknown dynamics and use actions demonstrated by it to optimize trajectory distribution for training agent; also  connected to s RLfD ideas.\n",
    "\n",
    "- Using expert actions is proven way to speed up training by exploring more relevant state-action space regiones;\n",
    "\n",
    "- It can also lead to suboptimal policies when demonstrated actions are suboptimal (learnt model is irrelevant) and agent is strictly entingled   to expert trajectories (cannot explore and act on it's own);\n",
    "- So it is essential to find ballanced degree of agent dependensy on advisor;\n",
    "\n",
    "##### Papers:\n",
    "- Levine et al., [Learning Neural Network Policies with Guided PolicySearch under Unknown Dynamics](https://people.eecs.berkeley.edu/~svlevine/papers/mfcgps.pdf)\n",
    "\n",
    "- Brys et al., [Reinforcement Learning from Demonstration through Shaping](https://www.ijcai.org/Proceedings/15/Papers/472.pdf)\n",
    "\n",
    "- Wiewiora et al., [Principled Methods for Advising Reinforcement Learning Agents](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.14.6412&rep=rep1&type=pdf)\n",
    "\n",
    "- Abbeel, Ng, [Exploration and Apprenticeship Learning in Reinforcement Learning](https://icml.cc/Conferences/2005/proceedings/papers/001_Exploration_AbbeelNg.pdf)\n",
    "\n",
    "##### Implementation details:\n",
    "\n",
    "- For each train episode expert has access to entire data range but does not provides complete state-action trajectories. Instead it fits simple local model of `external` part of the environment which provides  `advises` in form of categorical probability distribution in action space. Morover, imitation loss is defined in actions subspace, namely over 'buy' and 'sell' actions. Such relaxed conditions seems to work better than strictly following [possibly suboptimal] expert trajectory.\n",
    "\n",
    "- Note that expert advices conditioned on external state observations only, i.e. price dynamics; expert doesn't account current inner agent state (opened position, acoount value etc.). It can be said it acts similar to real-life financial advisor'...now it probably time to buy'; it does not bear any responsibility for advices given so it is on agent's side to deside follow or not. The degree of agent 'independency' from oracle is regulated by `guided_lambda` hyperparamter. When set to zero - oracle advises completely ignored. When set >> 1 it can be clearly seen (with sine wave data for example) than agent performace is degraded due to strong dependency from oracle; reasonable values for this setup seems to lie  within 0.1 .. 1.0 range;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Expert estimated actions example:\n",
    "- currently oracle implements extremely simple strategy: it finds local peaks along entire given episode data, filters it by time and value and estimate signals by reversing position after every up- or down- peak. Than smoothes by convolving signals with gaussian kernel, normalises and outputs discrete prob. distribution over action space.\n",
    "\n",
    "![Signal](./img/oracle_signal.png)\n",
    "\n",
    "##### Expert actions distribution:\n",
    "\n",
    "- It seems that using MSE error on action subsets leaves agent better degree of freedom than cross-entropy loss over entire action space; though it should be highly dependend on quality of advised actions.\n",
    "\n",
    "![A.Distr](./img/oracle_p_distr.png)\n",
    "\n",
    "##### Example of distribution over actions learnt by agent *:\n",
    "*sample, irrelevant to image above\n",
    "![AgentA.Distr](./img/learnt_a_p_d.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Addition for those who like to look: extended summaries.\n",
    "- Tensorboard summaries (see images tab) are updated and now include:\n",
    "\n",
    "##### - action probabilities for an episode (see above);\n",
    "\n",
    "##### - value function for an episode:\n",
    "![Vfn](./img/v_fn_ep.png)\n",
    "\n",
    "##### - visualisations of hidden state of LSTM blocks:\n",
    "- 256 cells along vertical axis, environment timesteps - along horisontal one; it can be clearly seen how rnn state gets reset when deal is closed.\n",
    "![LSTM_state](./img/lstm_h_ep.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TODO:\n",
    "- 1. Relax agent dependency on advised actions by annelaing `guided_lambda` in the course of training;\n",
    "- 2. Implemnet potential-based approach to advising as in Wiewiora et al."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\") # suppress h5py deprecation warning\n",
    "\n",
    "import os\n",
    "import backtrader as bt\n",
    "import numpy as np\n",
    "from gym import spaces\n",
    "\n",
    "from btgym import BTgymEnv, BTgymDataset, BTgymRandomDataDomain\n",
    "from btgym.algorithms import Launcher\n",
    "\n",
    "from btgym.research.gps.aac import GuidedAAC\n",
    "from btgym.research.gps.policy import GuidedPolicy_0_0\n",
    "from btgym.research.gps.strategy import GuidedStrategy_0_0, ExpertObserver\n",
    "\n",
    "from btgym.research.verbose_env_runner import VerboseEnvRunnerFn\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set backtesting engine and parameters:\n",
    "\n",
    "engine = bt.Cerebro()\n",
    "\n",
    "engine.addstrategy(\n",
    "    GuidedStrategy_0_0,\n",
    "    drawdown_call=10, # max % to loose, in percent of initial cash\n",
    "    target_call=10,  # max % to win, same\n",
    "    skip_frame=10,\n",
    "    gamma=0.99,\n",
    "    state_ext_scale=np.linspace(4e3, 1e3, num=6),\n",
    "    reward_scale=7,\n",
    "    expert_config=  # see btgym.research.gps.oracle.Oracle class for details\n",
    "        {\n",
    "            'time_threshold': 5,\n",
    "            'pips_threshold': 10, \n",
    "            'pips_scale': 1e-4,\n",
    "            'kernel_size': 10,\n",
    "            'kernel_stddev': 1,\n",
    "        },\n",
    ")\n",
    "\n",
    "# Expert actions observer:\n",
    "engine.addobserver(ExpertObserver)\n",
    "\n",
    "# Set leveraged account:\n",
    "engine.broker.setcash(2000)\n",
    "engine.broker.setcommission(commission=0.0001, leverage=10.0) # commisssion to imitate spread\n",
    "engine.addsizer(bt.sizers.SizerFix, stake=5000)  \n",
    "\n",
    "# Data: uncomment to get up to six month of 1 minute bars:\n",
    "data_m1_6_month = [\n",
    "    './data/DAT_ASCII_EURUSD_M1_201701.csv',\n",
    "    './data/DAT_ASCII_EURUSD_M1_201702.csv',\n",
    "    './data/DAT_ASCII_EURUSD_M1_201703.csv',\n",
    "    #'./data/DAT_ASCII_EURUSD_M1_201704.csv',\n",
    "    #'./data/DAT_ASCII_EURUSD_M1_201705.csv',\n",
    "    #'./data/DAT_ASCII_EURUSD_M1_201706.csv',\n",
    "]\n",
    "\n",
    "# Uncomment single choice of source file:\n",
    "dataset = BTgymRandomDataDomain(  \n",
    "    #filename=data_m1_6_month,\n",
    "    #filename='./data/DAT_ASCII_EURUSD_M1_2016.csv', # full year\n",
    "    filename='./data/test_sine_1min_period256_delta0002.csv',  # simple sine \n",
    "\n",
    "    trial_params=dict(\n",
    "        start_weekdays={0, 1, 2, 3, 4, 5, 6},\n",
    "        sample_duration={'days': 3, 'hours': 0, 'minutes': 0},\n",
    "        start_00=False,\n",
    "        time_gap={'days': 1, 'hours': 10},\n",
    "        test_period={'days': 0, 'hours': 0, 'minutes': 0},\n",
    "    ),\n",
    "    episode_params=dict(\n",
    "        start_weekdays={0, 1, 2, 3, 4, 5, 6},\n",
    "        sample_duration={'days': 1, 'hours': 23, 'minutes': 50},\n",
    "        start_00=False,\n",
    "        time_gap={'days': 1, 'hours': 0},\n",
    "    ),\n",
    ")\n",
    "\n",
    "env_config = dict(\n",
    "    class_ref=BTgymEnv, \n",
    "    kwargs=dict(\n",
    "        dataset=dataset,\n",
    "        engine=engine,\n",
    "        render_modes=['episode', 'human', 'external', 'internal'],\n",
    "        render_state_as_image=True,\n",
    "        render_ylabel='OHL_diff. / Internals',\n",
    "        render_size_episode=(12,8),\n",
    "        render_size_human=(9, 4),\n",
    "        render_size_state=(11, 3),\n",
    "        render_dpi=75,\n",
    "        port=5000,\n",
    "        data_port=4999,\n",
    "        connect_timeout=90,\n",
    "        verbose=0,\n",
    "    )\n",
    ")\n",
    "\n",
    "cluster_config = dict(\n",
    "    host='127.0.0.1',\n",
    "    port=12230,\n",
    "    num_workers=4,  # Set according CPU's available or so\n",
    "    num_ps=1,\n",
    "    num_envs=1,\n",
    "    log_dir=os.path.expanduser('~/tmp/gps'),\n",
    ")\n",
    "\n",
    "policy_config = dict(\n",
    "    class_ref=GuidedPolicy_0_0,\n",
    "    kwargs={\n",
    "        'lstm_layers': (256, 256),\n",
    "        'lstm_2_init_period': 50,\n",
    "        'conv_2d_layer_config': (\n",
    "             (32, (3, 1), (2, 1)),\n",
    "             (32, (3, 1), (2, 1)),\n",
    "             (64, (3, 1), (2, 1)),\n",
    "             (64, (3, 1), (2, 1))\n",
    "         ),\n",
    "        'encode_internal_state': False,\n",
    "    }\n",
    ")\n",
    "\n",
    "trainer_config = dict(\n",
    "    class_ref=GuidedAAC,\n",
    "    kwargs=dict(\n",
    "        opt_learn_rate=1e-4, # scalar or random log-uniform \n",
    "        opt_end_learn_rate=1e-5,\n",
    "        opt_decay_steps=20*10**6,\n",
    "        model_gamma=0.99,\n",
    "        model_gae_lambda=1.0,\n",
    "        model_beta=0.01, # Entropy reg, scalar or random log-uniform\n",
    "        aac_lambda=1.0, # main a3c loss weight\n",
    "        guided_lambda=1.0,  # Imitation loss weight\n",
    "        guided_decay_steps=10*10**6,  # annealing guided_lambda to zero in 10M steps\n",
    "        rollout_length=20,\n",
    "        time_flat=True,\n",
    "        use_value_replay=False,\n",
    "        episode_train_test_cycle=[1,0],\n",
    "        model_summary_freq=100,\n",
    "        episode_summary_freq=5,\n",
    "        env_render_freq=5,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2018-04-30 11:11:22.337851] NOTICE: LauncherShell: </Users/muzikin/tmp/gps> created.\n",
      "[2018-04-30 11:11:25.173027] NOTICE: GuidedA3C_0: learn_rate: 0.000100, entropy_beta: 0.010000\n",
      "\n",
      "********************************************************************************************\n",
      "**  Press `Ctrl-C` or jupyter:[Kernel]->[Interrupt] to stop training and close launcher.  **\n",
      "********************************************************************************************\n",
      "\n",
      "[2018-04-30 11:11:29.041775] NOTICE: GuidedA3C_3: learn_rate: 0.000100, entropy_beta: 0.010000\n",
      "[2018-04-30 11:11:29.041709] NOTICE: GuidedA3C_2: learn_rate: 0.000100, entropy_beta: 0.010000\n",
      "[2018-04-30 11:11:29.041677] NOTICE: GuidedA3C_1: learn_rate: 0.000100, entropy_beta: 0.010000\n",
      "[2018-04-30 11:11:31.137693] NOTICE: GuidedA3C_0: guided_lambda: 1.000000, guided_decay_steps: 10000000\n",
      "[2018-04-30 11:11:35.628495] NOTICE: GuidedA3C_1: guided_lambda: 1.000000, guided_decay_steps: 10000000\n",
      "[2018-04-30 11:11:35.658271] NOTICE: GuidedA3C_3: guided_lambda: 1.000000, guided_decay_steps: 10000000\n",
      "[2018-04-30 11:11:35.703234] NOTICE: GuidedA3C_2: guided_lambda: 1.000000, guided_decay_steps: 10000000\n",
      "WARNING:tensorflow:From /Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/algorithms/worker.py:263: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please switch to tf.train.MonitoredTrainingSession\n",
      "WARNING:tensorflow:From /Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/algorithms/worker.py:263: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please switch to tf.train.MonitoredTrainingSession\n",
      "WARNING:tensorflow:From /Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/algorithms/worker.py:263: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please switch to tf.train.MonitoredTrainingSession\n",
      "WARNING:tensorflow:From /Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/algorithms/worker.py:263: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please switch to tf.train.MonitoredTrainingSession\n",
      "INFO:tensorflow:Starting queue runners.\n",
      "INFO:tensorflow:Starting queue runners.\n",
      "INFO:tensorflow:Starting queue runners.\n",
      "[2018-04-30 11:11:52.038764] NOTICE: Worker_1: started training at step: 0\n",
      "[2018-04-30 11:11:52.098803] NOTICE: Worker_3: started training at step: 0\n",
      "[2018-04-30 11:11:52.129917] NOTICE: Worker_2: started training at step: 0\n",
      "INFO:tensorflow:Starting standard services.\n",
      "INFO:tensorflow:Saving checkpoint to path /Users/muzikin/tmp/gps/train/model.ckpt\n",
      "INFO:tensorflow:Starting queue runners.\n",
      "INFO:tensorflow:global/global_step/sec: 0\n",
      "[2018-04-30 11:12:04.179039] NOTICE: Worker_0: started training at step: 0\n",
      "INFO:tensorflow:global/global_step/sec: 184.272\n",
      "INFO:tensorflow:global/global_step/sec: 190.793\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Process BTgymDataFeedServer-2:1:\n",
      "Process DrawCerebro-2:2:31:\n",
      "Process BTgymServer-4:1:\n",
      "Process BTgymServer-3:1:\n",
      "Process BTgymServer-5:1:\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2018-04-30 11:16:19.215415] NOTICE: LauncherShell: worker_1 has joined.\n",
      "[2018-04-30 11:16:19.228459] NOTICE: LauncherShell: worker_2 has joined.\n",
      "[2018-04-30 11:16:19.230069] NOTICE: LauncherShell: worker_3 has joined.\n",
      "[2018-04-30 11:16:19.235352] NOTICE: LauncherShell: chief_worker_0 has joined.\n",
      "[2018-04-30 11:16:19.236864] NOTICE: LauncherShell: parameter_server_0 has joined.\n",
      "[2018-04-30 11:16:19.238155] NOTICE: LauncherShell: Launcher closed.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/dataserver.py\", line 159, in run\n",
      "    service_input = socket.recv_pyobj()\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/zmq/sugar/socket.py\", line 491, in recv_pyobj\n",
      "    msg = self.recv(flags)\n",
      "  File \"zmq/backend/cython/socket.pyx\", line 693, in zmq.backend.cython.socket.Socket.recv (zmq/backend/cython/socket.c:7683)\n",
      "  File \"zmq/backend/cython/socket.pyx\", line 727, in zmq.backend.cython.socket.Socket.recv (zmq/backend/cython/socket.c:7460)\n",
      "  File \"zmq/backend/cython/socket.pyx\", line 145, in zmq.backend.cython.socket._recv_copy (zmq/backend/cython/socket.c:2344)\n",
      "  File \"zmq/backend/cython/checkrc.pxd\", line 12, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:9621)\n",
      "KeyboardInterrupt\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/server.py\", line 606, in run\n",
      "    episode = cerebro.run(stdstats=True, preload=False, oldbuysell=True)[0]\n",
      "  File \"/Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/server.py\", line 606, in run\n",
      "    episode = cerebro.run(stdstats=True, preload=False, oldbuysell=True)[0]\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/cerebro.py\", line 1073, in run\n",
      "    runstrat = self.runstrategies(iterstrat)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/cerebro.py\", line 1073, in run\n",
      "    runstrat = self.runstrategies(iterstrat)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/cerebro.py\", line 1235, in runstrategies\n",
      "    self._runnext(runstrats)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/cerebro.py\", line 1235, in runstrategies\n",
      "    self._runnext(runstrats)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/cerebro.py\", line 1564, in _runnext\n",
      "    strat._next()\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/cerebro.py\", line 1557, in _runnext\n",
      "    self._brokernotify()\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/strategy.py\", line 326, in _next\n",
      "    minperstatus = self._getminperstatus()\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/cerebro.py\", line 1297, in _brokernotify\n",
      "    self._broker.next()\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/strategy.py\", line 248, in _getminperstatus\n",
      "    self._minperstatus = minperstatus = max(dlens)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/brokers/bbroker.py\", line 1068, in next\n",
      "    self._try_exec(order)\n",
      "KeyboardInterrupt\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/brokers/bbroker.py\", line 1012, in _try_exec\n",
      "    self._try_exec_market(order, popen, phigh, plow)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/brokers/bbroker.py\", line 822, in _try_exec_market\n",
      "    self._execute(order, ago=0, price=p, dtcoc=dtcoc)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/brokers/bbroker.py\", line 668, in _execute\n",
      "    comminfo = self.getcommissioninfo(order.data)\n",
      "KeyboardInterrupt\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/rendering/plotter.py\", line 74, in run\n",
      "    figfilename='_tmp_btgym_render.png',\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/cerebro.py\", line 930, in plot\n",
      "    start=start, end=end, use=use)\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/plot/plot.py\", line 222, in plot\n",
      "    self.plotdata(data, self.dplotsover[data])\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/multiprocessing/process.py\", line 249, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/plot/plot.py\", line 629, in plotdata\n",
      "    data, opens, highs, lows, closes, volumes, vollabel)\n",
      "  File \"/Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/server.py\", line 606, in run\n",
      "    episode = cerebro.run(stdstats=True, preload=False, oldbuysell=True)[0]\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/cerebro.py\", line 1073, in run\n",
      "    runstrat = self.runstrategies(iterstrat)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/plot/plot.py\", line 568, in plotvolume\n",
      "    alpha=volalpha, label=vollabel)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/plot/finance.py\", line 359, in plot_volume\n",
      "    **kwargs)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/cerebro.py\", line 1154, in runstrategies\n",
      "    strat = stratcls(*sargs, **skwargs)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/metabase.py\", line 88, in __call__\n",
      "    _obj, args, kwargs = cls.doinit(_obj, *args, **kwargs)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/plot/finance.py\", line 283, in __init__\n",
      "    **kwargs)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/metabase.py\", line 78, in doinit\n",
      "    _obj.__init__(*args, **kwargs)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/backtrader/plot/finance.py\", line 340, in barcollection\n",
      "    **kwargs)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/matplotlib/collections.py\", line 938, in __init__\n",
      "    self.set_verts(verts, closed)\n",
      "  File \"/Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/research/gps/strategy.py\", line 100, in __init__\n",
      "    super(GuidedStrategy_0_0, self).__init__(**kwargs)\n",
      "  File \"/Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/research/strategy_gen_4.py\", line 310, in __init__\n",
      "    super(DevStrat_4_7, self).__init__(**kwargs)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/matplotlib/collections.py\", line 950, in set_verts\n",
      "    if np.ma.isMaskedArray(xy):\n",
      "  File \"/Users/muzikin/Yandex.Disk.localized/work/btgym/btgym/research/strategy_gen_4.py\", line 119, in __init__\n",
      "    self.log.debug('DEV_dataset_stat:\\n{}'.format(self.p.dataset_stat))\n",
      "KeyboardInterrupt\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/pandas/core/base.py\", line 51, in __str__\n",
      "    return self.__unicode__()\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/pandas/core/frame.py\", line 586, in __unicode__\n",
      "    line_width=width, show_dimensions=show_dimensions)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/pandas/core/frame.py\", line 1529, in to_string\n",
      "    formatter.to_string()\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/pandas/io/formats/format.py\", line 584, in to_string\n",
      "    strcols = self._to_str_columns()\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/pandas/io/formats/format.py\", line 519, in _to_str_columns\n",
      "    fmt_values = self._format_col(i)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/pandas/io/formats/format.py\", line 696, in _format_col\n",
      "    space=self.col_space, decimal=self.decimal)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/pandas/io/formats/format.py\", line 1755, in format_array\n",
      "    return fmt_obj.get_result()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/pandas/io/formats/format.py\", line 1776, in get_result\n",
      "    return _make_fixed_width(fmt_values, self.justify)\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/pandas/io/formats/format.py\", line 2244, in _make_fixed_width\n",
      "    strings = [just(x) for x in strings]\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/pandas/io/formats/format.py\", line 2244, in <listcomp>\n",
      "    strings = [just(x) for x in strings]\n",
      "  File \"/Users/muzikin/anaconda/envs/tensorforce/lib/python3.6/site-packages/pandas/io/formats/format.py\", line 2240, in just\n",
      "    if (conf_max > 3) & (adj.len(x) > max_len):\n",
      "KeyboardInterrupt\n"
     ]
    }
   ],
   "source": [
    "launcher = Launcher(\n",
    "    cluster_config=cluster_config,\n",
    "    env_config=env_config,\n",
    "    trainer_config=trainer_config,\n",
    "    policy_config=policy_config,\n",
    "    test_mode=False,\n",
    "    max_env_steps=100*10**6,\n",
    "    root_random_seed=0,\n",
    "    purge_previous=1,  # ask to override previously saved model and logs\n",
    "    verbose=0,\n",
    ")\n",
    "\n",
    "# Train it:\n",
    "launcher.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}