D12_Hadoop - Ready! on CentOS6 - ZK,HDFS,YARN,HBase,Hive,Spark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebb3e0-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebb53e-495a-11e8-ba7d-0242ac130002",
     "previous": null
    }
   },
   "source": [
    "# About: Hadoop準備用Notebook\n",
    "\n",
    "----\n",
    "\n",
    "Hadoop環境の準備用のNotebookです。以下のソフトウェアをインストールします。\n",
    "\n",
    "- HDFS\n",
    "- YARN\n",
    "- HBase\n",
    "- Hive\n",
    "- Spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebb53e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebb67e-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebb3e0-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## *Operation Note*\n",
    "\n",
    "*This is a cell for your own recording.  ここに経緯を記述*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebb67e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebb7a0-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebb53e-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "# インストール対象設定\n",
    "\n",
    "インストール対象のグループ名(`hadoop_all_クラスタ名`)を指定してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebb7a0-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebb8cc-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebb67e-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "target_group = 'hadoop_all_testcluster'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebb8cc-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebb9ee-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebb7a0-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "なお、各マシンは以下 prerequisite 項目の実行済みとします。\n",
    "\n",
    " - disable ipv6\n",
    " - ensure /hadoop/dataX directory for mount point is present\n",
    " - deploy /etc/hosts\n",
    " - deploy /etc/resolv.conf\n",
    " - setup ntpserver"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebb9ee-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebbb10-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebb8cc-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "操作のためのPlaybookを準備する。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebbb10-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebbc32-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebb9ee-495a-11e8-ba7d-0242ac130002"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/tmp/tmpaDuGp8'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import tempfile\n",
    "\n",
    "work_dir = tempfile.mkdtemp()\n",
    "work_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebbc32-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebbd54-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebbb10-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "PlaybookはGitHubで公開しているものを使う。一時ディレクトリにcloneしておく。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebbd54-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebbe76-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebbc32-495a-11e8-ba7d-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into '/tmp/tmpaDuGp8/hadoop'...\n",
      "remote: Counting objects: 849, done.\u001b[K\n",
      "remote: Compressing objects: 100% (7/7), done.\u001b[K\n",
      "remote: Total 849 (delta 0), reused 0 (delta 0), pack-reused 841\u001b[K\n",
      "Receiving objects: 100% (849/849), 172.26 KiB | 0 bytes/s, done.\n",
      "Resolving deltas: 100% (267/267), done.\n",
      "Checking connectivity... done.\n",
      "/tmp/tmpaDuGp8/hadoop\n",
      "└── playbooks\n",
      "    ├── conf_base.retry\n",
      "    ├── conf_base.yml\n",
      "    ├── conf_hdfs_base.yml\n",
      "    ├── conf_hdfs_spark.yml\n",
      "    ├── conf_hdfs_tez.yml\n",
      "    ├── conf_hdfs_yarn.yml\n",
      "    ├── conf_namenode_bootstrapstandby.yml\n",
      "    ├── conf_tez.yml\n",
      "    ├── enter_hdfs_safemode.yml\n",
      "    ├── format_namenode.yml\n",
      "    ├── group_vars\n",
      "    │   └── all\n",
      "    │       ├── base\n",
      "    │       ├── cgroups\n",
      "    │       ├── collect\n",
      "    │       ├── f500.dumpall\n",
      "    │       ├── hbase_master\n",
      "    │       ├── hbase_regionserver\n",
      "    │       ├── hcatalog\n",
      "    │       ├── hdfs_base\n",
      "    │       ├── hdfs_spark\n",
      "    │       ├── hdfs_tez\n",
      "    │       ├── hdfs_yarn\n",
      "    │       ├── hive\n",
      "    │       ├── httpfs\n",
      "    │       ├── hue\n",
      "    │       ├── java7\n",
      "    │       ├── java8\n",
      "    │       ├── journalnode\n",
      "    │       ├── mapreduce_history\n",
      "    │       ├── namenode\n",
      "    │       ├── namenode_bootstrapstandby\n",
      "    │       ├── namenode_format\n",
      "    │       ├── os\n",
      "    │       ├── pig\n",
      "    │       ├── presto_client\n",
      "    │       ├── presto_coordinator\n",
      "    │       ├── presto_user\n",
      "    │       ├── presto_worker\n",
      "    │       ├── resourcemanager\n",
      "    │       ├── site-defaults\n",
      "    │       ├── slavenode\n",
      "    │       ├── spark\n",
      "    │       ├── spark_history\n",
      "    │       ├── spark_user\n",
      "    │       ├── storm\n",
      "    │       ├── tez\n",
      "    │       └── zookeeper_server\n",
      "    ├── install-base.yml\n",
      "    ├── install_client.yml\n",
      "    ├── install_hbase_master.yml\n",
      "    ├── install_hbase_regionserver.yml\n",
      "    ├── install_hcatalog.yml\n",
      "    ├── install_hive.yml\n",
      "    ├── install_httpfs.yml\n",
      "    ├── install_hue.yml\n",
      "    ├── install_journalnode.yml\n",
      "    ├── install_mapreduce_history.yml\n",
      "    ├── install_namenode.yml\n",
      "    ├── install_pig.yml\n",
      "    ├── install_resourcemanager.yml\n",
      "    ├── install_slavenode.yml\n",
      "    ├── install_spark_historyserver.yml\n",
      "    ├── install_spark.yml\n",
      "    ├── install_timelineservice.yml\n",
      "    ├── install_zookeeper.yml\n",
      "    ├── roles\n",
      "    │   ├── base\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── conf.yml\n",
      "    │   │   │   ├── kerberos.yml\n",
      "    │   │   │   ├── keytab.yml\n",
      "    │   │   │   ├── main.yml\n",
      "    │   │   │   ├── principal.yml\n",
      "    │   │   │   └── repo.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── capacity-scheduler.xml.j2\n",
      "    │   │       ├── container-executor.cfg.j2\n",
      "    │   │       ├── core-site.xml.j2\n",
      "    │   │       ├── hadoop-env.sh.j2\n",
      "    │   │       ├── hadoop-metrics2.properties.j2\n",
      "    │   │       ├── hadoop-metrics.properties.j2\n",
      "    │   │       ├── hdfs-site.xml.j2\n",
      "    │   │       ├── hdp.repo.j2\n",
      "    │   │       ├── hosts.exclude.j2\n",
      "    │   │       ├── hosts.list.j2\n",
      "    │   │       ├── log4j.properties.j2\n",
      "    │   │       ├── mapred-env.sh.j2\n",
      "    │   │       ├── mapred-site.xml.j2\n",
      "    │   │       ├── merge-keytabs.ktutil.j2\n",
      "    │   │       ├── ssl-client.xml.j2\n",
      "    │   │       ├── ssl-server.xml.j2\n",
      "    │   │       ├── yarn-env.sh.j2\n",
      "    │   │       ├── yarn-site.xml.j2\n",
      "    │   │       └── zk-acl.txt.j2\n",
      "    │   ├── cgroups\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── conf.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   ├── main.yml\n",
      "    │   │   │   └── resource.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── cgconfig.conf.j2\n",
      "    │   │       └── cgroups.sh.j2\n",
      "    │   ├── client\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── install.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── collect\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── handlers\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── README.md\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── vars\n",
      "    │   │       └── main.yml\n",
      "    │   ├── datanode_server_deletedata\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── delete.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── f500.dumpall\n",
      "    │   │   ├── COPYING\n",
      "    │   │   ├── COPYING.LESSER\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── README.md\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       └── dumpall.j2\n",
      "    │   ├── hbase_master\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   ├── main.yml\n",
      "    │   │   │   └── principal.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── hadoop-metrics2-hbase.properties.j2\n",
      "    │   │       ├── hbase-env.sh.j2\n",
      "    │   │       ├── hbase-master.j2\n",
      "    │   │       ├── hbase-policy.xml.j2\n",
      "    │   │       ├── hbase-service-test.rb.j2\n",
      "    │   │       ├── hbase-site.xml.j2\n",
      "    │   │       ├── log4j.properties.j2\n",
      "    │   │       ├── regionservers.j2\n",
      "    │   │       └── zk-jaas.conf.j2\n",
      "    │   ├── hbase_regionserver\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── files\n",
      "    │   │   │   └── graceful_stop.sh\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   ├── main.yml\n",
      "    │   │   │   └── principal.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── hadoop-metrics2-hbase.properties.j2\n",
      "    │   │       ├── hbase-env.sh.j2\n",
      "    │   │       ├── hbase-policy.xml.j2\n",
      "    │   │       ├── hbase-regionserver.j2\n",
      "    │   │       ├── hbase-site.xml.j2\n",
      "    │   │       ├── log4j.properties.j2\n",
      "    │   │       ├── regionservers.j2\n",
      "    │   │       └── zk-jaas.conf.j2\n",
      "    │   ├── hcatalog\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       └── hcat-env.sh.j2\n",
      "    │   ├── hdfs_base\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── config.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── hdfs_spark\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── config.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── hdfs_tez\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── config.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── hdfs_yarn\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── config.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── hive\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   ├── main.yml\n",
      "    │   │   │   └── principal.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── hive-exec-log4j.properties.j2\n",
      "    │   │       ├── hive-log4j.properties.j2\n",
      "    │   │       └── hive-site.xml.j2\n",
      "    │   ├── httpfs\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── hadoop-httpfs-default.j2\n",
      "    │   │       ├── hadoop-httpfs.j2\n",
      "    │   │       ├── httpfs-env.sh.j2\n",
      "    │   │       ├── httpfs-log4j.properties.j2\n",
      "    │   │       ├── httpfs.sh.j2\n",
      "    │   │       ├── httpfs-signature.secret.j2\n",
      "    │   │       └── httpfs-site.xml.j2\n",
      "    │   ├── hue\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── hue_httpd.conf.j2\n",
      "    │   │       ├── hue.ini.j2\n",
      "    │   │       └── log.conf.j2\n",
      "    │   ├── java7\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── files\n",
      "    │   │   │   ├── env_keep_javahome\n",
      "    │   │   │   └── java.sh\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── config.yml\n",
      "    │   │       ├── install.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── java8\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── install.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── journalnode\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       └── default_hadoop-hdfs-journalnode.j2\n",
      "    │   ├── journalnode_server_createdir\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── conf.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── journalnode_server_deletedata\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── delete.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── mapreduce_history\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       └── default_hadoop-mapreduce-historyserver.j2\n",
      "    │   ├── namenode\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── default_hadoop-hdfs-namenode.j2\n",
      "    │   │       ├── default_hadoop-hdfs-zkfc.j2\n",
      "    │   │       ├── hdfs-balancer.sh.j2\n",
      "    │   │       └── jaas-hdfs.conf.j2\n",
      "    │   ├── namenode_bootstrapstandby\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── config.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── namenode_format\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── config.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── os\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── kernel.yml\n",
      "    │   │       ├── limits.yml\n",
      "    │   │       ├── main.yml\n",
      "    │   │       └── thp.yml\n",
      "    │   ├── pig\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── log4j.properties.j2\n",
      "    │   │       └── pig.properties.j2\n",
      "    │   ├── presto_client\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── config.properties.j2\n",
      "    │   │       ├── hive.properties.j2\n",
      "    │   │       ├── jvm.config.j2\n",
      "    │   │       ├── launcher.j2\n",
      "    │   │       ├── log.properties.j2\n",
      "    │   │       └── node.properties.j2\n",
      "    │   ├── presto_coordinator\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── files\n",
      "    │   │   │   └── env_keep_prestohome\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── catalog.yml\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── config.properties.j2\n",
      "    │   │       ├── hive.properties.j2\n",
      "    │   │       ├── jvm.config.j2\n",
      "    │   │       ├── launcher.j2\n",
      "    │   │       ├── log.properties.j2\n",
      "    │   │       ├── node.properties.j2\n",
      "    │   │       └── presto.sh.j2\n",
      "    │   ├── prestogres\n",
      "    │   ├── presto_user\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── main.yml\n",
      "    │   │       └── user.yml\n",
      "    │   ├── presto_worker\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── files\n",
      "    │   │   │   └── env_keep_prestohome\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── catalog.yml\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── config.properties.j2\n",
      "    │   │       ├── hive.properties.j2\n",
      "    │   │       ├── jvm.config.j2\n",
      "    │   │       ├── launcher.j2\n",
      "    │   │       ├── log.properties.j2\n",
      "    │   │       ├── node.properties.j2\n",
      "    │   │       └── presto.sh.j2\n",
      "    │   ├── resourcemanager\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       └── default_hadoop-yarn-resourcemanager.j2\n",
      "    │   ├── site-defaults\n",
      "    │   │   └── defaults\n",
      "    │   │       └── main.yml\n",
      "    │   ├── slavenode\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── default_hadoop-hdfs-datanode.j2\n",
      "    │   │       └── default_hadoop-yarn-nodemanager.j2\n",
      "    │   ├── spark\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── files\n",
      "    │   │   │   └── env_keep_sparkhome\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install-tarball.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   ├── main.yml\n",
      "    │   │   │   └── principal.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── fairscheduler.xml.j2\n",
      "    │   │       ├── log4j.properties.j2\n",
      "    │   │       ├── metrics.properties.j2\n",
      "    │   │       ├── spark-defaults.conf.j2\n",
      "    │   │       ├── spark-env.sh.j2\n",
      "    │   │       └── spark.sh.j2\n",
      "    │   ├── spark_history\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── config.yml\n",
      "    │   │       └── main.yml\n",
      "    │   ├── spark_user\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── tasks\n",
      "    │   │       ├── main.yml\n",
      "    │   │       └── user.yml\n",
      "    │   ├── storm\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── files\n",
      "    │   │   │   ├── storm-drpc\n",
      "    │   │   │   ├── storm-nimbus\n",
      "    │   │   │   ├── storm.py\n",
      "    │   │   │   ├── storm-supervisor\n",
      "    │   │   │   └── storm-ui\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   ├── main.yml\n",
      "    │   │   │   └── user.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── storm_env.ini.j2\n",
      "    │   │       ├── storm-env.sh.j2\n",
      "    │   │       ├── storm-slider-env.sh.j2\n",
      "    │   │       └── storm.yaml.j2\n",
      "    │   ├── tez\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       └── tez-site.xml.j2\n",
      "    │   ├── timelineservice\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   └── templates\n",
      "    │   │       └── default_hadoop-yarn-timelineserver.j2\n",
      "    │   ├── zookeeper_server\n",
      "    │   │   ├── defaults\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── meta\n",
      "    │   │   │   └── main.yml\n",
      "    │   │   ├── tasks\n",
      "    │   │   │   ├── config.yml\n",
      "    │   │   │   ├── install.yml\n",
      "    │   │   │   ├── main.yml\n",
      "    │   │   │   └── principal.yml\n",
      "    │   │   └── templates\n",
      "    │   │       ├── jaas.conf.j2\n",
      "    │   │       ├── log4j.properties.j2\n",
      "    │   │       ├── myid.j2\n",
      "    │   │       ├── zoo.cfg.j2\n",
      "    │   │       ├── zookeeper-env.sh.j2\n",
      "    │   │       └── zookeeper-server.j2\n",
      "    │   └── zookeeper_server_deletedata\n",
      "    │       └── tasks\n",
      "    │           ├── delete.yml\n",
      "    │           └── main.yml\n",
      "    ├── start_datanode.yml\n",
      "    ├── start_hbase_master.yml\n",
      "    ├── start_hbase_regionserver.yml\n",
      "    ├── start_hcatalog.yml\n",
      "    ├── start_httpfs.yml\n",
      "    ├── start_hue.yml\n",
      "    ├── start_journalnode.yml\n",
      "    ├── start_mapreduce_historyserver.yml\n",
      "    ├── start_namenode.retry\n",
      "    ├── start_namenode.yml\n",
      "    ├── start_nodemanager.yml\n",
      "    ├── start_resourcemanager.yml\n",
      "    ├── start_spark_historyserver.yml\n",
      "    ├── start_timelineservice.yml\n",
      "    ├── start_zookeeper-server.yml\n",
      "    ├── stop_datanode.yml\n",
      "    ├── stop_hbase_master.yml\n",
      "    ├── stop_hbase_regionserver.yml\n",
      "    ├── stop_hcatalog.yml\n",
      "    ├── stop_journalnode.yml\n",
      "    ├── stop_mapreduce_historyserver.yml\n",
      "    ├── stop_namenode.yml\n",
      "    ├── stop_nodemanager.yml\n",
      "    ├── stop_resourcemanager.yml\n",
      "    ├── stop_spark_historyserver.yml\n",
      "    ├── stop_timelineservice.yml\n",
      "    ├── stop_zookeeper-server.yml\n",
      "    ├── sync_kdc.yml\n",
      "    └── upgrade_namenode.yml\n",
      "\n",
      "194 directories, 404 files\n"
     ]
    }
   ],
   "source": [
    "!rm -fr {work_dir}/hadoop\n",
    "!git clone https://github.com/NII-cloud-operation/Literate-computing-Hadoop.git {work_dir}/hadoop\n",
    "!tree {work_dir}/hadoop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebbe76-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebbf8e-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebbd54-495a-11e8-ba7d-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 244\r\n",
      "drwxr-xr-x  4 root root 4096 Aug 26 11:44 .\r\n",
      "drwxr-xr-x  4 root root 4096 Aug 26 11:44 ..\r\n",
      "-rw-r--r--  1 root root   13 Aug 26 11:44 conf_base.retry\r\n",
      "-rw-r--r--  1 root root   39 Aug 26 11:44 conf_base.yml\r\n",
      "-rw-r--r--  1 root root  136 Aug 26 11:44 conf_hdfs_base.yml\r\n",
      "-rw-r--r--  1 root root  137 Aug 26 11:44 conf_hdfs_spark.yml\r\n",
      "-rw-r--r--  1 root root  135 Aug 26 11:44 conf_hdfs_tez.yml\r\n",
      "-rw-r--r--  1 root root  136 Aug 26 11:44 conf_hdfs_yarn.yml\r\n",
      "-rw-r--r--  1 root root  188 Aug 26 11:44 conf_namenode_bootstrapstandby.yml\r\n"
     ]
    }
   ],
   "source": [
    "playbook_dir = os.path.join(work_dir, 'hadoop/playbooks')\n",
    "!ls -la {playbook_dir} | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebbf8e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebc0a6-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebbe76-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "これでPlaybookの準備はOK。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebc0a6-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebc1be-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebbf8e-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## Notebook用変数の定義\n",
    "\n",
    "Notebook上の各セルでスクリプトを実行する際に必要な変数を定義・・・"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebc1be-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebc2e0-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebc0a6-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "active_namenode_host = 'XXX.XXX.XXX.70'\n",
      "standby_namenode_host = 'XXX.XXX.XXX.71'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "namenode_stdout = !ansible hadoop_namenode_primary -m ping -l {target_group}\n",
    "active_namenode_host = map(lambda l: l.split()[0], filter(lambda l: 'SUCCESS' in l, namenode_stdout))[0]\n",
    "namenode_stdout = !ansible hadoop_namenode_backup -m ping -l {target_group}\n",
    "standby_namenode_host = map(lambda l: l.split()[0], filter(lambda l: 'SUCCESS' in l, namenode_stdout))[0]\n",
    "print(\"active_namenode_host = '%s'\\nstandby_namenode_host = '%s'\\n\" % (active_namenode_host, standby_namenode_host))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebc2e0-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebc3f8-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebc1be-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## インストール対象マシンの確認\n",
    "\n",
    "疎通確認・・・"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebc3f8-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebc51a-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebc2e0-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.71 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.70 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.73 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.112 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.113 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.114 | SUCCESS => {\n",
      "    \"changed\": false, \n",
      "    \"ping\": \"pong\"\n",
      "}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible hadoop_all -m ping -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebc51a-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebc63c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebc3f8-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "# ソフトウェアのインストールと起動\n",
    "\n",
    "ソフトウェアは以下の手順でインストール・起動していきます。\n",
    "\n",
    "- OS設定/cgroupsインストール・起動\n",
    "- ZooKeeperインストール・起動\n",
    "- HDFSインストール・起動\n",
    "  - JournalNodeインストール・起動\n",
    "  - NameNodeインストール・フォーマット・起動\n",
    "  - DataNodeインストール・起動\n",
    "- YARNインストール・起動\n",
    "  - ResourceManagerインストール・起動\n",
    "  - NodeManagerインストール・起動\n",
    "  - MapReduceHistoryServerインストール・起動\n",
    "  - TimelineServiceインストール・起動\n",
    "- Tezインストール\n",
    "- HBaseインストール・起動\n",
    "- Hiveインストール\n",
    "- Sparkインストール・起動"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebc63c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebc754-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebc51a-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## インストール(OS/cgroups)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebc754-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebc876-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebc63c-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_all] **************************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [os : include] ************************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/os/tasks/limits.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72, XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [os : set_nofile_soft_limit] **********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [os : set_nofile_hard_limit] **********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [os : set_core_soft_limit] ************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [os : set_core_hard_limit] ************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [os : include] ************************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/os/tasks/thp.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72, XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [os : set_transparent_hugepage] *******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [os : include] ************************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/os/tasks/kernel.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72, XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [os : set_local_port_range] ***********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [os : set_somaxconn] ******************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [cgroups : include] *******************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/cgroups/tasks/install.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72, XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [cgroups : install_cgroups] ***********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=[u'libcgroup', u'libcgroup-devel'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=[u'libcgroup', u'libcgroup-devel'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=[u'libcgroup', u'libcgroup-devel'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=[u'libcgroup', u'libcgroup-devel'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=[u'libcgroup', u'libcgroup-devel'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=[u'libcgroup', u'libcgroup-devel'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=[u'libcgroup', u'libcgroup-devel'])\u001b[0m\n",
      "\n",
      "TASK [cgroups : include] *******************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/cgroups/tasks/conf.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72, XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [cgroups : create_direcoty_cgroups_scripts] *******************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [cgroups : copy_cgroups_scripts] ******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=cgroups.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=cgroups.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=cgroups.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=cgroups.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=cgroups.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=cgroups.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=cgroups.sh)\u001b[0m\n",
      "\n",
      "TASK [cgroups : copy_cgconfig.conf] ********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [cgroups : check_chkconfig_cgconfig] **************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [cgroups : set_on_to_cgconfig_of_chkconfig] *******************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [cgroups : started_cgconfig] **********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [cgroups : reboot] ********************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [cgroups : wait for SSH port down] ****************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70 -> localhost]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71 -> localhost]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72 -> localhost]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73 -> localhost]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112 -> localhost]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113 -> localhost]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114 -> localhost]\u001b[0m\n",
      "\n",
      "TASK [cgroups : wait for SSH port up] ******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72 -> XXX.XXX.XXX.1]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71 -> XXX.XXX.XXX.1]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70 -> XXX.XXX.XXX.1]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73 -> XXX.XXX.XXX.1]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112 -> XXX.XXX.XXX.1]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113 -> XXX.XXX.XXX.1]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114 -> XXX.XXX.XXX.1]\u001b[0m\n",
      "\n",
      "TASK [cgroups : started_cgconfig] **********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.112\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m14\u001b[0m   unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.113\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m14\u001b[0m   unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.114\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m14\u001b[0m   unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m14\u001b[0m   unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m14\u001b[0m   unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m14\u001b[0m   unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.73\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m14\u001b[0m   unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install-base.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebc876-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebc98e-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebc754-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "念のため、各NodeからMasterへの疎通確認。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebc98e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebcab0-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebc876-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\n",
      "PING XXX.XXX.XXX.70 (XXX.XXX.XXX.70) 56(84) bytes of data.\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=1 ttl=64 time=1.02 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=2 ttl=64 time=0.363 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=3 ttl=64 time=0.153 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=4 ttl=64 time=0.271 ms\n",
      "\n",
      "--- XXX.XXX.XXX.70 ping statistics ---\n",
      "4 packets transmitted, 4 received, 0% packet loss, time 3001ms\n",
      "rtt min/avg/max/mdev = 0.153/0.452/1.023/0.338 ms\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.70 | SUCCESS | rc=0 >>\n",
      "PING XXX.XXX.XXX.70 (XXX.XXX.XXX.70) 56(84) bytes of data.\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=1 ttl=64 time=0.031 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=2 ttl=64 time=0.048 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=3 ttl=64 time=0.012 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=4 ttl=64 time=0.031 ms\n",
      "\n",
      "--- XXX.XXX.XXX.70 ping statistics ---\n",
      "4 packets transmitted, 4 received, 0% packet loss, time 2999ms\n",
      "rtt min/avg/max/mdev = 0.012/0.030/0.048/0.013 ms\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.71 | SUCCESS | rc=0 >>\n",
      "PING XXX.XXX.XXX.70 (XXX.XXX.XXX.70) 56(84) bytes of data.\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=1 ttl=64 time=0.886 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=2 ttl=64 time=0.193 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=3 ttl=64 time=0.143 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=4 ttl=64 time=0.134 ms\n",
      "\n",
      "--- XXX.XXX.XXX.70 ping statistics ---\n",
      "4 packets transmitted, 4 received, 0% packet loss, time 3000ms\n",
      "rtt min/avg/max/mdev = 0.134/0.339/0.886/0.316 ms\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.73 | SUCCESS | rc=0 >>\n",
      "PING XXX.XXX.XXX.70 (XXX.XXX.XXX.70) 56(84) bytes of data.\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=1 ttl=64 time=0.586 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=2 ttl=64 time=0.261 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=3 ttl=64 time=0.156 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=4 ttl=64 time=0.133 ms\n",
      "\n",
      "--- XXX.XXX.XXX.70 ping statistics ---\n",
      "4 packets transmitted, 4 received, 0% packet loss, time 3000ms\n",
      "rtt min/avg/max/mdev = 0.133/0.284/0.586/0.180 ms\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.113 | SUCCESS | rc=0 >>\n",
      "PING XXX.XXX.XXX.70 (XXX.XXX.XXX.70) 56(84) bytes of data.\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=1 ttl=64 time=1.04 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=2 ttl=64 time=0.104 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=3 ttl=64 time=0.229 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=4 ttl=64 time=0.193 ms\n",
      "\n",
      "--- XXX.XXX.XXX.70 ping statistics ---\n",
      "4 packets transmitted, 4 received, 0% packet loss, time 3000ms\n",
      "rtt min/avg/max/mdev = 0.104/0.391/1.040/0.377 ms\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.112 | SUCCESS | rc=0 >>\n",
      "PING XXX.XXX.XXX.70 (XXX.XXX.XXX.70) 56(84) bytes of data.\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=1 ttl=64 time=0.452 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=2 ttl=64 time=0.122 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=3 ttl=64 time=0.230 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=4 ttl=64 time=0.201 ms\n",
      "\n",
      "--- XXX.XXX.XXX.70 ping statistics ---\n",
      "4 packets transmitted, 4 received, 0% packet loss, time 3000ms\n",
      "rtt min/avg/max/mdev = 0.122/0.251/0.452/0.122 ms\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.114 | SUCCESS | rc=0 >>\n",
      "PING XXX.XXX.XXX.70 (XXX.XXX.XXX.70) 56(84) bytes of data.\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=1 ttl=64 time=0.675 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=2 ttl=64 time=0.215 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=3 ttl=64 time=0.226 ms\n",
      "64 bytes from XXX.XXX.XXX.70: icmp_seq=4 ttl=64 time=0.257 ms\n",
      "\n",
      "--- XXX.XXX.XXX.70 ping statistics ---\n",
      "4 packets transmitted, 4 received, 0% packet loss, time 3000ms\n",
      "rtt min/avg/max/mdev = 0.215/0.343/0.675/0.192 ms\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible hadoop_all -a 'ping -c 4 {active_namenode_host}' -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebcab0-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebcbc8-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebc98e-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## ZooKeeper構築\n",
    "\n",
    "### ZooKeeperのインストール"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebcbc8-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebccea-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebcab0-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_zookeeperserver] **************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using get_url or uri module rather than running wget\n",
      "\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/repo.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [base : install_hdp_repo] *************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/conf.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [base : create_hadoop_conf_dir] *******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [base : copy_conf_files] **************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hosts.list)\u001b[0m\n",
      "\n",
      "TASK [base : copy_secure_conf_files] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=container-executor.cfg) \u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : include] **********************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/zookeeper_server/tasks/install.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : install_zookeeper-server_packages] ********************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=[u'zookeeper-server'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=[u'zookeeper-server'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=[u'zookeeper-server'])\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : include] **********************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/zookeeper_server/tasks/config.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : create_config_dir] ************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : include] **********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : copy_jaas_conf] ***************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : copy_config_files] ************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=zoo.cfg)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=zoo.cfg)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=zoo.cfg)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=log4j.properties)\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : create_zookeeper_data_dir] ****************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : create_zookeeper_data_dir] ****************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : copy_myid_file] ***************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : create_symbolic_link_to_/usr/bin] *********************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=zookeeper-server)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=zookeeper-server)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=zookeeper-server)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=zookeeper-client)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=zookeeper-client)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=zookeeper-client)\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : fix_init_scripts] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=zookeeper-server) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=zookeeper-server) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=zookeeper-server) \u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : create_symbolic_link_to_/etc/init.d] ******************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=zookeeper-server) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=zookeeper-server) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=zookeeper-server) \u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : create_symbolic_link_to_/etc/init.d] ******************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=zookeeper-server)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=zookeeper-server)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=zookeeper-server)\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : copy_zookeeper-env.sh] ********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=zookeeper-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=zookeeper-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=zookeeper-env.sh)\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : create_zookeeper_log_dir] *****************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [zookeeper_server : create_zookeeper_log_dir] *****************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m25\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m14\u001b[0m   unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m25\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m14\u001b[0m   unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m25\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m14\u001b[0m   unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook {playbook_dir}/install_zookeeper.yml -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebccea-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebce02-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebcbc8-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### ZooKeeperの起動"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebce02-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebcf1a-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebccea-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_zookeeperserver] **************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [start_zookeeper-server] **************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook {playbook_dir}/start_zookeeper-server.yml -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebcf1a-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebd032-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebce02-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### ZooKeeperの動作確認"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebd032-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebd154-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebcf1a-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "ZK上で *ls /* を実行した場合に *zookeeper* が返ってくればOK。(すでにZK上でサービスが動作している場合は他のファイルがあるかも)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebd154-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebd26c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebd032-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'XXX.XXX.XXX.71'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zknode_stdout = !ansible hadoop_zookeeperserver -m ping -l {target_group}\n",
    "zknodes = map(lambda l: l.split()[0], filter(lambda l: 'SUCCESS' in l, zknode_stdout))[0]\n",
    "zknodes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebd26c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebd38e-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebd154-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "`zk-shell` コマンド経由で動作確認を行う。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebd38e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebd4a6-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebd26c-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "zookeeper\r\n"
     ]
    }
   ],
   "source": [
    "!zk-shell { zknodes } --run-once \"ls /\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "97ebd4a6-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebd5be-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebd38e-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## HDFS構築・起動\n",
    "\n",
    "### JournalNode\n",
    "\n",
    "#### JournalNodeのインストール"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebd5be-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebd6d6-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebd4a6-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_journalnode] ******************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/repo.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [base : install_hdp_repo] *************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/conf.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [base : create_hadoop_conf_dir] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [base : copy_conf_files] **************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hosts.list)\u001b[0m\n",
      "\n",
      "TASK [base : copy_secure_conf_files] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=container-executor.cfg) \u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [journalnode : include] ***************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/journalnode/tasks/install.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [journalnode : install_journalnode_packages] ******************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=[u'hadoop-hdfs-journalnode'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=[u'hadoop-hdfs-journalnode'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=[u'hadoop-hdfs-journalnode'])\u001b[0m\n",
      "\n",
      "TASK [journalnode : include] ***************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/journalnode/tasks/config.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71, XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [journalnode : create_journalnode_data_dir] *******************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [journalnode : fix_init_scripts] ******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-journalnode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-hdfs-journalnode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-journalnode)\u001b[0m\n",
      "\n",
      "TASK [journalnode : create_symbolic_link_to/etc/init.d] ************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-hdfs-journalnode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-journalnode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-journalnode)\u001b[0m\n",
      "\n",
      "TASK [journalnode : create_hdfs_log_dir] ***************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [journalnode : copy_journalnode_defaults_file] ****************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-journalnode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-journalnode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-hdfs-journalnode)\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m19\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m6\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m19\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m6\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m19\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m6\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install_journalnode.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebd6d6-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebd7f8-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebd5be-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "#### JournalNodeの起動"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebd7f8-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebd910-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebd6d6-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_journalnode] ******************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [start_hadoop-hdfs-journalnode] *******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/start_journalnode.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebd910-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebda28-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebd7f8-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### NameNode\n",
    "\n",
    "#### NameNodeのインストール"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebda28-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebdb4a-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebd910-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_namenode] *********************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/repo.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [base : install_hdp_repo] *************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/conf.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [base : create_hadoop_conf_dir] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [base : copy_conf_files] **************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hosts.list)\u001b[0m\n",
      "\n",
      "TASK [base : copy_secure_conf_files] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=container-executor.cfg) \u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [namenode : include] ******************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/namenode/tasks/install.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [namenode : install_namenode_packages] ************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=[u'hadoop-hdfs-namenode', u'hadoop-hdfs-zkfc'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=[u'hadoop-hdfs-namenode', u'hadoop-hdfs-zkfc'])\u001b[0m\n",
      "\n",
      "TASK [namenode : include] ******************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/namenode/tasks/config.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [namenode : create_namenode_data_dir] *************************************\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{dfs_namenode_name_dirs}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{dfs_namenode_name_dirs}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=/hadoop/data/dfs/namedir)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=/hadoop/data/dfs/namedir)\u001b[0m\n",
      "\n",
      "TASK [namenode : fix_init_scripts] *********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\n",
      "TASK [namenode : create_symbolic_link_to/etc/init.d] ***************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\n",
      "TASK [namenode : create_symbolic_link_to/usr/hdp/current/hadoop] ***************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [namenode : create_symbolic_link_to/usr/hdp/current/hadoop-hdfs-zkfc] *****\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [namenode : deploy kick script for balancer.] *****************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hdfs-balancer.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hdfs-balancer.sh)\u001b[0m\n",
      "\n",
      "TASK [namenode : create_hdfs_log_dir] ******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [namenode : create_hdfs_pid_dir] ******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [namenode : create_jaas_conf] *********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [namenode : copy_defaults_file] *******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m9\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m9\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install_namenode.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebdb4a-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebdc6c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebda28-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "#### Primary側のNameNodeのフォーマット"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebdc6c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebdda2-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebdb4a-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_namenode_primary] *************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/repo.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [base : install_hdp_repo] *************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/conf.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [base : create_hadoop_conf_dir] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [base : copy_conf_files] **************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hosts.list)\u001b[0m\n",
      "\n",
      "TASK [base : copy_secure_conf_files] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=container-executor.cfg) \u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [namenode : include] ******************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/namenode/tasks/install.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [namenode : install_namenode_packages] ************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=[u'hadoop-hdfs-namenode', u'hadoop-hdfs-zkfc'])\u001b[0m\n",
      "\n",
      "TASK [namenode : include] ******************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/namenode/tasks/config.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [namenode : create_namenode_data_dir] *************************************\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{dfs_namenode_name_dirs}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=/hadoop/data/dfs/namedir)\u001b[0m\n",
      "\n",
      "TASK [namenode : fix_init_scripts] *********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\n",
      "TASK [namenode : create_symbolic_link_to/etc/init.d] ***************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\n",
      "TASK [namenode : create_symbolic_link_to/usr/hdp/current/hadoop] ***************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [namenode : create_symbolic_link_to/usr/hdp/current/hadoop-hdfs-zkfc] *****\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [namenode : deploy kick script for balancer.] *****************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hdfs-balancer.sh)\u001b[0m\n",
      "\n",
      "TASK [namenode : create_hdfs_log_dir] ******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [namenode : create_hdfs_pid_dir] ******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [namenode : create_jaas_conf] *********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [namenode : copy_defaults_file] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\n",
      "TASK [namenode_format : include] ***********************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/namenode_format/tasks/config.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [namenode_format : check_format_state] ************************************\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{dfs_namenode_name_dirs}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=/hadoop/data/dfs/namedir)\u001b[0m\n",
      "\n",
      "TASK [namenode_format : FORMAT_namenode] ***************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [namenode_format : FORMAT_ZKFC] *******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m27\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m2\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/format_namenode.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebdda2-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebdece-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebdc6c-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "#### Active側のNameNodeを起動"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebdece-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebdfe6-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebdda2-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_namenode] *********************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [start_hadoop-hdfs-zkfc] **************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [start_hadoop-hdfs-namenode] **********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m3\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m2\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/start_namenode.yml -l { active_namenode_host }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebdfe6-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebe0fe-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebdece-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "#### Backup側のNameNodeをPrimary側に同期させる"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebe0fe-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebe216-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebdfe6-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_namenode] *********************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/repo.yml for XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [base : install_hdp_repo] *************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/conf.yml for XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [base : create_hadoop_conf_dir] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [base : copy_conf_files] **************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hosts.list)\u001b[0m\n",
      "\n",
      "TASK [base : copy_secure_conf_files] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=container-executor.cfg) \u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [namenode : include] ******************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/namenode/tasks/install.yml for XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [namenode : install_namenode_packages] ************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=[u'hadoop-hdfs-namenode', u'hadoop-hdfs-zkfc'])\u001b[0m\n",
      "\n",
      "TASK [namenode : include] ******************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/namenode/tasks/config.yml for XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [namenode : create_namenode_data_dir] *************************************\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{dfs_namenode_name_dirs}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=/hadoop/data/dfs/namedir)\u001b[0m\n",
      "\n",
      "TASK [namenode : fix_init_scripts] *********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\n",
      "TASK [namenode : create_symbolic_link_to/etc/init.d] ***************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\n",
      "TASK [namenode : create_symbolic_link_to/usr/hdp/current/hadoop] ***************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [namenode : create_symbolic_link_to/usr/hdp/current/hadoop-hdfs-zkfc] *****\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [namenode : deploy kick script for balancer.] *****************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hdfs-balancer.sh)\u001b[0m\n",
      "\n",
      "TASK [namenode : create_hdfs_log_dir] ******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [namenode : create_hdfs_pid_dir] ******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [namenode : create_jaas_conf] *********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [namenode : copy_defaults_file] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-zkfc)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-hdfs-namenode)\u001b[0m\n",
      "\n",
      "TASK [namenode_bootstrapstandby : include] *************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/namenode_bootstrapstandby/tasks/config.yml for XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [namenode_bootstrapstandby : BootstrapStandby_namenode] *******************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m25\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/conf_namenode_bootstrapstandby.yml -l { standby_namenode_host }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebe216-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebe338-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebe0fe-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "#### Standby側のNameNodeを起動させる"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebe338-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebe45a-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebe216-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_namenode] *********************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [start_hadoop-hdfs-zkfc] **************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [start_hadoop-hdfs-namenode] **********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m3\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m2\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/start_namenode.yml -l { standby_namenode_host }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebe45a-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebe57c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebe338-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### DataNode(SlaveNode)\n",
    "\n",
    "Slaveノードの構築の際に、DataNode, NodeManagerがインストールされる\n",
    "\n",
    "#### DataNode/NodeManagerのインストール"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebe57c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebe694-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebe45a-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "*2016/08/06*\n",
    "\n",
    "Notebookだけだと /hadoop/ のパーミッションが厳しすぎ(hdfsユーザのみアクセス可能)、YARN NodeManagerが /hadoop/tmp にアクセスすることができない。VMだったのでDisk mount用Prerequisiteを適用しなかったことが要因。\n",
    "\n",
    "とりあえず応急措置として、fileモジュールを実行しておく。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebe694-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebe7b6-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebe57c-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;33mXXX.XXX.XXX.113 | SUCCESS => {\n",
      "    \"changed\": true, \n",
      "    \"gid\": 0, \n",
      "    \"group\": \"root\", \n",
      "    \"mode\": \"0755\", \n",
      "    \"owner\": \"root\", \n",
      "    \"path\": \"/hadoop\", \n",
      "    \"size\": 4096, \n",
      "    \"state\": \"directory\", \n",
      "    \"uid\": 0\n",
      "}\u001b[0m\n",
      "\u001b[0;33mXXX.XXX.XXX.73 | SUCCESS => {\n",
      "    \"changed\": true, \n",
      "    \"gid\": 0, \n",
      "    \"group\": \"root\", \n",
      "    \"mode\": \"0755\", \n",
      "    \"owner\": \"root\", \n",
      "    \"path\": \"/hadoop\", \n",
      "    \"size\": 4096, \n",
      "    \"state\": \"directory\", \n",
      "    \"uid\": 0\n",
      "}\u001b[0m\n",
      "\u001b[0;33mXXX.XXX.XXX.112 | SUCCESS => {\n",
      "    \"changed\": true, \n",
      "    \"gid\": 0, \n",
      "    \"group\": \"root\", \n",
      "    \"mode\": \"0755\", \n",
      "    \"owner\": \"root\", \n",
      "    \"path\": \"/hadoop\", \n",
      "    \"size\": 4096, \n",
      "    \"state\": \"directory\", \n",
      "    \"uid\": 0\n",
      "}\u001b[0m\n",
      "\u001b[0;33mXXX.XXX.XXX.114 | SUCCESS => {\n",
      "    \"changed\": true, \n",
      "    \"gid\": 0, \n",
      "    \"group\": \"root\", \n",
      "    \"mode\": \"0755\", \n",
      "    \"owner\": \"root\", \n",
      "    \"path\": \"/hadoop\", \n",
      "    \"size\": 4096, \n",
      "    \"state\": \"directory\", \n",
      "    \"uid\": 0\n",
      "}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible -b -m file -a 'path=/hadoop state=directory owner=root group=root mode=0755' hadoop_slavenode -l {target_group}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebe7b6-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebe8ce-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebe694-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_slavenode] ********************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/repo.yml for XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [base : install_hdp_repo] *************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/conf.yml for XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [base : create_hadoop_conf_dir] *******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [base : copy_conf_files] **************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hosts.list)\u001b[0m\n",
      "\n",
      "TASK [base : copy_secure_conf_files] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114] => (item=container-executor.cfg) \u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.73, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using get_url or uri module rather than running wget\n",
      "\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [slavenode : include] *****************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/slavenode/tasks/install.yml for XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [slavenode : install_slavenode_packages] **********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=[u'hadoop-hdfs-datanode', u'hadoop-yarn-nodemanager', u'hadoop-mapreduce'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=[u'hadoop-hdfs-datanode', u'hadoop-yarn-nodemanager', u'hadoop-mapreduce'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=[u'hadoop-hdfs-datanode', u'hadoop-yarn-nodemanager', u'hadoop-mapreduce'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=[u'hadoop-hdfs-datanode', u'hadoop-yarn-nodemanager', u'hadoop-mapreduce'])\u001b[0m\n",
      "\n",
      "TASK [slavenode : include] *****************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/slavenode/tasks/config.yml for XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [slavenode : create_datanode_data_dir] ************************************\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{dfs_datanode_data_dirs}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{dfs_datanode_data_dirs}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{dfs_datanode_data_dirs}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{dfs_datanode_data_dirs}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=/hadoop/data01/dfs/datadir)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=/hadoop/data01/dfs/datadir)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=/hadoop/data01/dfs/datadir)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=/hadoop/data01/dfs/datadir)\u001b[0m\n",
      "\n",
      "TASK [slavenode : fix_init_scripts] ********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item={u'path': u'hadoop-hdfs', u'name': u'hadoop-hdfs-datanode'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item={u'path': u'hadoop-hdfs', u'name': u'hadoop-hdfs-datanode'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item={u'path': u'hadoop-hdfs', u'name': u'hadoop-hdfs-datanode'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item={u'path': u'hadoop-yarn', u'name': u'hadoop-yarn-nodemanager'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item={u'path': u'hadoop-yarn', u'name': u'hadoop-yarn-nodemanager'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item={u'path': u'hadoop-yarn', u'name': u'hadoop-yarn-nodemanager'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item={u'path': u'hadoop-hdfs', u'name': u'hadoop-hdfs-datanode'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item={u'path': u'hadoop-yarn', u'name': u'hadoop-yarn-nodemanager'})\u001b[0m\n",
      "\n",
      "TASK [slavenode : create_symbolic_link_to/etc/init.d] **************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item={u'path': u'hadoop-hdfs', u'name': u'hadoop-hdfs-datanode'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item={u'path': u'hadoop-hdfs', u'name': u'hadoop-hdfs-datanode'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item={u'path': u'hadoop-hdfs', u'name': u'hadoop-hdfs-datanode'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item={u'path': u'hadoop-yarn', u'name': u'hadoop-yarn-nodemanager'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item={u'path': u'hadoop-yarn', u'name': u'hadoop-yarn-nodemanager'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item={u'path': u'hadoop-yarn', u'name': u'hadoop-yarn-nodemanager'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item={u'path': u'hadoop-hdfs', u'name': u'hadoop-hdfs-datanode'})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item={u'path': u'hadoop-yarn', u'name': u'hadoop-yarn-nodemanager'})\u001b[0m\n",
      "\n",
      "TASK [slavenode : create_holder_directory_for_hadoop_tmp_dir] ******************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [slavenode : create_yarn_pid_dir] *****************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [slavenode : create_hdfs_log_dir] *****************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [slavenode : create_yarn_log_dir] *****************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [slavenode : copy_defaults_file] ******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hadoop-hdfs-datanode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hadoop-hdfs-datanode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hadoop-hdfs-datanode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hadoop-yarn-nodemanager)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hadoop-yarn-nodemanager)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hadoop-yarn-nodemanager)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hadoop-hdfs-datanode)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hadoop-yarn-nodemanager)\u001b[0m\n",
      "\n",
      "TASK [slavenode : ensure_version_link] *****************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.112\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m26\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m17\u001b[0m   unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.113\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m26\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m17\u001b[0m   unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.114\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m26\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m17\u001b[0m   unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.73\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m26\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m17\u001b[0m   unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install_slavenode.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebe8ce-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebe9f0-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebe7b6-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "#### DataNodeの起動"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebe9f0-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebeb08-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebe8ce-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_slavenode] ********************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [start_hadoop-hdfs-datanode] **********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.112\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.113\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.114\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.73\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/start_datanode.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebeb08-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebec2a-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebe9f0-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### HDFSの初期設定\n",
    "\n",
    "HDFS上に必要なディレクトリを作成する。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebec2a-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebed42-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebeb08-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_namenode_primary] *************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_base : include] *****************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/hdfs_base/tasks/config.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [hdfs_base : check_hdfs_root_permission] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_base : chmod_hdfs_root] *********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_base : check_exists_hfds_dir] ***************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item={u'owner': u'hdfs', u'path': u'/tmp', u'group': u'hadoop', u'mode': u'1777'})\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item={u'owner': u'hdfs', u'path': u'/user', u'group': u'hadoop', u'mode': u'1777'})\u001b[0m\n",
      "\n",
      "TASK [hdfs_base : create_hdfs_dir] *********************************************\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{check_exists_hdfs_dir.results}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:05:36.634892', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/tmp', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /tmp', u'stderr': u\"ls: `/tmp': No such file or directory\", u'delta': u'0:00:03.108849', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /tmp', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:05:33.526043', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:05:39.933775', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/user', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /user', u'stderr': u\"ls: `/user': No such file or directory\", u'delta': u'0:00:02.851559', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /user', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:05:37.082216', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\n",
      "TASK [hdfs_base : change_owner_and_group_of_hdfs_dir] **************************\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{check_exists_hdfs_dir.results}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:05:36.634892', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/tmp', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /tmp', u'stderr': u\"ls: `/tmp': No such file or directory\", u'delta': u'0:00:03.108849', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /tmp', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:05:33.526043', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:05:39.933775', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/user', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /user', u'stderr': u\"ls: `/user': No such file or directory\", u'delta': u'0:00:02.851559', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /user', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:05:37.082216', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\n",
      "TASK [hdfs_base : change_mode_of_hdfs_dir] *************************************\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{check_exists_hdfs_dir.results}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:05:36.634892', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/tmp', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /tmp', u'stderr': u\"ls: `/tmp': No such file or directory\", u'delta': u'0:00:03.108849', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /tmp', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:05:33.526043', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:05:39.933775', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/user', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /user', u'stderr': u\"ls: `/user': No such file or directory\", u'delta': u'0:00:02.851559', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /user', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:05:37.082216', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m13\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m4\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/conf_hdfs_base.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebed42-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebee5a-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebec2a-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "これでHDFS環境の準備は完了。以下のURLからNameNodeの状態が確認できる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebee5a-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebef7c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebed42-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://XXX.XXX.XXX.70:50070\n"
     ]
    }
   ],
   "source": [
    "print(\"http://%s:50070\" % active_namenode_host)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebef7c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebf08a-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebee5a-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## YARNのインストール・起動\n",
    "\n",
    "### ResourceManagerのインストール"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebf08a-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebf1ac-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebef7c-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_resourcemanager] **************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/repo.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [base : install_hdp_repo] *************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/conf.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [base : create_hadoop_conf_dir] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [base : copy_conf_files] **************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hosts.list)\u001b[0m\n",
      "\n",
      "TASK [base : copy_secure_conf_files] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=container-executor.cfg) \u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [resourcemanager : include] ***********************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/resourcemanager/tasks/install.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [resourcemanager : install_resourcemanager_packages] **********************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=[u'hadoop-yarn-resourcemanager', u'hadoop-mapreduce'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=[u'hadoop-yarn-resourcemanager', u'hadoop-mapreduce'])\u001b[0m\n",
      "\n",
      "TASK [resourcemanager : include] ***********************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/resourcemanager/tasks/config.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [resourcemanager : fix_init_scripts] **************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-yarn-resourcemanager)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-yarn-resourcemanager)\u001b[0m\n",
      "\n",
      "TASK [resourcemanager : create_symbolic_link_to/etc/init.d] ********************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-yarn-resourcemanager)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-yarn-resourcemanager)\u001b[0m\n",
      "\n",
      "TASK [resourcemanager : create_yarn_pid_dir] ***********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [resourcemanager : create_yarn_log_dir] ***********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [resourcemanager : copy_default_file] *************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-yarn-resourcemanager)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-yarn-resourcemanager)\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m19\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m6\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m19\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m6\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install_resourcemanager.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebf1ac-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebf2c4-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebf08a-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### NodeManagerのインストール\n",
    "\n",
    "*HDFS内の手順『SlaveNodeのインストール』ですでにインストールされているため不要*\n",
    "\n",
    "### HDFSの準備\n",
    "\n",
    "HDFSにYARNの動作に必要なディレクトリを作成する"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebf2c4-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebf3dc-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebf1ac-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_namenode_primary] *************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_yarn : include] *****************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/hdfs_yarn/tasks/config.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [hdfs_yarn : check_exists_hfds_dir] ***************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item={u'owner': u'hdfs', u'path': u'/var', u'group': u'hadoop', u'mode': u'1777'})\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item={u'owner': u'hdfs', u'path': u'/var/log', u'group': u'hadoop', u'mode': u'1777'})\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item={u'owner': u'mapred', u'path': u'/mapred', u'group': u'hadoop', u'mode': u'755'})\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item={u'owner': u'mapred', u'path': u'/mapred/staging', u'group': u'hadoop', u'mode': u'1777'})\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item={u'owner': u'mapred', u'path': u'/mapred/system', u'group': u'hadoop', u'mode': u'700'})\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item={u'owner': u'yarn', u'path': u'/tmp/hadoop-yarn', u'group': u'hadoop', u'mode': u'1777'})\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item={u'owner': u'mapred', u'path': u'/tmp/hadoop-yarn/staging', u'group': u'hadoop', u'mode': u'1777'})\u001b[0m\n",
      "\n",
      "TASK [hdfs_yarn : create_hdfs_dir] *********************************************\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{check_exists_hdfs_dir.results}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:13.492510', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/var', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /var', u'stderr': u\"ls: `/var': No such file or directory\", u'delta': u'0:00:04.359122', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /var', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:09.133388', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:18.288142', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/var/log', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /var/log', u'stderr': u\"ls: `/var/log': No such file or directory\", u'delta': u'0:00:04.400405', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /var/log', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:13.887737', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:22.037698', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/mapred', u'group': u'hadoop', u'mode': u'755'}, u'cmd': u'hdfs dfs -ls /mapred', u'stderr': u\"ls: `/mapred': No such file or directory\", u'delta': u'0:00:03.359806', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /mapred', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:18.677892', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:26.581420', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/mapred/staging', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /mapred/staging', u'stderr': u\"ls: `/mapred/staging': No such file or directory\", u'delta': u'0:00:04.140930', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /mapred/staging', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:22.440490', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:30.747899', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/mapred/system', u'group': u'hadoop', u'mode': u'700'}, u'cmd': u'hdfs dfs -ls /mapred/system', u'stderr': u\"ls: `/mapred/system': No such file or directory\", u'delta': u'0:00:03.798698', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /mapred/system', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:26.949201', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:36.890592', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'yarn', u'path': u'/tmp/hadoop-yarn', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /tmp/hadoop-yarn', u'stderr': u\"ls: `/tmp/hadoop-yarn': No such file or directory\", u'delta': u'0:00:05.776944', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /tmp/hadoop-yarn', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:31.113648', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:41.520431', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/tmp/hadoop-yarn/staging', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /tmp/hadoop-yarn/staging', u'stderr': u\"ls: `/tmp/hadoop-yarn/staging': No such file or directory\", u'delta': u'0:00:04.162352', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /tmp/hadoop-yarn/staging', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:37.358079', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\n",
      "TASK [hdfs_yarn : change_owner_and_group_of_hdfs_dir] **************************\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{check_exists_hdfs_dir.results}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:13.492510', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/var', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /var', u'stderr': u\"ls: `/var': No such file or directory\", u'delta': u'0:00:04.359122', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /var', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:09.133388', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:18.288142', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/var/log', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /var/log', u'stderr': u\"ls: `/var/log': No such file or directory\", u'delta': u'0:00:04.400405', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /var/log', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:13.887737', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:22.037698', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/mapred', u'group': u'hadoop', u'mode': u'755'}, u'cmd': u'hdfs dfs -ls /mapred', u'stderr': u\"ls: `/mapred': No such file or directory\", u'delta': u'0:00:03.359806', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /mapred', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:18.677892', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:26.581420', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/mapred/staging', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /mapred/staging', u'stderr': u\"ls: `/mapred/staging': No such file or directory\", u'delta': u'0:00:04.140930', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /mapred/staging', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:22.440490', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:30.747899', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/mapred/system', u'group': u'hadoop', u'mode': u'700'}, u'cmd': u'hdfs dfs -ls /mapred/system', u'stderr': u\"ls: `/mapred/system': No such file or directory\", u'delta': u'0:00:03.798698', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /mapred/system', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:26.949201', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:36.890592', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'yarn', u'path': u'/tmp/hadoop-yarn', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /tmp/hadoop-yarn', u'stderr': u\"ls: `/tmp/hadoop-yarn': No such file or directory\", u'delta': u'0:00:05.776944', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /tmp/hadoop-yarn', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:31.113648', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:41.520431', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/tmp/hadoop-yarn/staging', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /tmp/hadoop-yarn/staging', u'stderr': u\"ls: `/tmp/hadoop-yarn/staging': No such file or directory\", u'delta': u'0:00:04.162352', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /tmp/hadoop-yarn/staging', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:37.358079', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\n",
      "TASK [hdfs_yarn : change_mode_of_hdfs_dir] *************************************\n",
      "\u001b[0;35m[DEPRECATION WARNING]: Using bare variables is deprecated. Update your playbooks\n",
      " so that the environment value uses the full variable syntax \n",
      "('{{check_exists_hdfs_dir.results}}').\n",
      "This feature will be removed in a future \n",
      "release. Deprecation warnings can be disabled by setting \n",
      "deprecation_warnings=False in ansible.cfg.\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:13.492510', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/var', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /var', u'stderr': u\"ls: `/var': No such file or directory\", u'delta': u'0:00:04.359122', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /var', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:09.133388', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:18.288142', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'hdfs', u'path': u'/var/log', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /var/log', u'stderr': u\"ls: `/var/log': No such file or directory\", u'delta': u'0:00:04.400405', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /var/log', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:13.887737', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:22.037698', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/mapred', u'group': u'hadoop', u'mode': u'755'}, u'cmd': u'hdfs dfs -ls /mapred', u'stderr': u\"ls: `/mapred': No such file or directory\", u'delta': u'0:00:03.359806', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /mapred', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:18.677892', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:26.581420', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/mapred/staging', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /mapred/staging', u'stderr': u\"ls: `/mapred/staging': No such file or directory\", u'delta': u'0:00:04.140930', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /mapred/staging', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:22.440490', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:30.747899', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/mapred/system', u'group': u'hadoop', u'mode': u'700'}, u'cmd': u'hdfs dfs -ls /mapred/system', u'stderr': u\"ls: `/mapred/system': No such file or directory\", u'delta': u'0:00:03.798698', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /mapred/system', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:26.949201', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:36.890592', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'yarn', u'path': u'/tmp/hadoop-yarn', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /tmp/hadoop-yarn', u'stderr': u\"ls: `/tmp/hadoop-yarn': No such file or directory\", u'delta': u'0:00:05.776944', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /tmp/hadoop-yarn', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:31.113648', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item={'_ansible_item_result': True, u'end': u'2016-08-26 12:09:41.520431', '_ansible_no_log': False, u'stdout': u'', u'changed': False, u'rc': 1, 'item': {u'owner': u'mapred', u'path': u'/tmp/hadoop-yarn/staging', u'group': u'hadoop', u'mode': u'1777'}, u'cmd': u'hdfs dfs -ls /tmp/hadoop-yarn/staging', u'stderr': u\"ls: `/tmp/hadoop-yarn/staging': No such file or directory\", u'delta': u'0:00:04.162352', 'invocation': {'module_name': u'command', u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'_raw_params': u'hdfs dfs -ls /tmp/hadoop-yarn/staging', u'removes': None, u'warn': True, u'chdir': None}}, 'stdout_lines': [], 'failed_when_result': False, u'start': u'2016-08-26 12:09:37.358079', u'warnings': [], 'failed': False})\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m11\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m3\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/conf_hdfs_yarn.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebf3dc-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebf4f4-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebf2c4-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### MapReduceHistoryServerのインストール"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebf4f4-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebf60c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebf3dc-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_mapreduce_historyserver] ******************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/repo.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [base : install_hdp_repo] *************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/conf.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [base : create_hadoop_conf_dir] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [base : copy_conf_files] **************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hosts.list)\u001b[0m\n",
      "\n",
      "TASK [base : copy_secure_conf_files] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=container-executor.cfg) \u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [mapreduce_history : include] *********************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/mapreduce_history/tasks/install.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [mapreduce_history : install_mapreduce-historyserver_packages] ************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=[u'hadoop-mapreduce-historyserver'])\u001b[0m\n",
      "\n",
      "TASK [mapreduce_history : include] *********************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/mapreduce_history/tasks/config.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [mapreduce_history : fix_init_scripts] ************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-mapreduce-historyserver)\u001b[0m\n",
      "\n",
      "TASK [mapreduce_history : create_symbolic_link_to/etc/init.d] ******************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-mapreduce-historyserver)\u001b[0m\n",
      "\n",
      "TASK [mapreduce_history : create_symbolic_link_to/usr/hdp/current/hadoop-mapreduce] ***\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [mapreduce_history : create_pid_dir] **************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [mapreduce_history : create_log_dir] **************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [mapreduce_history : copy_default_file] ***********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-mapreduce-historyserver)\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m20\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m7\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install_mapreduce_history.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebf60c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebf724-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebf4f4-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### TimelineServiceのインストール"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebf724-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebf83c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebf60c-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_timelineservice] **************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/repo.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [base : install_hdp_repo] *************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/conf.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [base : create_hadoop_conf_dir] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [base : copy_conf_files] **************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=hosts.list)\u001b[0m\n",
      "\n",
      "TASK [base : copy_secure_conf_files] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72] => (item=container-executor.cfg) \u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [timelineservice : include] ***********************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/timelineservice/tasks/install.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [timelineservice : install_timelineserver_packages] ***********************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=[u'hadoop-yarn-timelineserver'])\u001b[0m\n",
      "\n",
      "TASK [timelineservice : include] ***********************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/timelineservice/tasks/config.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [timelineservice : fix_init_scripts] **************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-yarn-timelineserver)\u001b[0m\n",
      "\n",
      "TASK [timelineservice : create_symbolic_link_to/etc/init.d] ********************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-yarn-timelineserver)\u001b[0m\n",
      "\n",
      "TASK [timelineservice : create_symbolic_link_to/usr/hdp/current/hadoop-yarn] ***\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [timelineservice : create_yarn_pid_dir] ***********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [timelineservice : create_yarn_log_dir] ***********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [timelineservice : create_timeline_dir] ***********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [timelineservice : copy_default_file] *************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hadoop-yarn-timelineserver)\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m21\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m8\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install_timelineservice.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebf83c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebf95e-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebf724-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### ResourceManagerを起動"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebf95e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebfa6c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebf83c-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_resourcemanager] **************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [start_hadoop-yarn-resourcemanager] ***************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/start_resourcemanager.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebfa6c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebfb8e-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebf95e-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### NodeManagerを起動"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebfb8e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebfcb0-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebfa6c-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_slavenode] ********************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [start_hadoop-yarn-nodemanager] *******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.112\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.113\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.114\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.73\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/start_nodemanager.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebfcb0-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebfdd2-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebfb8e-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### MapReduceHistoryServerを起動"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebfdd2-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ebfeea-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebfcb0-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_mapreduce_historyserver] ******************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [start_mapreduce-historyserver] *******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/start_mapreduce_historyserver.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ebfeea-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec0002-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebfdd2-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### TimelineServiceを起動"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec0002-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec011a-495a-11e8-ba7d-0242ac130002",
     "previous": "97ebfeea-495a-11e8-ba7d-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_timelineservice] **************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [start_hadoop-yarn-timelineserver] ****************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/start_timelineservice.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec011a-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec023c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec0002-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "これでYARNのインストールは完了。以下から動作状態を確認できる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec023c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec0354-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec011a-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://XXX.XXX.XXX.70:8088/cluster/apps\n"
     ]
    }
   ],
   "source": [
    "rmadmin_stdout = !ansible hadoop_resourcemanager -s -U yarn -m shell -a 'timeout 15 yarn rmadmin -getServiceState $(hostname)' -l {target_group}\n",
    "rmadmin_result = map(lambda l: l.split()[0], filter(lambda l: len(l) > 0, rmadmin_stdout))\n",
    "active_resourcemanager_host = rmadmin_result[rmadmin_result.index(\"active\") - 1]\n",
    "print(\"http://%s:8088/cluster/apps\" % active_resourcemanager_host)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec0354-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec0476-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec023c-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## Tez"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec0476-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec058e-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec0354-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_namenode_primary] *************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_tez : include] ******************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/hdfs_tez/tasks/config.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [hdfs_tez : check_exists_hfds_dir] ****************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_tez : create_hdfs_dir] **********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_tez : change_owner_and_group_of_hdfs_dir] ***************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_tez : change_mode_of_hdfs_dir] **************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m11\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m3\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/conf_hdfs_tez.yml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec058e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec06a6-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec0476-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_tez] **************************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [tez : include] ***********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/tez/tasks/install.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [tez : install_tez_packages] **********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=[u'tez'])\u001b[0m\n",
      "\n",
      "TASK [tez : check_tez_packages_on_HDFS] ****************************************\n",
      "\u001b[0;31mfatal: [XXX.XXX.XXX.72]: FAILED! => {\"changed\": false, \"cmd\": \"hdfs dfs -ls /apps/tez/tez.tar.gz\", \"delta\": \"0:00:05.822343\", \"end\": \"2016-08-26 12:16:44.591093\", \"failed\": true, \"rc\": 1, \"start\": \"2016-08-26 12:16:38.768750\", \"stderr\": \"ls: `/apps/tez/tez.tar.gz': No such file or directory\", \"stdout\": \"\", \"stdout_lines\": [], \"warnings\": []}\u001b[0m\n",
      "\u001b[0;36m...ignoring\u001b[0m\n",
      "\n",
      "TASK [tez : copy_tez_packages_to_HDFS] *****************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [tez : change_owner_of_tez_packages] **************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [tez : change_mode_of_tez_packages] ***************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [tez : include] ***********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/tez/tasks/config.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [tez : create_tez_conf_dir] ***********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [tez : copy_tez_conf] *****************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=tez-site.xml)\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m10\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m6\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/conf_tez.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec06a6-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec07be-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec058e-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## HBase\n",
    "\n",
    "### インストール\n",
    "\n",
    "Masterをインストールする・・・"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec07be-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec08e0-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec06a6-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_hbase_master] *****************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/repo.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [base : install_hdp_repo] *************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/conf.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [base : create_hadoop_conf_dir] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [base : copy_conf_files] **************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71] => (item=hosts.list)\u001b[0m\n",
      "\n",
      "TASK [base : copy_secure_conf_files] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71] => (item=container-executor.cfg) \u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [hbase_master : include] **************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/hbase_master/tasks/install.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [hbase_master : install_hbase-master_packages] ****************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=[u'hbase-master'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=[u'hbase-master'])\u001b[0m\n",
      "\n",
      "TASK [hbase_master : include] **************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/hbase_master/tasks/config.yml for XXX.XXX.XXX.70, XXX.XXX.XXX.71\u001b[0m\n",
      "\n",
      "TASK [hbase_master : create_hbase_conf_dir] ************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [hbase_master : include] **************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [hbase_master : copy_jaas_conf] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [hbase_master : copy_hbase_conf_files] ************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hbase-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hbase-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hbase-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hbase-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hbase-policy.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hbase-policy.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=regionservers)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=regionservers)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hadoop-metrics2-hbase.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hadoop-metrics2-hbase.properties)\u001b[0m\n",
      "\n",
      "TASK [hbase_master : copy_hbase_default_file] **********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hbase-master)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hbase-master)\u001b[0m\n",
      "\n",
      "TASK [hbase_master : fix_init_scripts] *****************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hbase-master)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hbase-master)\u001b[0m\n",
      "\n",
      "TASK [hbase_master : remove_template_variable_in_init_scripts] *****************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\n",
      "TASK [hbase_master : create_symbolic_link_to/etc/init.d] ***********************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hbase-master)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hbase-master)\u001b[0m\n",
      "\n",
      "TASK [hbase_master : create_hbase_log_dir] *************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hbase_master : deploy_service_test_script] *******************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71] => (item=hbase-service-test.rb)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70] => (item=hbase-service-test.rb)\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m22\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m8\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m22\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m8\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install_hbase_master.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec08e0-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec09f8-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec07be-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "RegionServerをインストールする・・・"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec09f8-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec0b10-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec08e0-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_hbase_regionserver] ***********************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/repo.yml for XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [base : install_hdp_repo] *************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/base/tasks/conf.yml for XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [base : create_hadoop_conf_dir] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [base : copy_conf_files] **************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112] => (item=hosts.list)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=core-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=hdfs-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=yarn-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=mapred-site.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=hadoop-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=yarn-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=mapred-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=hadoop-metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=hadoop-metrics2.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=capacity-scheduler.xml)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=hosts.exclude)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114] => (item=hosts.list)\u001b[0m\n",
      "\n",
      "TASK [base : copy_secure_conf_files] *******************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113] => (item=container-executor.cfg) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114] => (item=ssl-server.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114] => (item=ssl-client.xml) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114] => (item=zk-acl.txt) \u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114] => (item=container-executor.cfg) \u001b[0m\n",
      "\n",
      "TASK [base : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : include] ********************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/hbase_regionserver/tasks/install.yml for XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : install_hbase-regionserver_packages] ****************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=[u'hbase-regionserver'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=[u'hbase-regionserver'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=[u'hbase-regionserver'])\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=[u'hbase-regionserver'])\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : include] ********************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/hbase_regionserver/tasks/config.yml for XXX.XXX.XXX.73, XXX.XXX.XXX.112, XXX.XXX.XXX.113, XXX.XXX.XXX.114\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : create_hbase_conf_dir] ******************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : include] ********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : copy_jaas_conf] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : copy_hbase_conf_files] ******************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hbase-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hbase-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hbase-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hbase-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hbase-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hbase-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hbase-policy.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hbase-policy.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hbase-policy.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=regionservers)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=regionservers)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=regionservers)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hadoop-metrics2-hbase.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hadoop-metrics2-hbase.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hadoop-metrics2-hbase.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hbase-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hbase-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hbase-policy.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=regionservers)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hadoop-metrics2-hbase.properties)\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : copy_hbase_default_file] ****************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hbase-regionserver)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hbase-regionserver)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hbase-regionserver)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hbase-regionserver)\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : fix_init_scripts] ***********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hbase-regionserver)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hbase-regionserver)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hbase-regionserver)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hbase-regionserver)\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : remove_template_variable_in_init_scripts] ***********\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : create_symbolic_link_to/etc/init.d] *****************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=hbase-regionserver)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=hbase-regionserver)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=hbase-regionserver)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=hbase-regionserver)\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : create_hbase_log_dir] *******************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : deploy_graceful_stop_script_from_upstream] **********\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=graceful_stop.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=graceful_stop.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=graceful_stop.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=graceful_stop.sh)\u001b[0m\n",
      "\n",
      "TASK [hbase_regionserver : create_symbolic_link_for_graceful_stop_script] ******\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112] => (item=graceful_stop.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73] => (item=graceful_stop.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113] => (item=graceful_stop.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114] => (item=graceful_stop.sh)\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.112\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m9\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.113\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m9\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.114\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m9\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.73\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m23\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m9\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install_hbase_regionserver.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec0b10-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec0c1e-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec09f8-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### 起動\n",
    "\n",
    "Masterを起動する・・・"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec0c1e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec0d40-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec0b10-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_hbase_master] *****************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [start_hbase-master] ******************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.71]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.71\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/start_hbase_master.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec0d40-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec0e62-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec0c1e-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "RegionServerを起動する・・・"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec0e62-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec0f7a-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec0d40-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_hbase_regionserver] ***********************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "TASK [start_hbase-regionserver] ************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.112]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.73]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.113]\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.114]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.112\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.113\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.114\u001b[0m              : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\u001b[0;33mXXX.XXX.XXX.73\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m2\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/start_hbase_regionserver.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec0f7a-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec1092-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec0e62-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "これでインストール完了。\n",
    "以下から状態を確認できる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec1092-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec11aa-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec0f7a-495a-11e8-ba7d-0242ac130002"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('testvm001', 'XXX.XXX.XXX.70'), ('testvm002', 'XXX.XXX.XXX.71')]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hostname_stdout = !ansible -a hostname hadoop_hbase_master -l {target_group}\n",
    "hbase_masters = map(lambda m: (m[1], m[0].split()[0]), filter(lambda m: 'SUCCESS' in m[0], zip(hostname_stdout, hostname_stdout[1:])))\n",
    "hbase_masters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec11aa-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec12c2-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec1092-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://XXX.XXX.XXX.71:60010\n"
     ]
    }
   ],
   "source": [
    "from kazoo.client import KazooClient\n",
    "zk = KazooClient(hosts='%s:2181' % zknodes, read_only=True)\n",
    "zk.start()\n",
    "(master_result,v) = zk.get(\"/hbase/master\")\n",
    "zk.stop()\n",
    "hbase_master_host = filter(lambda m: m[0] in master_result, hbase_masters)[0][1]\n",
    "print(\"http://%s:60010\" % hbase_master_host)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec12c2-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec13e4-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec11aa-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## Hive\n",
    "\n",
    "Hiveのインストール・・・"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec13e4-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec14fc-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec12c2-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "*2016/08/06*\n",
    "\n",
    "Notebookだけだと /hadoop/ のパーミッションが厳しすぎ(hdfsユーザのみアクセス可能)、metastoreを作成することができない。VMだったのでDisk mount用Prerequisiteを適用しなかったことが要因。\n",
    "\n",
    "とりあえず応急措置として、タスクを実行しておく。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec14fc-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec1614-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec13e4-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;33mXXX.XXX.XXX.72 | SUCCESS => {\n",
      "    \"changed\": true, \n",
      "    \"gid\": 0, \n",
      "    \"group\": \"root\", \n",
      "    \"mode\": \"0755\", \n",
      "    \"owner\": \"root\", \n",
      "    \"path\": \"/hadoop\", \n",
      "    \"size\": 4096, \n",
      "    \"state\": \"directory\", \n",
      "    \"uid\": 0\n",
      "}\u001b[0m\n",
      "\u001b[0;33mXXX.XXX.XXX.72 | SUCCESS => {\n",
      "    \"changed\": true, \n",
      "    \"gid\": 0, \n",
      "    \"group\": \"root\", \n",
      "    \"mode\": \"0755\", \n",
      "    \"owner\": \"root\", \n",
      "    \"path\": \"/hadoop/data\", \n",
      "    \"size\": 4096, \n",
      "    \"state\": \"directory\", \n",
      "    \"uid\": 0\n",
      "}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible -b -m file -a 'path=/hadoop state=directory owner=root group=root mode=0755' hadoop_hive -l {target_group}\n",
    "!ansible -b -m file -a 'path=/hadoop/data state=directory owner=root group=root mode=0755' hadoop_hive -l {target_group}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec1614-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec172c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec14fc-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_hive] *************************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [hive : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/hive/tasks/install.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [hive : install_hive_packages] ********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=[u'hive'])\u001b[0m\n",
      "\n",
      "TASK [hive : include] **********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/hive/tasks/config.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [hive : create_hive_metastore_dir] ****************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [hive : create_hive_conf_dir] *********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [hive : include] **********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [hive : copy_hive_conf] ***************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hive-site.xml)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hive-log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=hive-exec-log4j.properties)\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m7\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m3\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install_hive.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec172c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec1844-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec1614-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "OK・・・！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec1844-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec195c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec172c-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## Sparkのインストール・起動\n",
    "\n",
    "Sparkをインストールする。\n",
    "\n",
    "### HDFSの準備"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec195c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec1a74-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec1844-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_namenode_primary] *************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_spark : include] ****************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/hdfs_spark/tasks/config.yml for XXX.XXX.XXX.70\u001b[0m\n",
      "\n",
      "TASK [hdfs_spark : check_exists_hfds_dir] **************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_spark : create_hdfs_dir] ********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_spark : change_owner_and_group_of_hdfs_dir] *************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "TASK [hdfs_spark : change_mode_of_hdfs_dir] ************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.70]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.70\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m11\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m3\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/conf_hdfs_spark.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec1a74-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec1b8c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec195c-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### Sparkのインストール"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec1b8c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec1ca4-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec1a74-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_spark] ************************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark_user : include] ****************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/spark_user/tasks/user.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [spark_user : create_spark_user] ******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : include] *********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/spark/tasks/install.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [spark : install_spark_packages] ******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=[u'spark', u'spark-python'])\u001b[0m\n",
      "\n",
      "TASK [spark : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/spark/tasks/config.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [spark : set_SPARK_HOME] **************************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : create_spark_conf_dir] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : create_symbolic_link_to_spark_conf_dir] **************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : include] *********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : copy_spark_conf_files] *******************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=spark-defaults.conf)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=spark-env.sh)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=metrics.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72] => (item=fairscheduler.xml)\u001b[0m\n",
      "\n",
      "TASK [spark : copy_sudoers_conf_of_SPARK_HOME] *********************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m15\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m5\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install_spark.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec1ca4-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec1dbc-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec1b8c-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### Spark HistoryServerのインストール"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec1dbc-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec1ed4-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec1ca4-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_spark_history] ****************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/install.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : check_jdk7_installed] ********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\u001b[1;35m [WARNING]: Consider using yum, dnf or zypper module rather than running rpm\n",
      "\u001b[0m\n",
      "\n",
      "TASK [java7 : download_oraclejdk7_by_wget] *************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : md5sum_rpm] ******************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : check_md5sum] ****************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : install_oraclejdk] ***********************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/java7/tasks/config.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_bash_profile] ***********************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [java7 : copy_sudoers_conf_of_JAVA_HOME] **********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark_user : include] ****************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/spark_user/tasks/user.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [spark_user : create_spark_user] ******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : include] *********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/spark/tasks/install.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [spark : install_spark_packages] ******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=[u'spark', u'spark-python'])\u001b[0m\n",
      "\n",
      "TASK [spark : include] *********************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/spark/tasks/config.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [spark : set_SPARK_HOME] **************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : create_spark_conf_dir] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : create_symbolic_link_to_spark_conf_dir] **************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : include] *********************************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark : copy_spark_conf_files] *******************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=spark-defaults.conf)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=spark-env.sh)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=metrics.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=log4j.properties)\u001b[0m\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72] => (item=fairscheduler.xml)\u001b[0m\n",
      "\n",
      "TASK [spark : copy_sudoers_conf_of_SPARK_HOME] *********************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark_history : include] *************************************************\n",
      "\u001b[0;36mincluded: /tmp/tmpaDuGp8/hadoop/playbooks/roles/spark_history/tasks/config.yml for XXX.XXX.XXX.72\u001b[0m\n",
      "\n",
      "TASK [spark_history : create_spark_log_dir] ************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark_history : create_symbolic_link_to_spark_log_dir] *******************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark_history : remove_spark_log_dir] ************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark_history : create_symbolic_link_to_spark_log_dir] *******************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [spark_history : create_spark_pid_dir] ************************************\n",
      "\u001b[0;36mskipping: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m19\u001b[0m   \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m2\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/install_spark_historyserver.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec1ed4-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec1fec-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec1dbc-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "### Spark HistoryServerの起動"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec1fec-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec2104-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec1ed4-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [hadoop_spark_history] ****************************************************\n",
      "\n",
      "TASK [setup] *******************************************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [check_status_spark_history_server] ***************************************\n",
      "\u001b[0;32mok: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "TASK [start_spark_history_server] **********************************************\n",
      "\u001b[0;33mchanged: [XXX.XXX.XXX.72]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33mXXX.XXX.XXX.72\u001b[0m               : \u001b[0;32mok\u001b[0m\u001b[0;32m=\u001b[0m\u001b[0;32m3\u001b[0m    \u001b[0;33mchanged\u001b[0m\u001b[0;33m=\u001b[0m\u001b[0;33m1\u001b[0m    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook -l {target_group} {playbook_dir}/start_spark_historyserver.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec2104-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec2230-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec1fec-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "これでSparkの準備は完了。以下のURLからSparkの実行履歴が確認できる。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec2230-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec2352-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec2104-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://XXX.XXX.XXX.72:18080/\n"
     ]
    }
   ],
   "source": [
    "sparkhistory_stdout = !ansible hadoop_spark_history -m ping -l {target_group}\n",
    "sparkhistory_nodes = map(lambda l: l.split()[0], filter(lambda l: 'SUCCESS' in l, sparkhistory_stdout))\n",
    "print('http://%s:18080/' % sparkhistory_nodes[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec2352-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec246a-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec2230-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "# インストール後確認\n",
    "\n",
    "ディスクの使用状況は？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec246a-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec2578-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec2352-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\n",
      "Filesystem      Size  Used Avail Use% Mounted on\n",
      "/dev/vda1       106G  2.8G   98G   3% /\n",
      "tmpfs           5.2G     0  5.2G   0% /dev/shm\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.71 | SUCCESS | rc=0 >>\n",
      "Filesystem      Size  Used Avail Use% Mounted on\n",
      "/dev/vda1       106G  2.4G   98G   3% /\n",
      "tmpfs           5.2G     0  5.2G   0% /dev/shm\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.70 | SUCCESS | rc=0 >>\n",
      "Filesystem      Size  Used Avail Use% Mounted on\n",
      "/dev/vda1       106G  2.4G   98G   3% /\n",
      "tmpfs           5.2G     0  5.2G   0% /dev/shm\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.113 | SUCCESS | rc=0 >>\n",
      "Filesystem      Size  Used Avail Use% Mounted on\n",
      "/dev/vda1       106G  2.4G   98G   3% /\n",
      "tmpfs           5.2G     0  5.2G   0% /dev/shm\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.112 | SUCCESS | rc=0 >>\n",
      "Filesystem      Size  Used Avail Use% Mounted on\n",
      "/dev/vda1       106G  2.5G   98G   3% /\n",
      "tmpfs           5.2G     0  5.2G   0% /dev/shm\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.73 | SUCCESS | rc=0 >>\n",
      "Filesystem      Size  Used Avail Use% Mounted on\n",
      "/dev/vda1       106G  2.5G   98G   3% /\n",
      "tmpfs           5.2G     0  5.2G   0% /dev/shm\n",
      "\u001b[0m\n",
      "\u001b[0;32mXXX.XXX.XXX.114 | SUCCESS | rc=0 >>\n",
      "Filesystem      Size  Used Avail Use% Mounted on\n",
      "/dev/vda1       106G  2.5G   98G   3% /\n",
      "tmpfs           5.2G     0  5.2G   0% /dev/shm\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ansible -a \"df -H\" {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec2578-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec2690-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec246a-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "# 動作確認"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec2690-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec27a8-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec2578-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## HDFSの動作確認\n",
    "\n",
    "`hadoop_client` とした仮想マシンで、 `hdfs dfs -ls /` のように実行してみる。\n",
    "\n",
    "(以下、Ansible経由でコマンド実行をしているが、SSH経由で実行してもOK)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec27a8-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec28ca-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec2690-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\r\n",
      "Found 6 items\r\n",
      "drwxr-xr-x   - hdfs   supergroup          0 2016-08-26 12:16 /apps\r\n",
      "drwxr-xr-x   - hbase  supergroup          0 2016-08-26 12:20 /hbase\r\n",
      "drwxr-xr-x   - mapred hadoop              0 2016-08-26 12:10 /mapred\r\n",
      "drwxrwxrwt   - hdfs   hadoop              0 2016-08-26 12:10 /tmp\r\n",
      "drwxrwxrwt   - hdfs   hadoop              0 2016-08-26 12:05 /user\r\n",
      "drwxrwxrwt   - hdfs   hadoop              0 2016-08-26 12:09 /var\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible -a \"hdfs dfs -ls /\" -l {target_group} hadoop_client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec28ca-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec29e2-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec27a8-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "NameNodeのWeb UIの *Browsing HDFS* でも確認可能"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec29e2-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec2afa-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec28ca-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://XXX.XXX.XXX.70:50070/explorer.html\n"
     ]
    }
   ],
   "source": [
    "haadmin_stdout = !ansible hadoop_namenode -l {target_group} -s -U hdfs -m shell -a 'timeout 15 hdfs haadmin -getServiceState $(hostname)'\n",
    "haadmin_result = map(lambda line: line.split()[0], filter(lambda line: len(line) > 0, haadmin_stdout))\n",
    "print(\"http://%s:50070/explorer.html\" % haadmin_result[haadmin_result.index(\"active\") - 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec2afa-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec2c12-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec29e2-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## MapReduceの動作確認\n",
    "\n",
    "サンプルジョブを動かしてみる。\n",
    "`hadoop_client` で、 `yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi 10 1000` のように実行してみる。ただしこの際、YARNのパーミッション定義の関係で、sudoでyarnユーザとして実行しないとエラーとなる。\n",
    "\n",
    "> 詳しくは [T12b_Hadoop - Simple YARN job for Test](T12b_Hadoop - Simple YARN job for Test.ipynb) を参考にしてみてください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec2c12-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec2d2a-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec2afa-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\r\n",
      "Number of Maps  = 10\r\n",
      "Samples per Map = 1000\r\n",
      "Wrote input for Map #0\r\n",
      "Wrote input for Map #1\r\n",
      "Wrote input for Map #2\r\n",
      "Wrote input for Map #3\r\n",
      "Wrote input for Map #4\r\n",
      "Wrote input for Map #5\r\n",
      "Wrote input for Map #6\r\n",
      "Wrote input for Map #7\r\n",
      "Wrote input for Map #8\r\n",
      "Wrote input for Map #9\r\n",
      "Starting Job\r\n",
      "Job Finished in 38.543 seconds\r\n",
      "Estimated value of Pi is 3.1408000000000000000016/08/26 12:27:14 INFO impl.TimelineClientImpl: Timeline service address: http://testvm003:8188/ws/v1/timeline/\r\n",
      "16/08/26 12:27:15 INFO input.FileInputFormat: Total input paths to process : 10\r\n",
      "16/08/26 12:27:15 INFO mapreduce.JobSubmitter: number of splits:10\r\n",
      "16/08/26 12:27:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1472181274763_0001\r\n",
      "16/08/26 12:27:16 INFO client.YARNRunner: Number of stages: 2\r\n",
      "16/08/26 12:27:17 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=XXX.XXX.XXX.2.4.2.0-258, revision=fa554fdce4e3495e09a310e0a32bb34ccc5946ad, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=20160425-0638 ]\r\n",
      "16/08/26 12:27:17 INFO impl.TimelineClientImpl: Timeline service address: http://testvm003:8188/ws/v1/timeline/\r\n",
      "16/08/26 12:27:17 INFO client.TezClient: Submitting DAG application with id: application_1472181274763_0001\r\n",
      "16/08/26 12:27:17 INFO client.TezClientUtils: Using tez.lib.uris value from configuration: hdfs://hdfs-cluster//apps/tez/tez.tar.gz\r\n",
      "16/08/26 12:27:17 INFO client.TezClient: Tez system stage directory hdfs://hdfs-cluster/tmp/hadoop-yarn/staging/yarn/.staging/job_1472181274763_0001/.tez/application_1472181274763_0001 doesn't exist and is created\r\n",
      "16/08/26 12:27:18 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1472181274763_0001, dagName=QuasiMonteCarlo\r\n",
      "16/08/26 12:27:19 INFO impl.YarnClientImpl: Submitted application application_1472181274763_0001\r\n",
      "16/08/26 12:27:19 INFO client.TezClient: The url to track the Tez AM: http://testvm001:8088/proxy/application_1472181274763_0001/\r\n",
      "16/08/26 12:27:19 INFO impl.TimelineClientImpl: Timeline service address: http://testvm003:8188/ws/v1/timeline/\r\n",
      "16/08/26 12:27:20 INFO impl.TimelineClientImpl: Timeline service address: http://testvm003:8188/ws/v1/timeline/\r\n",
      "16/08/26 12:27:20 INFO mapreduce.Job: The url to track the job: http://testvm001:8088/proxy/application_1472181274763_0001/\r\n",
      "16/08/26 12:27:20 INFO mapreduce.Job: Running job: job_1472181274763_0001\r\n",
      "16/08/26 12:27:33 INFO mapreduce.Job: Job job_1472181274763_0001 running in uber mode : false\r\n",
      "16/08/26 12:27:33 INFO mapreduce.Job:  map 0% reduce 0%\r\n",
      "16/08/26 12:27:48 INFO mapreduce.Job:  map 20% reduce 0%\r\n",
      "16/08/26 12:27:49 INFO mapreduce.Job:  map 40% reduce 0%\r\n",
      "16/08/26 12:27:50 INFO mapreduce.Job:  map 70% reduce 0%\r\n",
      "16/08/26 12:27:51 INFO mapreduce.Job:  map 100% reduce 0%\r\n",
      "16/08/26 12:27:51 INFO mapreduce.Job:  map 100% reduce 100%\r\n",
      "16/08/26 12:27:51 INFO mapreduce.Job: Job job_1472181274763_0001 completed successfully\r\n",
      "16/08/26 12:27:51 INFO mapreduce.Job: Counters: 0\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible -a \"yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar pi 10 1000\" --sudo --sudo-user yarn -l {target_group} hadoop_client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec2d2a-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec2e4c-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec2c12-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "MapReduceの実行に関する情報は、Resource ManagerのWeb UIから確認可能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec2e4c-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec2f6e-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec2d2a-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://XXX.XXX.XXX.70:8088/cluster/apps\n"
     ]
    }
   ],
   "source": [
    "rmadmin_stdout = !ansible hadoop_resourcemanager -l {target_group} -s -U yarn -m shell -a 'timeout 15 yarn rmadmin -getServiceState $(hostname)'\n",
    "rmadmin_result = map(lambda line: line.split()[0], filter(lambda line: len(line) > 0, rmadmin_stdout))\n",
    "active_resourcemanager_host = rmadmin_result[rmadmin_result.index(\"active\") - 1]\n",
    "print(\"http://%s:8088/cluster/apps\" % active_resourcemanager_host)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec2f6e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec3086-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec2e4c-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## HBaseの動作確認\n",
    "\n",
    "HBase Shellコマンドは HBase Master にインストールしてある。ここではテーブル一覧の確認だけしてみる。\n",
    "\n",
    "> 詳しくは [T12c_Hadoop - Simple HBase query for Test](T12c_Hadoop - Simple HBase query for Test.ipynb) を参考にしてみてください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec3086-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec319e-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec2f6e-495a-11e8-ba7d-0242ac130002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.71 | SUCCESS | rc=0 >>\r\n",
      "HBase Shell; enter 'help<RETURN>' for list of supported commands.\r\n",
      "Type \"exit<RETURN>\" to leave the HBase Shell\r\n",
      "Version XXX.XXX.XXX.2.4.2.0-258, rUnknown, Mon Apr 25 06:36:21 UTC 2016\r\n",
      "\r\n",
      "list;\r\n",
      "TABLE\r\n",
      "0 row(s) in 0.2920 seconds\r\n",
      "\r\n",
      "[]SLF4J: Class path contains multiple SLF4J bindings.\r\n",
      "SLF4J: Found binding in [jar:file:/usr/hdp/XXX.XXX.XXX.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\n",
      "SLF4J: Found binding in [jar:file:/usr/hdp/XXX.XXX.XXX.0-258/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\n",
      "SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\r\n",
      "SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible -m shell -a \"echo 'list;' | hbase shell\" -l {target_group} {hbase_master_host}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec319e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec32b6-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec3086-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## Hiveの動作確認\n",
    "\n",
    "Hiveは hadoop_client (IPは出力参照) にインストールしてある。\n",
    "ここではヘルプの確認のみおこなっています。\n",
    "\n",
    "> 詳しくは [T13b_Hadoop - Simple Hivemall query for Test](T13b_Hadoop - Simple Hivemall query for Test.ipynb) を参考にしてみてください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec32b6-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec33ce-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec319e-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\r\n",
      "Usage ./hive <parameters> --service serviceName <service parameters>\r\n",
      "Service List: beeline cleardanglingscratchdir cli help hiveburninclient hiveserver2 hiveserver hwi jar lineage metastore metatool orcfiledump rcfilecat schemaTool version \r\n",
      "Parameters parsed:\r\n",
      "  --auxpath : Auxillary jars \r\n",
      "  --config : Hive configuration directory\r\n",
      "  --service : Starts specific service/component. cli is default\r\n",
      "Parameters used:\r\n",
      "  HADOOP_HOME or HADOOP_PREFIX : Hadoop install directory\r\n",
      "  HIVE_OPT : Hive options\r\n",
      "For help on a particular service:\r\n",
      "  ./hive --service serviceName --help\r\n",
      "Debug help:  ./hive --debug --help\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible -a \"hive --help\" -l {target_group} --sudo --sudo-user yarn hadoop_client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec33ce-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec34e6-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec32b6-495a-11e8-ba7d-0242ac130002"
    }
   },
   "source": [
    "## Sparkの動作確認\n",
    "\n",
    "Sparkクライアントは hadoop_client (IPは出力参照) にインストールしてある。\n",
    "ここではヘルプの確認のみおこなっています。\n",
    "\n",
    "> 詳しくは [T12d_Hadoop - Simple Spark script for Test](T12d_Hadoop - Simple Spark script for Test.ipynb) を参考にしてみてください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "lc_cell_meme": {
     "current": "97ec34e6-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec35fe-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec33ce-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.72 | SUCCESS | rc=0 >>\r\n",
      "Usage: spark-submit [options] <app jar | python file> [app arguments]\r\n",
      "Usage: spark-submit --kill [submission ID] --master [spark://...]\r\n",
      "Usage: spark-submit --status [submission ID] --master [spark://...]\r\n",
      "\r\n",
      "Options:\r\n",
      "  --master MASTER_URL         spark://host:port, mesos://host:port, yarn, or local.\r\n",
      "  --deploy-mode DEPLOY_MODE   Whether to launch the driver program locally (\"client\") or\r\n",
      "                              on one of the worker machines inside the cluster (\"cluster\")\r\n",
      "                              (Default: client).\r\n",
      "  --class CLASS_NAME          Your application's main class (for Java / Scala apps).\r\n",
      "  --name NAME                 A name of your application.\r\n",
      "  --jars JARS                 Comma-separated list of local jars to include on the driver\r\n",
      "                              and executor classpaths.\r\n",
      "  --packages                  Comma-separated list of maven coordinates of jars to include\r\n",
      "                              on the driver and executor classpaths. Will search the local\r\n",
      "                              maven repo, then maven central and any additional remote\r\n",
      "                              repositories given by --repositories. The format for the\r\n",
      "                              coordinates should be groupId:artifactId:version.\r\n",
      "  --exclude-packages          Comma-separated list of groupId:artifactId, to exclude while\r\n",
      "                              resolving the dependencies provided in --packages to avoid\r\n",
      "                              dependency conflicts.\r\n",
      "  --repositories              Comma-separated list of additional remote repositories to\r\n",
      "                              search for the maven coordinates given with --packages.\r\n",
      "  --py-files PY_FILES         Comma-separated list of .zip, .egg, or .py files to place\r\n",
      "                              on the PYTHONPATH for Python apps.\r\n",
      "  --files FILES               Comma-separated list of files to be placed in the working\r\n",
      "                              directory of each executor.\r\n",
      "\r\n",
      "  --conf PROP=VALUE           Arbitrary Spark configuration property.\r\n",
      "  --properties-file FILE      Path to a file from which to load extra properties. If not\r\n",
      "                              specified, this will look for conf/spark-defaults.conf.\r\n",
      "\r\n",
      "  --driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 1024M).\r\n",
      "  --driver-java-options       Extra Java options to pass to the driver.\r\n",
      "  --driver-library-path       Extra library path entries to pass to the driver.\r\n",
      "  --driver-class-path         Extra class path entries to pass to the driver. Note that\r\n",
      "                              jars added with --jars are automatically included in the\r\n",
      "                              classpath.\r\n",
      "\r\n",
      "  --executor-memory MEM       Memory per executor (e.g. 1000M, 2G) (Default: 1G).\r\n",
      "\r\n",
      "  --proxy-user NAME           User to impersonate when submitting the application.\r\n",
      "                              This argument does not work with --principal / --keytab.\r\n",
      "\r\n",
      "  --help, -h                  Show this help message and exit\r\n",
      "  --verbose, -v               Print additional debug output\r\n",
      "  --version,                  Print the version of current Spark\r\n",
      "\r\n",
      " Spark standalone with cluster deploy mode only:\r\n",
      "  --driver-cores NUM          Cores for driver (Default: 1).\r\n",
      "\r\n",
      " Spark standalone or Mesos with cluster deploy mode only:\r\n",
      "  --supervise                 If given, restarts the driver on failure.\r\n",
      "  --kill SUBMISSION_ID        If given, kills the driver specified.\r\n",
      "  --status SUBMISSION_ID      If given, requests the status of the driver specified.\r\n",
      "\r\n",
      " Spark standalone and Mesos only:\r\n",
      "  --total-executor-cores NUM  Total cores for all executors.\r\n",
      "\r\n",
      " Spark standalone and YARN only:\r\n",
      "  --executor-cores NUM        Number of cores per executor. (Default: 1 in YARN mode,\r\n",
      "                              or all available cores on the worker in standalone mode)\r\n",
      "\r\n",
      " YARN-only:\r\n",
      "  --driver-cores NUM          Number of cores used by the driver, only in cluster mode\r\n",
      "                              (Default: 1).\r\n",
      "  --queue QUEUE_NAME          The YARN queue to submit to (Default: \"default\").\r\n",
      "  --num-executors NUM         Number of executors to launch (Default: 2).\r\n",
      "  --archives ARCHIVES         Comma separated list of archives to be extracted into the\r\n",
      "                              working directory of each executor.\r\n",
      "  --principal PRINCIPAL       Principal to be used to login to KDC, while running on\r\n",
      "                              secure HDFS.\r\n",
      "  --keytab KEYTAB             The full path to the file that contains the keytab for the\r\n",
      "                              principal specified above. This keytab will be copied to\r\n",
      "                              the node running the Application Master via the Secure\r\n",
      "                              Distributed Cache, for renewing the login tickets and the\r\n",
      "                              delegation tokens periodically.\r\n",
      "      \r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible -a 'spark-submit --help' hadoop_client -l {target_group}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "97ec35fe-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec3720-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec34e6-495a-11e8-ba7d-0242ac130002"
    },
    "scrolled": true
   },
   "source": [
    "# 後始末\n",
    "\n",
    "一時ディレクトリを削除する。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "97ec3720-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": "97ec382e-495a-11e8-ba7d-0242ac130002",
     "previous": "97ec35fe-495a-11e8-ba7d-0242ac130002"
    }
   },
   "outputs": [],
   "source": [
    "!rm -fr {work_dir}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "97ec382e-495a-11e8-ba7d-0242ac130002",
     "history": [],
     "next": null,
     "previous": "97ec3720-495a-11e8-ba7d-0242ac130002"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  },
  "lc_notebook_meme": {
   "current": "97ebb160-495a-11e8-ba7d-0242ac130002",
   "history": [],
   "root_cells": [
    "97ebb3e0-495a-11e8-ba7d-0242ac130002",
    "97ebb53e-495a-11e8-ba7d-0242ac130002",
    "97ebb67e-495a-11e8-ba7d-0242ac130002",
    "97ebb7a0-495a-11e8-ba7d-0242ac130002",
    "97ebb8cc-495a-11e8-ba7d-0242ac130002",
    "97ebb9ee-495a-11e8-ba7d-0242ac130002",
    "97ebbb10-495a-11e8-ba7d-0242ac130002",
    "97ebbc32-495a-11e8-ba7d-0242ac130002",
    "97ebbd54-495a-11e8-ba7d-0242ac130002",
    "97ebbe76-495a-11e8-ba7d-0242ac130002",
    "97ebbf8e-495a-11e8-ba7d-0242ac130002",
    "97ebc0a6-495a-11e8-ba7d-0242ac130002",
    "97ebc1be-495a-11e8-ba7d-0242ac130002",
    "97ebc2e0-495a-11e8-ba7d-0242ac130002",
    "97ebc3f8-495a-11e8-ba7d-0242ac130002",
    "97ebc51a-495a-11e8-ba7d-0242ac130002",
    "97ebc63c-495a-11e8-ba7d-0242ac130002",
    "97ebc754-495a-11e8-ba7d-0242ac130002",
    "97ebc876-495a-11e8-ba7d-0242ac130002",
    "97ebc98e-495a-11e8-ba7d-0242ac130002",
    "97ebcab0-495a-11e8-ba7d-0242ac130002",
    "97ebcbc8-495a-11e8-ba7d-0242ac130002",
    "97ebccea-495a-11e8-ba7d-0242ac130002",
    "97ebce02-495a-11e8-ba7d-0242ac130002",
    "97ebcf1a-495a-11e8-ba7d-0242ac130002",
    "97ebd032-495a-11e8-ba7d-0242ac130002",
    "97ebd154-495a-11e8-ba7d-0242ac130002",
    "97ebd26c-495a-11e8-ba7d-0242ac130002",
    "97ebd38e-495a-11e8-ba7d-0242ac130002",
    "97ebd4a6-495a-11e8-ba7d-0242ac130002",
    "97ebd5be-495a-11e8-ba7d-0242ac130002",
    "97ebd6d6-495a-11e8-ba7d-0242ac130002",
    "97ebd7f8-495a-11e8-ba7d-0242ac130002",
    "97ebd910-495a-11e8-ba7d-0242ac130002",
    "97ebda28-495a-11e8-ba7d-0242ac130002",
    "97ebdb4a-495a-11e8-ba7d-0242ac130002",
    "97ebdc6c-495a-11e8-ba7d-0242ac130002",
    "97ebdda2-495a-11e8-ba7d-0242ac130002",
    "97ebdece-495a-11e8-ba7d-0242ac130002",
    "97ebdfe6-495a-11e8-ba7d-0242ac130002",
    "97ebe0fe-495a-11e8-ba7d-0242ac130002",
    "97ebe216-495a-11e8-ba7d-0242ac130002",
    "97ebe338-495a-11e8-ba7d-0242ac130002",
    "97ebe45a-495a-11e8-ba7d-0242ac130002",
    "97ebe57c-495a-11e8-ba7d-0242ac130002",
    "97ebe694-495a-11e8-ba7d-0242ac130002",
    "97ebe7b6-495a-11e8-ba7d-0242ac130002",
    "97ebe8ce-495a-11e8-ba7d-0242ac130002",
    "97ebe9f0-495a-11e8-ba7d-0242ac130002",
    "97ebeb08-495a-11e8-ba7d-0242ac130002",
    "97ebec2a-495a-11e8-ba7d-0242ac130002",
    "97ebed42-495a-11e8-ba7d-0242ac130002",
    "97ebee5a-495a-11e8-ba7d-0242ac130002",
    "97ebef7c-495a-11e8-ba7d-0242ac130002",
    "97ebf08a-495a-11e8-ba7d-0242ac130002",
    "97ebf1ac-495a-11e8-ba7d-0242ac130002",
    "97ebf2c4-495a-11e8-ba7d-0242ac130002",
    "97ebf3dc-495a-11e8-ba7d-0242ac130002",
    "97ebf4f4-495a-11e8-ba7d-0242ac130002",
    "97ebf60c-495a-11e8-ba7d-0242ac130002",
    "97ebf724-495a-11e8-ba7d-0242ac130002",
    "97ebf83c-495a-11e8-ba7d-0242ac130002",
    "97ebf95e-495a-11e8-ba7d-0242ac130002",
    "97ebfa6c-495a-11e8-ba7d-0242ac130002",
    "97ebfb8e-495a-11e8-ba7d-0242ac130002",
    "97ebfcb0-495a-11e8-ba7d-0242ac130002",
    "97ebfdd2-495a-11e8-ba7d-0242ac130002",
    "97ebfeea-495a-11e8-ba7d-0242ac130002",
    "97ec0002-495a-11e8-ba7d-0242ac130002",
    "97ec011a-495a-11e8-ba7d-0242ac130002",
    "97ec023c-495a-11e8-ba7d-0242ac130002",
    "97ec0354-495a-11e8-ba7d-0242ac130002",
    "97ec0476-495a-11e8-ba7d-0242ac130002",
    "97ec058e-495a-11e8-ba7d-0242ac130002",
    "97ec06a6-495a-11e8-ba7d-0242ac130002",
    "97ec07be-495a-11e8-ba7d-0242ac130002",
    "97ec08e0-495a-11e8-ba7d-0242ac130002",
    "97ec09f8-495a-11e8-ba7d-0242ac130002",
    "97ec0b10-495a-11e8-ba7d-0242ac130002",
    "97ec0c1e-495a-11e8-ba7d-0242ac130002",
    "97ec0d40-495a-11e8-ba7d-0242ac130002",
    "97ec0e62-495a-11e8-ba7d-0242ac130002",
    "97ec0f7a-495a-11e8-ba7d-0242ac130002",
    "97ec1092-495a-11e8-ba7d-0242ac130002",
    "97ec11aa-495a-11e8-ba7d-0242ac130002",
    "97ec12c2-495a-11e8-ba7d-0242ac130002",
    "97ec13e4-495a-11e8-ba7d-0242ac130002",
    "97ec14fc-495a-11e8-ba7d-0242ac130002",
    "97ec1614-495a-11e8-ba7d-0242ac130002",
    "97ec172c-495a-11e8-ba7d-0242ac130002",
    "97ec1844-495a-11e8-ba7d-0242ac130002",
    "97ec195c-495a-11e8-ba7d-0242ac130002",
    "97ec1a74-495a-11e8-ba7d-0242ac130002",
    "97ec1b8c-495a-11e8-ba7d-0242ac130002",
    "97ec1ca4-495a-11e8-ba7d-0242ac130002",
    "97ec1dbc-495a-11e8-ba7d-0242ac130002",
    "97ec1ed4-495a-11e8-ba7d-0242ac130002",
    "97ec1fec-495a-11e8-ba7d-0242ac130002",
    "97ec2104-495a-11e8-ba7d-0242ac130002",
    "97ec2230-495a-11e8-ba7d-0242ac130002",
    "97ec2352-495a-11e8-ba7d-0242ac130002",
    "97ec246a-495a-11e8-ba7d-0242ac130002",
    "97ec2578-495a-11e8-ba7d-0242ac130002",
    "97ec2690-495a-11e8-ba7d-0242ac130002",
    "97ec27a8-495a-11e8-ba7d-0242ac130002",
    "97ec28ca-495a-11e8-ba7d-0242ac130002",
    "97ec29e2-495a-11e8-ba7d-0242ac130002",
    "97ec2afa-495a-11e8-ba7d-0242ac130002",
    "97ec2c12-495a-11e8-ba7d-0242ac130002",
    "97ec2d2a-495a-11e8-ba7d-0242ac130002",
    "97ec2e4c-495a-11e8-ba7d-0242ac130002",
    "97ec2f6e-495a-11e8-ba7d-0242ac130002",
    "97ec3086-495a-11e8-ba7d-0242ac130002",
    "97ec319e-495a-11e8-ba7d-0242ac130002",
    "97ec32b6-495a-11e8-ba7d-0242ac130002",
    "97ec33ce-495a-11e8-ba7d-0242ac130002",
    "97ec34e6-495a-11e8-ba7d-0242ac130002",
    "97ec35fe-495a-11e8-ba7d-0242ac130002",
    "97ec3720-495a-11e8-ba7d-0242ac130002",
    "97ec382e-495a-11e8-ba7d-0242ac130002"
   ]
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  },
  "toc_position": {
   "height": "282px",
   "left": "1138px",
   "right": "20px",
   "top": "120px",
   "width": "373px"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}