04_Store_Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915429dc-152a-11e7-b31a-0242ac12000d",
     "next": "91542c48-152a-11e7-b31a-0242ac12000d",
     "previous": null
    }
   },
   "source": [
    "# サーチエンジンとしての利用手順(データ蓄積)\n",
    "----\n",
    "LogstashからElasticsearchへ、様々な形式の データを蓄積する方法について説明します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91542c48-152a-11e7-b31a-0242ac12000d",
     "next": "91542e64-152a-11e7-b31a-0242ac12000d",
     "previous": "915429dc-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## 準備"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91542e64-152a-11e7-b31a-0242ac12000d",
     "next": "91543062-152a-11e7-b31a-0242ac12000d",
     "previous": "91542c48-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "本章のコマンドを実行するための設定を行います。  \n",
    "\n",
    "セットアップ済みの環境のうちコマンドを送る先であるCoordinating(Client) Nodeのホストアドレスを、次のセルに記入して実行し、保存してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "lc_cell_meme": {
     "current": "91543062-152a-11e7-b31a-0242ac12000d",
     "next": "91543256-152a-11e7-b31a-0242ac12000d",
     "previous": "91542e64-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: ES_HOST=XXX.XXX.XXX.232:9200\n"
     ]
    }
   ],
   "source": [
    "%env ES_HOST=XXX.XXX.XXX.232:9200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91543256-152a-11e7-b31a-0242ac12000d",
     "next": "91543454-152a-11e7-b31a-0242ac12000d",
     "previous": "91543062-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "また、データを蓄積する先のindexとtypeを、次のセルを実行して保存してください。  \n",
    "\n",
    "※特にセルの内容の書き換えは必要ありません。  \n",
    "　前の章と異なるindex名にしているのは、この章のコマンドで投入する内容が前の章までのものと混ざらないようにするためです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "lc_cell_meme": {
     "current": "91543454-152a-11e7-b31a-0242ac12000d",
     "next": "91543666-152a-11e7-b31a-0242ac12000d",
     "previous": "91543256-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: INDEX=sample_index\n",
      "env: TYPE=logs\n"
     ]
    }
   ],
   "source": [
    "%env INDEX=sample_index\n",
    "%env TYPE=logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91543666-152a-11e7-b31a-0242ac12000d",
     "next": "91543828-152a-11e7-b31a-0242ac12000d",
     "previous": "91543454-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## Logstashとは"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91543828-152a-11e7-b31a-0242ac12000d",
     "next": "915439fe-152a-11e7-b31a-0242ac12000d",
     "previous": "91543666-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Logstashは、Elastic Stackを構成するプロダクトの１つです。  \n",
    "データの取り込み、加工、出力（格納）を処理するパイプラインを簡単に構築できます。\n",
    "\n",
    "対応できる形式は次のように多岐に渡ります。\n",
    "\n",
    "- 入力：各種ファイル、RDB、通信ストリーム、ログ、S3、BeatsというElasticのデータ収集ツール…\n",
    "- 加工：各種形式の解釈、分割、集計、地理情報の付与、個人情報のマスク…\n",
    "- 出力：Elasticsearch、S3、各種ファイル、通信…\n",
    "\n",
    "対応できる形式などについては、Logstash([公式ページ](https://www.elastic.co/jp/products/logstash))を参照してください。  \n",
    "  \n",
    "本章では、入力に各種ファイル、格納先にElasticsearchを指定する、単純なケースでのデータ蓄積を実施します。  \n",
    "また、変換処理をLogstashで実施するだけでなく、ElasticsearchのIngest Nodeで分散実施するケースについても説明します。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915439fe-152a-11e7-b31a-0242ac12000d",
     "next": "91543bd4-152a-11e7-b31a-0242ac12000d",
     "previous": "91543828-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## Logstashの設定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91543bd4-152a-11e7-b31a-0242ac12000d",
     "next": "91543da0-152a-11e7-b31a-0242ac12000d",
     "previous": "915439fe-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Logstashを利用する際には、\".conf\"という拡張子のファイルに振る舞いを定義し、それを読み込んで実行させます。  \n",
    "sample.confというconfファイルを用いる場合の実行コマンドは次のようになります。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "lc_cell_meme": {
     "current": "91543da0-152a-11e7-b31a-0242ac12000d",
     "next": "91543f6c-152a-11e7-b31a-0242ac12000d",
     "previous": "91543bd4-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "logstash -f sample.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91543f6c-152a-11e7-b31a-0242ac12000d",
     "next": "9154412e-152a-11e7-b31a-0242ac12000d",
     "previous": "91543da0-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "confファイルはinput・filter・outputの３つの設定によって構成されます。  \n",
    "- **input** : 入力元や形式の設定\n",
    "- **filter** : 入力された内容の加工内容の設定\n",
    "- **output** : 出力先や形式の設定\n",
    "\n",
    "confファイルの例を次に示します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154412e-152a-11e7-b31a-0242ac12000d",
     "next": "9154443a-152a-11e7-b31a-0242ac12000d",
     "previous": "91543f6c-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "```ruby:example.conf\n",
    "input{\n",
    "  stdin{\n",
    "    codec => plain{\n",
    "      charset => \"UTF-8\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "filter{\n",
    "  grok { \n",
    "    match => [ \"message\", \"%{COMBINEDAPACHELOG}\" ] \n",
    "  }\n",
    "  geoip{\n",
    "    source => \"ip\"\n",
    "  }\n",
    "}\n",
    "output{\n",
    "  stdout{\n",
    "    codec => dots\n",
    "  }\n",
    "  elasticsearch{\n",
    "    hosts => localhost\n",
    "    user => \"elastic\"\n",
    "    password => \"changeme\"\n",
    "    index => \"apache_sample\"\n",
    "  }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154443a-152a-11e7-b31a-0242ac12000d",
     "next": "91544642-152a-11e7-b31a-0242ac12000d",
     "previous": "9154412e-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "設定ファイルの各要素の内容や、上の例で対応する部分の意味は次の通りです。\n",
    "\n",
    "||意味|設定例の内容|設定内容の参考ページ|\n",
    "|----|----|----|----|\n",
    "|**input**|データの入力元や形式、およびその詳細を定義します。|stdin{}は標準入力からデータを受けとる設定です。<br>また、文字コードをUTF-8に指定しています。<br>stdin{}の代わりにtwitter{},file{}のように指定すれば入力元を標準入力以外にすることもできます。|[Input plugins](https://www.elastic.co/guide/en/logstash/current/input-plugins.html)|\n",
    "|**filter**|inputから受け取った入力の加工内容を定義します。<br>複数のfilterを記述した場合、記述順に加工されます。|入力の特定部分をパースするgrok filterと<br>地理情報を付加するgeoip filterを指定しています。|[Filter plugins](https://www.elastic.co/guide/en/logstash/current/filter-plugins.html)|\n",
    "|**output**|データの出力先や形式、およびその詳細を定義します。|stdout{}とelasticsearch{}の2つを記述することで、標準出力とElasticsearchの両方に出力するよう指定しています。<br>Elasticsearchへの出力設定では、宛先ホストやアカウント、index名を指定しています。|[Output plugins](https://www.elastic.co/guide/en/logstash/current/output-plugins.html)|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91544642-152a-11e7-b31a-0242ac12000d",
     "next": "91544818-152a-11e7-b31a-0242ac12000d",
     "previous": "9154443a-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## データフォーマット"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91544818-152a-11e7-b31a-0242ac12000d",
     "next": "915449f8-152a-11e7-b31a-0242ac12000d",
     "previous": "91544642-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "ここでは、様々なデータフォーマットのデータをElasticsearchに登録する際の具体的な設定と手法について説明します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915449f8-152a-11e7-b31a-0242ac12000d",
     "next": "91544bce-152a-11e7-b31a-0242ac12000d",
     "previous": "91544818-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Logstashのサーバーに関する設定を行います。  \n",
    "次のセルを実行してユーザー名と、秘密鍵のパスを設定してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "91544bce-152a-11e7-b31a-0242ac12000d",
     "next": "91544da4-152a-11e7-b31a-0242ac12000d",
     "previous": "915449f8-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [],
   "source": [
    "#Logstashのサーバーでコマンドを実行するユーザー\n",
    "USER='ansible'\n",
    "\n",
    "#公開鍵認証を行う場合の秘密鍵のパス\n",
    "KEYPATH='~/.ssh/ansible_id_rsa'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91544da4-152a-11e7-b31a-0242ac12000d",
     "next": "91544f70-152a-11e7-b31a-0242ac12000d",
     "previous": "91544bce-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### JSON"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91544f70-152a-11e7-b31a-0242ac12000d",
     "next": "91545146-152a-11e7-b31a-0242ac12000d",
     "previous": "91544da4-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "JSON形式のデータとして、次の内容のサンプルを登録します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "lc_cell_meme": {
     "current": "91545146-152a-11e7-b31a-0242ac12000d",
     "next": "91545344-152a-11e7-b31a-0242ac12000d",
     "previous": "91544f70-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"timestamp\":\"1\",\"atmospheric_pressure\":\"1000.4\",\"sea_level_pressure\":\"1003.4\",\"precipitation_day\":\"0\",\"precipitation_max_hour\":\"0\",\"precipitation_max_10min\":\"0\",\"temperature_avg\":\"3.7\",\"temperature_max\":\"8.2\",\"temperature_min\":\"0.7\",\"humidity_avg\":\"41\",\"humidity_min\":\"30\",\"wind_speed_avg\":\"4.4\",\"wind_speed_max\":\"9.4\",\"wind_direction\":\"西\",\"wind_speed_max_moment\":\"18.7\",\"wind_direction_max_moment\":\"北西\",\"sunshine_duration\":\"1.3\",\"snowfall\":\"\",\"snowfall_max\":\"\",\"information_daytime\":\"時々晴一時雪\",\"information_night\":\"晴\"}\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 1 sample_data/weather.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91545344-152a-11e7-b31a-0242ac12000d",
     "next": "91545510-152a-11e7-b31a-0242ac12000d",
     "previous": "91545146-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "confファイルには、codec => \"json\" と指定することでJSON形式を解釈できるように指定します。  \n",
    "ファイル名は\"json.conf\"とします。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91545510-152a-11e7-b31a-0242ac12000d",
     "next": "915456e6-152a-11e7-b31a-0242ac12000d",
     "previous": "91545344-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "```ruby:json.conf\n",
    "input{  \n",
    "  stdin{ \n",
    "    codec => \"json\"  \n",
    "  }  \n",
    "}  \n",
    "filter{  \n",
    "}  \n",
    "output{  \n",
    "  elasticsearch{  \n",
    "    hosts => <ES_HOST>  \n",
    "    index => <INDEX>  \n",
    "  }  \n",
    "}  \n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915456e6-152a-11e7-b31a-0242ac12000d",
     "next": "915458b2-152a-11e7-b31a-0242ac12000d",
     "previous": "91545510-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "weather.jsonとjson.confをLogstashのサーバーに配置してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915458b2-152a-11e7-b31a-0242ac12000d",
     "next": "91545aa6-152a-11e7-b31a-0242ac12000d",
     "previous": "915456e6-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "実際のデータ登録は次のコマンドで実行します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "lc_cell_meme": {
     "current": "91545aa6-152a-11e7-b31a-0242ac12000d",
     "next": "91545c7c-152a-11e7-b31a-0242ac12000d",
     "previous": "915458b2-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\r\n",
      "Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties.\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible logstash-server -i ./hosts -m shell -a \"cat weather.json | sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/json.conf\" -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91545c7c-152a-11e7-b31a-0242ac12000d",
     "next": "91545e52-152a-11e7-b31a-0242ac12000d",
     "previous": "91545aa6-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "コマンドの実行に成功したら、次の検索APIで正しく登録されているか確認してください。  \n",
    "正しく登録されていれば、json形式で登録されたデータが出力されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "lc_cell_meme": {
     "current": "91545e52-152a-11e7-b31a-0242ac12000d",
     "next": "9154601e-152a-11e7-b31a-0242ac12000d",
     "previous": "91545c7c-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"took\" : 22,\n",
      "  \"timed_out\" : false,\n",
      "  \"_shards\" : {\n",
      "    \"total\" : 5,\n",
      "    \"successful\" : 5,\n",
      "    \"failed\" : 0\n",
      "  },\n",
      "  \"hits\" : {\n",
      "    \"total\" : 30,\n",
      "    \"max_score\" : 1.0,\n",
      "    \"hits\" : [\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_json\",\n",
      "        \"_id\" : \"AVjxYFq1eJQ1fjM9_cyd\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"precipitation_max_10min\" : \"0\",\n",
      "          \"wind_speed_avg\" : \"4.4\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"0\",\n",
      "          \"temperature_max\" : \"8.2\",\n",
      "          \"wind_direction\" : \"西\",\n",
      "          \"information_daytime\" : \"時々晴一時雪\",\n",
      "          \"precipitation_day\" : \"0\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:52:52.957Z\",\n",
      "          \"humidity_avg\" : \"41\",\n",
      "          \"humidity_min\" : \"30\",\n",
      "          \"wind_speed_max\" : \"9.4\",\n",
      "          \"sunshine_duration\" : \"1.3\",\n",
      "          \"temperature_avg\" : \"3.7\",\n",
      "          \"temperature_min\" : \"0.7\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"atmospheric_pressure\" : \"1000.4\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"wind_speed_max_moment\" : \"18.7\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1003.4\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"1\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_json\",\n",
      "        \"_id\" : \"AVjxYFq1eJQ1fjM9_cyi\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"precipitation_max_10min\" : \"1.5\",\n",
      "          \"wind_speed_avg\" : \"5.2\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"5\",\n",
      "          \"temperature_max\" : \"16\",\n",
      "          \"wind_direction\" : \"南西\",\n",
      "          \"information_daytime\" : \"後時々雨\",\n",
      "          \"precipitation_day\" : \"6.5\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:52:53.293Z\",\n",
      "          \"humidity_avg\" : \"64\",\n",
      "          \"humidity_min\" : \"40\",\n",
      "          \"wind_speed_max\" : \"8.9\",\n",
      "          \"sunshine_duration\" : \"0.2\",\n",
      "          \"temperature_avg\" : \"10.8\",\n",
      "          \"temperature_min\" : \"5.8\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"atmospheric_pressure\" : \"1000.6\",\n",
      "          \"wind_direction_max_moment\" : \"\",\n",
      "          \"wind_speed_max_moment\" : \"19.3\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1003.5\",\n",
      "          \"information_night\" : \"時々晴\",\n",
      "          \"timestamp\" : \"6\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_json\",\n",
      "        \"_id\" : \"AVjxYFq1eJQ1fjM9_cyj\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"precipitation_max_10min\" : \"\",\n",
      "          \"wind_speed_avg\" : \"4.2\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"\",\n",
      "          \"temperature_max\" : \"10.2\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"information_daytime\" : \"時々曇\",\n",
      "          \"precipitation_day\" : \"\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:52:53.298Z\",\n",
      "          \"humidity_avg\" : \"35\",\n",
      "          \"humidity_min\" : \"23\",\n",
      "          \"wind_speed_max\" : \"7\",\n",
      "          \"sunshine_duration\" : \"3.3\",\n",
      "          \"temperature_avg\" : \"5.6\",\n",
      "          \"temperature_min\" : \"3\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"atmospheric_pressure\" : \"1000.5\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"wind_speed_max_moment\" : \"16.1\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1003.4\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"7\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_json\",\n",
      "        \"_id\" : \"AVjxYFq1eJQ1fjM9_cyn\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"precipitation_max_10min\" : \"\",\n",
      "          \"wind_speed_avg\" : \"2.7\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"\",\n",
      "          \"temperature_max\" : \"10.8\",\n",
      "          \"wind_direction\" : \"南東\",\n",
      "          \"information_daytime\" : \"晴\",\n",
      "          \"precipitation_day\" : \"\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:52:53.306Z\",\n",
      "          \"humidity_avg\" : \"46\",\n",
      "          \"humidity_min\" : \"30\",\n",
      "          \"wind_speed_max\" : \"5.7\",\n",
      "          \"sunshine_duration\" : \"9.1\",\n",
      "          \"temperature_avg\" : \"6.4\",\n",
      "          \"temperature_min\" : \"0.2\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"atmospheric_pressure\" : \"1008.7\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"wind_speed_max_moment\" : \"10\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1011.7\",\n",
      "          \"information_night\" : \"一時曇\",\n",
      "          \"timestamp\" : \"11\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_json\",\n",
      "        \"_id\" : \"AVjxYFq1eJQ1fjM9_cyr\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"precipitation_max_10min\" : \"2.5\",\n",
      "          \"wind_speed_avg\" : \"3.2\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"11\",\n",
      "          \"temperature_max\" : \"8.2\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"information_daytime\" : \"雨一時曇\",\n",
      "          \"precipitation_day\" : \"35.5\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:52:53.310Z\",\n",
      "          \"humidity_avg\" : \"76\",\n",
      "          \"humidity_min\" : \"46\",\n",
      "          \"wind_speed_max\" : \"8.1\",\n",
      "          \"sunshine_duration\" : \"0\",\n",
      "          \"temperature_avg\" : \"5.4\",\n",
      "          \"temperature_min\" : \"3.4\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"atmospheric_pressure\" : \"1009.7\",\n",
      "          \"wind_direction_max_moment\" : \"西\",\n",
      "          \"wind_speed_max_moment\" : \"14.1\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1012.7\",\n",
      "          \"information_night\" : \"後晴\",\n",
      "          \"timestamp\" : \"15\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_json\",\n",
      "        \"_id\" : \"AVjxYFq1eJQ1fjM9_cy2\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"precipitation_max_10min\" : \"0.5\",\n",
      "          \"wind_speed_avg\" : \"2\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"0.5\",\n",
      "          \"temperature_max\" : \"12.9\",\n",
      "          \"wind_direction\" : \"南東\",\n",
      "          \"information_daytime\" : \"時々晴\",\n",
      "          \"precipitation_day\" : \"0.5\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:52:53.325Z\",\n",
      "          \"humidity_avg\" : \"74\",\n",
      "          \"humidity_min\" : \"52\",\n",
      "          \"wind_speed_max\" : \"3.8\",\n",
      "          \"sunshine_duration\" : \"4.4\",\n",
      "          \"temperature_avg\" : \"9\",\n",
      "          \"temperature_min\" : \"5.6\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"atmospheric_pressure\" : \"1019.1\",\n",
      "          \"wind_direction_max_moment\" : \"南東\",\n",
      "          \"wind_speed_max_moment\" : \"6\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1022.1\",\n",
      "          \"information_night\" : \"一時曇\",\n",
      "          \"timestamp\" : \"26\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_json\",\n",
      "        \"_id\" : \"AVjxYFq1eJQ1fjM9_cy3\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"precipitation_max_10min\" : \"1.5\",\n",
      "          \"wind_speed_avg\" : \"3.1\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"6\",\n",
      "          \"temperature_max\" : \"16.4\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"information_daytime\" : \"時々曇一時雨\",\n",
      "          \"precipitation_day\" : \"14.5\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:52:53.325Z\",\n",
      "          \"humidity_avg\" : \"71\",\n",
      "          \"humidity_min\" : \"40\",\n",
      "          \"wind_speed_max\" : \"6.3\",\n",
      "          \"sunshine_duration\" : \"5.1\",\n",
      "          \"temperature_avg\" : \"9.8\",\n",
      "          \"temperature_min\" : \"5.8\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"atmospheric_pressure\" : \"1007\",\n",
      "          \"wind_direction_max_moment\" : \"西\",\n",
      "          \"wind_speed_max_moment\" : \"12.1\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1009.9\",\n",
      "          \"information_night\" : \"時々雨後一時みぞれ\",\n",
      "          \"timestamp\" : \"27\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_json\",\n",
      "        \"_id\" : \"AVjxYFq1eJQ1fjM9_cy4\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"precipitation_max_10min\" : \"0\",\n",
      "          \"wind_speed_avg\" : \"4\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"0\",\n",
      "          \"temperature_max\" : \"9.7\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"information_daytime\" : \"時々曇一時雪\",\n",
      "          \"precipitation_day\" : \"0\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:52:53.326Z\",\n",
      "          \"humidity_avg\" : \"41\",\n",
      "          \"humidity_min\" : \"20\",\n",
      "          \"wind_speed_max\" : \"7.6\",\n",
      "          \"sunshine_duration\" : \"6.2\",\n",
      "          \"temperature_avg\" : \"5.2\",\n",
      "          \"temperature_min\" : \"1.4\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"atmospheric_pressure\" : \"1014.7\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"wind_speed_max_moment\" : \"14.6\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1017.7\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"28\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_json\",\n",
      "        \"_id\" : \"AVjxYFq1eJQ1fjM9_cyf\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"precipitation_max_10min\" : \"\",\n",
      "          \"wind_speed_avg\" : \"3\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"\",\n",
      "          \"temperature_max\" : \"8.9\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"information_daytime\" : \"晴\",\n",
      "          \"precipitation_day\" : \"\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:52:53.291Z\",\n",
      "          \"humidity_avg\" : \"42\",\n",
      "          \"humidity_min\" : \"19\",\n",
      "          \"wind_speed_max\" : \"7.4\",\n",
      "          \"sunshine_duration\" : \"8.9\",\n",
      "          \"temperature_avg\" : \"3.8\",\n",
      "          \"temperature_min\" : \"-1.1\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"atmospheric_pressure\" : \"1012\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"wind_speed_max_moment\" : \"10.9\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1015.1\",\n",
      "          \"information_night\" : \"後薄曇\",\n",
      "          \"timestamp\" : \"3\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_json\",\n",
      "        \"_id\" : \"AVjxYFq1eJQ1fjM9_cyg\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"precipitation_max_10min\" : \"\",\n",
      "          \"wind_speed_avg\" : \"1.3\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"\",\n",
      "          \"temperature_max\" : \"9.3\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"information_daytime\" : \"後晴\",\n",
      "          \"precipitation_day\" : \"\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:52:53.292Z\",\n",
      "          \"humidity_avg\" : \"51\",\n",
      "          \"humidity_min\" : \"33\",\n",
      "          \"wind_speed_max\" : \"2.8\",\n",
      "          \"sunshine_duration\" : \"6\",\n",
      "          \"temperature_avg\" : \"4\",\n",
      "          \"temperature_min\" : \"-0.4\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"atmospheric_pressure\" : \"1010.4\",\n",
      "          \"wind_direction_max_moment\" : \"北東\",\n",
      "          \"wind_speed_max_moment\" : \"3.8\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1013.4\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"4\"\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100 11393  100 11393    0     0   375k      0 --:--:-- --:--:-- --:--:--  383k\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/json_$INDEX/_search?pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154601e-152a-11e7-b31a-0242ac12000d",
     "next": "915461f4-152a-11e7-b31a-0242ac12000d",
     "previous": "91545e52-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### 複数行JSON"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915461f4-152a-11e7-b31a-0242ac12000d",
     "next": "915463c0-152a-11e7-b31a-0242ac12000d",
     "previous": "9154601e-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "1つのJSONデータが改行を含んだ形式の場合を試します。  \n",
    "次の内容のサンプルを登録します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "lc_cell_meme": {
     "current": "915463c0-152a-11e7-b31a-0242ac12000d",
     "next": "915465b4-152a-11e7-b31a-0242ac12000d",
     "previous": "915461f4-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r",
      "\r\n",
      "\"timestamp\":\"1\",\r",
      "\r\n",
      "\"atmospheric_pressure\":\"1000.4\",\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 3 sample_data/weather_multiline.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915465b4-152a-11e7-b31a-0242ac12000d",
     "next": "91546780-152a-11e7-b31a-0242ac12000d",
     "previous": "915463c0-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "confファイルには、codec => multilineと指定することで、複数行にまたがるデータ形式を解釈できるように指定します。\n",
    "通常、Logstashでは1行ごとに1つのイベントとして処理しますが、multilineでは特定のパターンを見つけるまで行を読み込み、結合することで１つのイベントとして処理します。  \n",
    "\n",
    "ファイル名は\"json_multiline.conf\"とします。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91546780-152a-11e7-b31a-0242ac12000d",
     "next": "9154694c-152a-11e7-b31a-0242ac12000d",
     "previous": "915465b4-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "```rubby:weather_multiline.conf\n",
    "input{\n",
    "  stdin{\n",
    "    codec => multiline{\n",
    "      pattern => \"}\"\n",
    "      negate => \"true\"\n",
    "      what => \"next\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "filter{\n",
    "  json{\n",
    "    source => \"message\"\n",
    "  }\n",
    "}\n",
    "output{\n",
    "  elasticsearch{\n",
    "    hosts => <ES_HOST>\n",
    "    index => <INDEX>\n",
    "  }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154694c-152a-11e7-b31a-0242ac12000d",
     "next": "91546b22-152a-11e7-b31a-0242ac12000d",
     "previous": "91546780-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "実際のデータ登録は次のコマンドで実行します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "lc_cell_meme": {
     "current": "91546b22-152a-11e7-b31a-0242ac12000d",
     "next": "91546d0c-152a-11e7-b31a-0242ac12000d",
     "previous": "9154694c-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\r\n",
      "Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties.\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible logstash-server -i ./hosts -m shell -a \"cat weather_multiline.json | sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/json_multiline.conf\" -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91546d0c-152a-11e7-b31a-0242ac12000d",
     "next": "91546eec-152a-11e7-b31a-0242ac12000d",
     "previous": "91546b22-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "コマンドの実行に成功したら、次の検索APIで正しく登録されているか確認してください。  \n",
    "正しく登録されていれば、json形式で登録されたデータが出力されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "lc_cell_meme": {
     "current": "91546eec-152a-11e7-b31a-0242ac12000d",
     "next": "915470b8-152a-11e7-b31a-0242ac12000d",
     "previous": "91546d0c-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"took\" : 10,\n",
      "  \"timed_out\" : false,\n",
      "  \"_shards\" : {\n",
      "    \"total\" : 5,\n",
      "    \"successful\" : 5,\n",
      "    \"failed\" : 0\n",
      "  },\n",
      "  \"hits\" : {\n",
      "    \"total\" : 31,\n",
      "    \"max_score\" : 1.0,\n",
      "    \"hits\" : [\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_multiline_json\",\n",
      "        \"_id\" : \"AVjxZOVqeJQ1fjM9_cy7\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"4.4\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"8.2\",\n",
      "          \"precipitation_day\" : \"0\",\n",
      "          \"wind_speed_max\" : \"9.4\",\n",
      "          \"sunshine_duration\" : \"1.3\",\n",
      "          \"temperature_avg\" : \"3.7\",\n",
      "          \"temperature_min\" : \"0.7\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"1\",\n",
      "          \"precipitation_max_10min\" : \"0\",\n",
      "          \"precipitation_max_hour\" : \"0\",\n",
      "          \"wind_direction\" : \"西\",\n",
      "          \"message\" : \"{\\r\\n\\\"timestamp\\\":\\\"1\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1000.4\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1003.4\\\",\\r\\n\\\"precipitation_day\\\":\\\"0\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"0\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"0\\\",\\r\\n\\\"temperature_avg\\\":\\\"3.7\\\",\\r\\n\\\"temperature_max\\\":\\\"8.2\\\",\\r\\n\\\"temperature_min\\\":\\\"0.7\\\",\\r\\n\\\"humidity_avg\\\":\\\"41\\\",\\r\\n\\\"humidity_min\\\":\\\"30\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"4.4\\\",\\r\\n\\\"wind_speed_max\\\":\\\"9.4\\\",\\r\\n\\\"wind_direction\\\":\\\"西\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"18.7\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"北西\\\",\\r\\n\\\"sunshine_duration\\\":\\\"1.3\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"時々晴一時雪\\\",\\r\\n\\\"information_night\\\":\\\"晴\\\"\\r\\n}\\r\",\n",
      "          \"information_daytime\" : \"時々晴一時雪\",\n",
      "          \"tags\" : [\n",
      "            \"multiline\"\n",
      "          ],\n",
      "          \"@timestamp\" : \"2016-12-12T04:57:50.634Z\",\n",
      "          \"humidity_avg\" : \"41\",\n",
      "          \"humidity_min\" : \"30\",\n",
      "          \"atmospheric_pressure\" : \"1000.4\",\n",
      "          \"wind_speed_max_moment\" : \"18.7\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1003.4\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_multiline_json\",\n",
      "        \"_id\" : \"AVjxZOVqeJQ1fjM9_czE\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"3.5\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"11.2\",\n",
      "          \"precipitation_day\" : \"\",\n",
      "          \"wind_speed_max\" : \"8.8\",\n",
      "          \"sunshine_duration\" : \"9.1\",\n",
      "          \"temperature_avg\" : \"6.3\",\n",
      "          \"temperature_min\" : \"1.9\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"西\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"10\",\n",
      "          \"precipitation_max_10min\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"\",\n",
      "          \"wind_direction\" : \"西\",\n",
      "          \"message\" : \"{\\r\\n\\\"timestamp\\\":\\\"10\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1008.8\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1011.8\\\",\\r\\n\\\"precipitation_day\\\":\\\"\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"\\\",\\r\\n\\\"temperature_avg\\\":\\\"6.3\\\",\\r\\n\\\"temperature_max\\\":\\\"11.2\\\",\\r\\n\\\"temperature_min\\\":\\\"1.9\\\",\\r\\n\\\"humidity_avg\\\":\\\"31\\\",\\r\\n\\\"humidity_min\\\":\\\"16\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"3.5\\\",\\r\\n\\\"wind_speed_max\\\":\\\"8.8\\\",\\r\\n\\\"wind_direction\\\":\\\"西\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"17.7\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"西\\\",\\r\\n\\\"sunshine_duration\\\":\\\"9.1\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"晴\\\",\\r\\n\\\"information_night\\\":\\\"晴\\\"\\r\\n}\\r\",\n",
      "          \"information_daytime\" : \"晴\",\n",
      "          \"tags\" : [\n",
      "            \"multiline\"\n",
      "          ],\n",
      "          \"@timestamp\" : \"2016-12-12T04:57:51.008Z\",\n",
      "          \"humidity_avg\" : \"31\",\n",
      "          \"humidity_min\" : \"16\",\n",
      "          \"atmospheric_pressure\" : \"1008.8\",\n",
      "          \"wind_speed_max_moment\" : \"17.7\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1011.8\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_multiline_json\",\n",
      "        \"_id\" : \"AVjxZOVqeJQ1fjM9_czN\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"2.5\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"11.7\",\n",
      "          \"precipitation_day\" : \"\",\n",
      "          \"wind_speed_max\" : \"5.7\",\n",
      "          \"sunshine_duration\" : \"9\",\n",
      "          \"temperature_avg\" : \"5.7\",\n",
      "          \"temperature_min\" : \"-0.7\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"19\",\n",
      "          \"precipitation_max_10min\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"message\" : \"{\\r\\n\\\"timestamp\\\":\\\"19\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1008.8\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1011.8\\\",\\r\\n\\\"precipitation_day\\\":\\\"\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"\\\",\\r\\n\\\"temperature_avg\\\":\\\"5.7\\\",\\r\\n\\\"temperature_max\\\":\\\"11.7\\\",\\r\\n\\\"temperature_min\\\":\\\"-0.7\\\",\\r\\n\\\"humidity_avg\\\":\\\"44\\\",\\r\\n\\\"humidity_min\\\":\\\"25\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"2.5\\\",\\r\\n\\\"wind_speed_max\\\":\\\"5.7\\\",\\r\\n\\\"wind_direction\\\":\\\"北西\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"10.2\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"北西\\\",\\r\\n\\\"sunshine_duration\\\":\\\"9\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"\\\",\\r\\n\\\"information_night\\\":\\\"晴\\\"\\r\\n}\\r\",\n",
      "          \"information_daytime\" : \"\",\n",
      "          \"tags\" : [\n",
      "            \"multiline\"\n",
      "          ],\n",
      "          \"@timestamp\" : \"2016-12-12T04:57:51.021Z\",\n",
      "          \"humidity_avg\" : \"44\",\n",
      "          \"humidity_min\" : \"25\",\n",
      "          \"atmospheric_pressure\" : \"1008.8\",\n",
      "          \"wind_speed_max_moment\" : \"10.2\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1011.8\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_multiline_json\",\n",
      "        \"_id\" : \"AVjxZOVqeJQ1fjM9_czP\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"3.1\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"3.8\",\n",
      "          \"precipitation_day\" : \"4\",\n",
      "          \"wind_speed_max\" : \"5\",\n",
      "          \"sunshine_duration\" : \"0\",\n",
      "          \"temperature_avg\" : \"2.5\",\n",
      "          \"temperature_min\" : \"1.5\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"information_night\" : \"一時雨\",\n",
      "          \"timestamp\" : \"21\",\n",
      "          \"precipitation_max_10min\" : \"0.5\",\n",
      "          \"precipitation_max_hour\" : \"1.5\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"message\" : \"{\\r\\n\\\"timestamp\\\":\\\"21\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1024.3\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1027.4\\\",\\r\\n\\\"precipitation_day\\\":\\\"4\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"1.5\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"0.5\\\",\\r\\n\\\"temperature_avg\\\":\\\"2.5\\\",\\r\\n\\\"temperature_max\\\":\\\"3.8\\\",\\r\\n\\\"temperature_min\\\":\\\"1.5\\\",\\r\\n\\\"humidity_avg\\\":\\\"69\\\",\\r\\n\\\"humidity_min\\\":\\\"38\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"3.1\\\",\\r\\n\\\"wind_speed_max\\\":\\\"5\\\",\\r\\n\\\"wind_direction\\\":\\\"北西\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"9.3\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"北西\\\",\\r\\n\\\"sunshine_duration\\\":\\\"0\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"時々雪後雨\\\",\\r\\n\\\"information_night\\\":\\\"一時雨\\\"\\r\\n}\\r\",\n",
      "          \"information_daytime\" : \"時々雪後雨\",\n",
      "          \"tags\" : [\n",
      "            \"multiline\"\n",
      "          ],\n",
      "          \"@timestamp\" : \"2016-12-12T04:57:51.024Z\",\n",
      "          \"humidity_avg\" : \"69\",\n",
      "          \"humidity_min\" : \"38\",\n",
      "          \"atmospheric_pressure\" : \"1024.3\",\n",
      "          \"wind_speed_max_moment\" : \"9.3\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1027.4\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_multiline_json\",\n",
      "        \"_id\" : \"AVjxZOVqeJQ1fjM9_czR\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"4.3\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"12.8\",\n",
      "          \"precipitation_day\" : \"7.5\",\n",
      "          \"wind_speed_max\" : \"9.1\",\n",
      "          \"sunshine_duration\" : \"6.8\",\n",
      "          \"temperature_avg\" : \"6.9\",\n",
      "          \"temperature_min\" : \"3.7\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"23\",\n",
      "          \"precipitation_max_10min\" : \"1\",\n",
      "          \"precipitation_max_hour\" : \"2.5\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"message\" : \"{\\r\\n\\\"timestamp\\\":\\\"23\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1004.1\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1007.1\\\",\\r\\n\\\"precipitation_day\\\":\\\"7.5\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"2.5\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"1\\\",\\r\\n\\\"temperature_avg\\\":\\\"6.9\\\",\\r\\n\\\"temperature_max\\\":\\\"12.8\\\",\\r\\n\\\"temperature_min\\\":\\\"3.7\\\",\\r\\n\\\"humidity_avg\\\":\\\"63\\\",\\r\\n\\\"humidity_min\\\":\\\"33\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"4.3\\\",\\r\\n\\\"wind_speed_max\\\":\\\"9.1\\\",\\r\\n\\\"wind_direction\\\":\\\"北西\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"17.6\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"\\\",\\r\\n\\\"sunshine_duration\\\":\\\"6.8\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"一時曇\\\",\\r\\n\\\"information_night\\\":\\\"晴\\\"\\r\\n}\\r\",\n",
      "          \"information_daytime\" : \"一時曇\",\n",
      "          \"tags\" : [\n",
      "            \"multiline\"\n",
      "          ],\n",
      "          \"@timestamp\" : \"2016-12-12T04:57:51.027Z\",\n",
      "          \"humidity_avg\" : \"63\",\n",
      "          \"humidity_min\" : \"33\",\n",
      "          \"atmospheric_pressure\" : \"1004.1\",\n",
      "          \"wind_speed_max_moment\" : \"17.6\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1007.1\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_multiline_json\",\n",
      "        \"_id\" : \"AVjxZOVqeJQ1fjM9_czT\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"2\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"12.2\",\n",
      "          \"precipitation_day\" : \"\",\n",
      "          \"wind_speed_max\" : \"4.9\",\n",
      "          \"sunshine_duration\" : \"7.1\",\n",
      "          \"temperature_avg\" : \"7\",\n",
      "          \"temperature_min\" : \"0.9\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"南東\",\n",
      "          \"information_night\" : \"後一時雨\",\n",
      "          \"timestamp\" : \"25\",\n",
      "          \"precipitation_max_10min\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"\",\n",
      "          \"wind_direction\" : \"東\",\n",
      "          \"message\" : \"{\\r\\n\\\"timestamp\\\":\\\"25\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1024.3\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1027.3\\\",\\r\\n\\\"precipitation_day\\\":\\\"\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"\\\",\\r\\n\\\"temperature_avg\\\":\\\"7\\\",\\r\\n\\\"temperature_max\\\":\\\"12.2\\\",\\r\\n\\\"temperature_min\\\":\\\"0.9\\\",\\r\\n\\\"humidity_avg\\\":\\\"59\\\",\\r\\n\\\"humidity_min\\\":\\\"40\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"2\\\",\\r\\n\\\"wind_speed_max\\\":\\\"4.9\\\",\\r\\n\\\"wind_direction\\\":\\\"東\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"8.2\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"南東\\\",\\r\\n\\\"sunshine_duration\\\":\\\"7.1\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"後薄曇\\\",\\r\\n\\\"information_night\\\":\\\"後一時雨\\\"\\r\\n}\\r\",\n",
      "          \"information_daytime\" : \"後薄曇\",\n",
      "          \"tags\" : [\n",
      "            \"multiline\"\n",
      "          ],\n",
      "          \"@timestamp\" : \"2016-12-12T04:57:51.029Z\",\n",
      "          \"humidity_avg\" : \"59\",\n",
      "          \"humidity_min\" : \"40\",\n",
      "          \"atmospheric_pressure\" : \"1024.3\",\n",
      "          \"wind_speed_max_moment\" : \"8.2\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1027.3\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_multiline_json\",\n",
      "        \"_id\" : \"AVjxZOVqeJQ1fjM9_czV\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"3.1\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"16.4\",\n",
      "          \"precipitation_day\" : \"14.5\",\n",
      "          \"wind_speed_max\" : \"6.3\",\n",
      "          \"sunshine_duration\" : \"5.1\",\n",
      "          \"temperature_avg\" : \"9.8\",\n",
      "          \"temperature_min\" : \"5.8\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"西\",\n",
      "          \"information_night\" : \"時々雨後一時みぞれ\",\n",
      "          \"timestamp\" : \"27\",\n",
      "          \"precipitation_max_10min\" : \"1.5\",\n",
      "          \"precipitation_max_hour\" : \"6\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"message\" : \"{\\r\\n\\\"timestamp\\\":\\\"27\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1007\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1009.9\\\",\\r\\n\\\"precipitation_day\\\":\\\"14.5\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"6\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"1.5\\\",\\r\\n\\\"temperature_avg\\\":\\\"9.8\\\",\\r\\n\\\"temperature_max\\\":\\\"16.4\\\",\\r\\n\\\"temperature_min\\\":\\\"5.8\\\",\\r\\n\\\"humidity_avg\\\":\\\"71\\\",\\r\\n\\\"humidity_min\\\":\\\"40\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"3.1\\\",\\r\\n\\\"wind_speed_max\\\":\\\"6.3\\\",\\r\\n\\\"wind_direction\\\":\\\"北西\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"12.1\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"西\\\",\\r\\n\\\"sunshine_duration\\\":\\\"5.1\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"時々曇一時雨\\\",\\r\\n\\\"information_night\\\":\\\"時々雨後一時みぞれ\\\"\\r\\n}\\r\",\n",
      "          \"information_daytime\" : \"時々曇一時雨\",\n",
      "          \"tags\" : [\n",
      "            \"multiline\"\n",
      "          ],\n",
      "          \"@timestamp\" : \"2016-12-12T04:57:51.030Z\",\n",
      "          \"humidity_avg\" : \"71\",\n",
      "          \"humidity_min\" : \"40\",\n",
      "          \"atmospheric_pressure\" : \"1007\",\n",
      "          \"wind_speed_max_moment\" : \"12.1\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1009.9\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_multiline_json\",\n",
      "        \"_id\" : \"AVjxZOVqeJQ1fjM9_cy-\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"1.3\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"9.3\",\n",
      "          \"precipitation_day\" : \"\",\n",
      "          \"wind_speed_max\" : \"2.8\",\n",
      "          \"sunshine_duration\" : \"6\",\n",
      "          \"temperature_avg\" : \"4\",\n",
      "          \"temperature_min\" : \"-0.4\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"北東\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"4\",\n",
      "          \"precipitation_max_10min\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"message\" : \"{\\r\\n\\\"timestamp\\\":\\\"4\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1010.4\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1013.4\\\",\\r\\n\\\"precipitation_day\\\":\\\"\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"\\\",\\r\\n\\\"temperature_avg\\\":\\\"4\\\",\\r\\n\\\"temperature_max\\\":\\\"9.3\\\",\\r\\n\\\"temperature_min\\\":\\\"-0.4\\\",\\r\\n\\\"humidity_avg\\\":\\\"51\\\",\\r\\n\\\"humidity_min\\\":\\\"33\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"1.3\\\",\\r\\n\\\"wind_speed_max\\\":\\\"2.8\\\",\\r\\n\\\"wind_direction\\\":\\\"北西\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"3.8\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"北東\\\",\\r\\n\\\"sunshine_duration\\\":\\\"6\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"後晴\\\",\\r\\n\\\"information_night\\\":\\\"晴\\\"\\r\\n}\\r\",\n",
      "          \"information_daytime\" : \"後晴\",\n",
      "          \"tags\" : [\n",
      "            \"multiline\"\n",
      "          ],\n",
      "          \"@timestamp\" : \"2016-12-12T04:57:50.992Z\",\n",
      "          \"humidity_avg\" : \"51\",\n",
      "          \"humidity_min\" : \"33\",\n",
      "          \"atmospheric_pressure\" : \"1010.4\",\n",
      "          \"wind_speed_max_moment\" : \"3.8\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1013.4\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_multiline_json\",\n",
      "        \"_id\" : \"AVjxZOVqeJQ1fjM9_czA\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"5.2\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"16\",\n",
      "          \"precipitation_day\" : \"6.5\",\n",
      "          \"wind_speed_max\" : \"8.9\",\n",
      "          \"sunshine_duration\" : \"0.2\",\n",
      "          \"temperature_avg\" : \"10.8\",\n",
      "          \"temperature_min\" : \"5.8\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"\",\n",
      "          \"information_night\" : \"時々晴\",\n",
      "          \"timestamp\" : \"6\",\n",
      "          \"precipitation_max_10min\" : \"1.5\",\n",
      "          \"precipitation_max_hour\" : \"5\",\n",
      "          \"wind_direction\" : \"南西\",\n",
      "          \"message\" : \"{\\r\\n\\\"timestamp\\\":\\\"6\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1000.6\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1003.5\\\",\\r\\n\\\"precipitation_day\\\":\\\"6.5\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"5\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"1.5\\\",\\r\\n\\\"temperature_avg\\\":\\\"10.8\\\",\\r\\n\\\"temperature_max\\\":\\\"16\\\",\\r\\n\\\"temperature_min\\\":\\\"5.8\\\",\\r\\n\\\"humidity_avg\\\":\\\"64\\\",\\r\\n\\\"humidity_min\\\":\\\"40\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"5.2\\\",\\r\\n\\\"wind_speed_max\\\":\\\"8.9\\\",\\r\\n\\\"wind_direction\\\":\\\"南西\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"19.3\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"\\\",\\r\\n\\\"sunshine_duration\\\":\\\"0.2\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"後時々雨\\\",\\r\\n\\\"information_night\\\":\\\"時々晴\\\"\\r\\n}\\r\",\n",
      "          \"information_daytime\" : \"後時々雨\",\n",
      "          \"tags\" : [\n",
      "            \"multiline\"\n",
      "          ],\n",
      "          \"@timestamp\" : \"2016-12-12T04:57:50.999Z\",\n",
      "          \"humidity_avg\" : \"64\",\n",
      "          \"humidity_min\" : \"40\",\n",
      "          \"atmospheric_pressure\" : \"1000.6\",\n",
      "          \"wind_speed_max_moment\" : \"19.3\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1003.5\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_multiline_json\",\n",
      "        \"_id\" : \"AVjxZOVqeJQ1fjM9_czB\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"4.2\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"10.2\",\n",
      "          \"precipitation_day\" : \"\",\n",
      "          \"wind_speed_max\" : \"7\",\n",
      "          \"sunshine_duration\" : \"3.3\",\n",
      "          \"temperature_avg\" : \"5.6\",\n",
      "          \"temperature_min\" : \"3\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"7\",\n",
      "          \"precipitation_max_10min\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"message\" : \"{\\r\\n\\\"timestamp\\\":\\\"7\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1000.5\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1003.4\\\",\\r\\n\\\"precipitation_day\\\":\\\"\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"\\\",\\r\\n\\\"temperature_avg\\\":\\\"5.6\\\",\\r\\n\\\"temperature_max\\\":\\\"10.2\\\",\\r\\n\\\"temperature_min\\\":\\\"3\\\",\\r\\n\\\"humidity_avg\\\":\\\"35\\\",\\r\\n\\\"humidity_min\\\":\\\"23\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"4.2\\\",\\r\\n\\\"wind_speed_max\\\":\\\"7\\\",\\r\\n\\\"wind_direction\\\":\\\"北西\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"16.1\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"北西\\\",\\r\\n\\\"sunshine_duration\\\":\\\"3.3\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"時々曇\\\",\\r\\n\\\"information_night\\\":\\\"晴\\\"\\r\\n}\\r\",\n",
      "          \"information_daytime\" : \"時々曇\",\n",
      "          \"tags\" : [\n",
      "            \"multiline\"\n",
      "          ],\n",
      "          \"@timestamp\" : \"2016-12-12T04:57:51.001Z\",\n",
      "          \"humidity_avg\" : \"35\",\n",
      "          \"humidity_min\" : \"23\",\n",
      "          \"atmospheric_pressure\" : \"1000.5\",\n",
      "          \"wind_speed_max_moment\" : \"16.1\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1003.4\"\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100 19387  100 19387    0     0   873k      0 --:--:-- --:--:-- --:--:--  901k\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/json_multiline_$INDEX/_search?pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915470b8-152a-11e7-b31a-0242ac12000d",
     "next": "91547298-152a-11e7-b31a-0242ac12000d",
     "previous": "91546eec-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### CSV\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91547298-152a-11e7-b31a-0242ac12000d",
     "next": "915474aa-152a-11e7-b31a-0242ac12000d",
     "previous": "915470b8-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "CSV形式のデータとして、次の内容のサンプルを登録します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "lc_cell_meme": {
     "current": "915474aa-152a-11e7-b31a-0242ac12000d",
     "next": "91547676-152a-11e7-b31a-0242ac12000d",
     "previous": "91547298-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1,1000.4,1003.4,0,0,0,3.7,8.2,0.7,41,30,4.4,9.4,西,18.7,北西,1.3,,,時々晴一時雪,晴\r",
      "\r\n",
      "2,1007.4,1010.4,,,,2.7,7.9,-2.2,41,19,2.1,4.2,西,7.8,西,6,,,時々曇,一時曇\r",
      "\r\n",
      "3,1012,1015.1,,,,3.8,8.9,-1.1,42,19,3,7.4,北西,10.9,北西,8.9,,,晴,後薄曇\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 3 sample_data/weather.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91547676-152a-11e7-b31a-0242ac12000d",
     "next": "91547842-152a-11e7-b31a-0242ac12000d",
     "previous": "915474aa-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "confファイルには、\"filter\"の下に\"csv\"定義を設定し、列ごとの名前を指定することで、CSVデータの各列を対応する要素に紐づけてCSV形式を解釈できるようになります。  \n",
    "ファイル名は\"csv.conf\"とします。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91547842-152a-11e7-b31a-0242ac12000d",
     "next": "91547a04-152a-11e7-b31a-0242ac12000d",
     "previous": "91547676-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "```ruby:csv.conf\n",
    "input{\n",
    "   stdin{}\n",
    "}\n",
    "filter{\n",
    "  csv{\n",
    "    columns => [\"timestamp\",\"atmospheric_pressure\",\"sea_level_pressure\",\"precipitation_day\",\"precipitation_max_hour\",\"precipitation_max_10min\",\"temperature_avg\",\"temperature_max\",\"temperature_min\",\"humidity_avg\",\"humidity_min\",\"wind_speed_avg\",\"wind_speed_max\",\"wind_direction\",\"wind_speed_max_moment\",\"wind_direction_max_moment\",\"sunshine_duration\",\"snowfall\",\"snowfall_max\",\"information_daytime\",\"information_night\"]\n",
    "  }\n",
    "}\n",
    "output{\n",
    "  elasticsearch{\n",
    "    hosts => <ES_HOST>  \n",
    "    index => <INDEX>\n",
    "  }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91547a04-152a-11e7-b31a-0242ac12000d",
     "next": "91547bd0-152a-11e7-b31a-0242ac12000d",
     "previous": "91547842-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "実際のデータ登録は次のコマンドで実行します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "lc_cell_meme": {
     "current": "91547bd0-152a-11e7-b31a-0242ac12000d",
     "next": "91547da6-152a-11e7-b31a-0242ac12000d",
     "previous": "91547a04-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\r\n",
      "Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties.\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible logstash-server -i ./hosts -m shell -a \"cat weather.csv | sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/csv.conf\" -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91547da6-152a-11e7-b31a-0242ac12000d",
     "next": "91547f68-152a-11e7-b31a-0242ac12000d",
     "previous": "91547bd0-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "コマンドの実行に成功したら、次の検索APIで正しく登録されているか確認してください。  \n",
    "正しく登録されていれば、json形式で登録されたデータが出力されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "lc_cell_meme": {
     "current": "91547f68-152a-11e7-b31a-0242ac12000d",
     "next": "91548152-152a-11e7-b31a-0242ac12000d",
     "previous": "91547da6-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"took\" : 11,\n",
      "  \"timed_out\" : false,\n",
      "  \"_shards\" : {\n",
      "    \"total\" : 5,\n",
      "    \"successful\" : 5,\n",
      "    \"failed\" : 0\n",
      "  },\n",
      "  \"hits\" : {\n",
      "    \"total\" : 30,\n",
      "    \"max_score\" : 1.0,\n",
      "    \"hits\" : [\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_csv\",\n",
      "        \"_id\" : \"AVjxZdZ8eJQ1fjM9_czo\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"3.2\",\n",
      "          \"snowfall\" : null,\n",
      "          \"temperature_max\" : \"8.2\",\n",
      "          \"precipitation_day\" : \"35.5\",\n",
      "          \"wind_speed_max\" : \"8.1\",\n",
      "          \"sunshine_duration\" : \"0\",\n",
      "          \"temperature_avg\" : \"5.4\",\n",
      "          \"temperature_min\" : \"3.4\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"西\",\n",
      "          \"information_night\" : \"後晴\",\n",
      "          \"timestamp\" : \"15\",\n",
      "          \"precipitation_max_10min\" : \"2.5\",\n",
      "          \"precipitation_max_hour\" : \"11\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"message\" : \"15,1009.7,1012.7,35.5,11,2.5,5.4,8.2,3.4,76,46,3.2,8.1,北西,14.1,西,0,,,雨一時曇,後晴\\r\",\n",
      "          \"information_daytime\" : \"雨一時曇\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:58:52.714Z\",\n",
      "          \"humidity_avg\" : \"76\",\n",
      "          \"humidity_min\" : \"46\",\n",
      "          \"atmospheric_pressure\" : \"1009.7\",\n",
      "          \"wind_speed_max_moment\" : \"14.1\",\n",
      "          \"snowfall_max\" : null,\n",
      "          \"sea_level_pressure\" : \"1012.7\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_csv\",\n",
      "        \"_id\" : \"AVjxZdZ8eJQ1fjM9_czp\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"2.8\",\n",
      "          \"snowfall\" : null,\n",
      "          \"temperature_max\" : \"13.2\",\n",
      "          \"precipitation_day\" : \"0\",\n",
      "          \"wind_speed_max\" : \"5.7\",\n",
      "          \"sunshine_duration\" : \"6.9\",\n",
      "          \"temperature_avg\" : \"8.9\",\n",
      "          \"temperature_min\" : \"4.9\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"information_night\" : \"後晴\",\n",
      "          \"timestamp\" : \"16\",\n",
      "          \"precipitation_max_10min\" : \"0\",\n",
      "          \"precipitation_max_hour\" : \"0\",\n",
      "          \"wind_direction\" : \"東\",\n",
      "          \"message\" : \"16,1009.7,1012.6,0,0,0,8.9,13.2,4.9,54,23,2.8,5.7,東,10.7,北西,6.9,,,後曇,後晴\\r\",\n",
      "          \"information_daytime\" : \"後曇\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:58:52.715Z\",\n",
      "          \"humidity_avg\" : \"54\",\n",
      "          \"humidity_min\" : \"23\",\n",
      "          \"atmospheric_pressure\" : \"1009.7\",\n",
      "          \"wind_speed_max_moment\" : \"10.7\",\n",
      "          \"snowfall_max\" : null,\n",
      "          \"sea_level_pressure\" : \"1012.6\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_csv\",\n",
      "        \"_id\" : \"AVjxZdZ8eJQ1fjM9_czr\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"3.3\",\n",
      "          \"snowfall\" : null,\n",
      "          \"temperature_max\" : \"10.9\",\n",
      "          \"precipitation_day\" : null,\n",
      "          \"wind_speed_max\" : \"6.4\",\n",
      "          \"sunshine_duration\" : \"9.3\",\n",
      "          \"temperature_avg\" : \"4.7\",\n",
      "          \"temperature_min\" : \"-0.5\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"information_night\" : \"時々薄曇\",\n",
      "          \"timestamp\" : \"18\",\n",
      "          \"precipitation_max_10min\" : null,\n",
      "          \"precipitation_max_hour\" : null,\n",
      "          \"wind_direction\" : \"西\",\n",
      "          \"message\" : \"18,1014.6,1017.6,,,,4.7,10.9,-0.5,36,18,3.3,6.4,西,11.5,北西,9.3,,,晴,時々薄曇\\r\",\n",
      "          \"information_daytime\" : \"晴\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:58:52.719Z\",\n",
      "          \"humidity_avg\" : \"36\",\n",
      "          \"humidity_min\" : \"18\",\n",
      "          \"atmospheric_pressure\" : \"1014.6\",\n",
      "          \"wind_speed_max_moment\" : \"11.5\",\n",
      "          \"snowfall_max\" : null,\n",
      "          \"sea_level_pressure\" : \"1017.6\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_csv\",\n",
      "        \"_id\" : \"AVjxZdZ8eJQ1fjM9_czu\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"3.1\",\n",
      "          \"snowfall\" : null,\n",
      "          \"temperature_max\" : \"3.8\",\n",
      "          \"precipitation_day\" : \"4\",\n",
      "          \"wind_speed_max\" : \"5\",\n",
      "          \"sunshine_duration\" : \"0\",\n",
      "          \"temperature_avg\" : \"2.5\",\n",
      "          \"temperature_min\" : \"1.5\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"information_night\" : \"一時雨\",\n",
      "          \"timestamp\" : \"21\",\n",
      "          \"precipitation_max_10min\" : \"0.5\",\n",
      "          \"precipitation_max_hour\" : \"1.5\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"message\" : \"21,1024.3,1027.4,4,1.5,0.5,2.5,3.8,1.5,69,38,3.1,5,北西,9.3,北西,0,,,時々雪後雨,一時雨\\r\",\n",
      "          \"information_daytime\" : \"時々雪後雨\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:58:52.721Z\",\n",
      "          \"humidity_avg\" : \"69\",\n",
      "          \"humidity_min\" : \"38\",\n",
      "          \"atmospheric_pressure\" : \"1024.3\",\n",
      "          \"wind_speed_max_moment\" : \"9.3\",\n",
      "          \"snowfall_max\" : null,\n",
      "          \"sea_level_pressure\" : \"1027.4\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_csv\",\n",
      "        \"_id\" : \"AVjxZdZ8eJQ1fjM9_czw\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"4.3\",\n",
      "          \"snowfall\" : null,\n",
      "          \"temperature_max\" : \"12.8\",\n",
      "          \"precipitation_day\" : \"7.5\",\n",
      "          \"wind_speed_max\" : \"9.1\",\n",
      "          \"sunshine_duration\" : \"6.8\",\n",
      "          \"temperature_avg\" : \"6.9\",\n",
      "          \"temperature_min\" : \"3.7\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : null,\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"23\",\n",
      "          \"precipitation_max_10min\" : \"1\",\n",
      "          \"precipitation_max_hour\" : \"2.5\",\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"message\" : \"23,1004.1,1007.1,7.5,2.5,1,6.9,12.8,3.7,63,33,4.3,9.1,北西,17.6,,6.8,,,一時曇,晴\\r\",\n",
      "          \"information_daytime\" : \"一時曇\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:58:52.722Z\",\n",
      "          \"humidity_avg\" : \"63\",\n",
      "          \"humidity_min\" : \"33\",\n",
      "          \"atmospheric_pressure\" : \"1004.1\",\n",
      "          \"wind_speed_max_moment\" : \"17.6\",\n",
      "          \"snowfall_max\" : null,\n",
      "          \"sea_level_pressure\" : \"1007.1\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_csv\",\n",
      "        \"_id\" : \"AVjxZdZ8eJQ1fjM9_cz2\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"2.5\",\n",
      "          \"snowfall\" : null,\n",
      "          \"temperature_max\" : \"8.2\",\n",
      "          \"precipitation_day\" : null,\n",
      "          \"wind_speed_max\" : \"5.1\",\n",
      "          \"sunshine_duration\" : \"6.6\",\n",
      "          \"temperature_avg\" : \"3.9\",\n",
      "          \"temperature_min\" : \"-0.9\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"西\",\n",
      "          \"information_night\" : \"後一時雪\",\n",
      "          \"timestamp\" : \"29\",\n",
      "          \"precipitation_max_10min\" : null,\n",
      "          \"precipitation_max_hour\" : null,\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"message\" : \"29,1023.4,1026.4,,,,3.9,8.2,-0.9,42,22,2.5,5.1,北西,9.7,西,6.6,,,後曇,後一時雪\\r\",\n",
      "          \"information_daytime\" : \"後曇\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:58:52.725Z\",\n",
      "          \"humidity_avg\" : \"42\",\n",
      "          \"humidity_min\" : \"22\",\n",
      "          \"atmospheric_pressure\" : \"1023.4\",\n",
      "          \"wind_speed_max_moment\" : \"9.7\",\n",
      "          \"snowfall_max\" : null,\n",
      "          \"sea_level_pressure\" : \"1026.4\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_csv\",\n",
      "        \"_id\" : \"AVjxZdZ8eJQ1fjM9_cza\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"4.4\",\n",
      "          \"snowfall\" : null,\n",
      "          \"temperature_max\" : \"8.2\",\n",
      "          \"precipitation_day\" : \"0\",\n",
      "          \"wind_speed_max\" : \"9.4\",\n",
      "          \"sunshine_duration\" : \"1.3\",\n",
      "          \"temperature_avg\" : \"3.7\",\n",
      "          \"temperature_min\" : \"0.7\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"1\",\n",
      "          \"precipitation_max_10min\" : \"0\",\n",
      "          \"precipitation_max_hour\" : \"0\",\n",
      "          \"wind_direction\" : \"西\",\n",
      "          \"message\" : \"1,1000.4,1003.4,0,0,0,3.7,8.2,0.7,41,30,4.4,9.4,西,18.7,北西,1.3,,,時々晴一時雪,晴\\r\",\n",
      "          \"information_daytime\" : \"時々晴一時雪\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:58:52.330Z\",\n",
      "          \"humidity_avg\" : \"41\",\n",
      "          \"humidity_min\" : \"30\",\n",
      "          \"atmospheric_pressure\" : \"1000.4\",\n",
      "          \"wind_speed_max_moment\" : \"18.7\",\n",
      "          \"snowfall_max\" : null,\n",
      "          \"sea_level_pressure\" : \"1003.4\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_csv\",\n",
      "        \"_id\" : \"AVjxZdZ8eJQ1fjM9_cze\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"2.4\",\n",
      "          \"snowfall\" : null,\n",
      "          \"temperature_max\" : \"12.9\",\n",
      "          \"precipitation_day\" : null,\n",
      "          \"wind_speed_max\" : \"5\",\n",
      "          \"sunshine_duration\" : \"8.9\",\n",
      "          \"temperature_avg\" : \"6.9\",\n",
      "          \"temperature_min\" : \"1.3\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"information_night\" : \"一時晴\",\n",
      "          \"timestamp\" : \"5\",\n",
      "          \"precipitation_max_10min\" : null,\n",
      "          \"precipitation_max_hour\" : null,\n",
      "          \"wind_direction\" : \"北西\",\n",
      "          \"message\" : \"5,1014.6,1017.5,,,,6.9,12.9,1.3,56,29,2.4,5,北西,9.1,北西,8.9,,,晴,一時晴\\r\",\n",
      "          \"information_daytime\" : \"晴\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:58:52.704Z\",\n",
      "          \"humidity_avg\" : \"56\",\n",
      "          \"humidity_min\" : \"29\",\n",
      "          \"atmospheric_pressure\" : \"1014.6\",\n",
      "          \"wind_speed_max_moment\" : \"9.1\",\n",
      "          \"snowfall_max\" : null,\n",
      "          \"sea_level_pressure\" : \"1017.5\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_csv\",\n",
      "        \"_id\" : \"AVjxZdZ8eJQ1fjM9_czh\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"4.4\",\n",
      "          \"snowfall\" : null,\n",
      "          \"temperature_max\" : \"11.2\",\n",
      "          \"precipitation_day\" : null,\n",
      "          \"wind_speed_max\" : \"8.4\",\n",
      "          \"sunshine_duration\" : \"8.9\",\n",
      "          \"temperature_avg\" : \"6.2\",\n",
      "          \"temperature_min\" : \"2\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"8\",\n",
      "          \"precipitation_max_10min\" : null,\n",
      "          \"precipitation_max_hour\" : null,\n",
      "          \"wind_direction\" : \"西\",\n",
      "          \"message\" : \"8,1006.2,1009.2,,,,6.2,11.2,2,33,20,4.4,8.4,西,13.6,北西,8.9,,,晴,晴\\r\",\n",
      "          \"information_daytime\" : \"晴\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:58:52.706Z\",\n",
      "          \"humidity_avg\" : \"33\",\n",
      "          \"humidity_min\" : \"20\",\n",
      "          \"atmospheric_pressure\" : \"1006.2\",\n",
      "          \"wind_speed_max_moment\" : \"13.6\",\n",
      "          \"snowfall_max\" : null,\n",
      "          \"sea_level_pressure\" : \"1009.2\"\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_csv\",\n",
      "        \"_id\" : \"AVjxZdZ8eJQ1fjM9_czz\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"wind_speed_avg\" : \"2\",\n",
      "          \"snowfall\" : null,\n",
      "          \"temperature_max\" : \"12.9\",\n",
      "          \"precipitation_day\" : \"0.5\",\n",
      "          \"wind_speed_max\" : \"3.8\",\n",
      "          \"sunshine_duration\" : \"4.4\",\n",
      "          \"temperature_avg\" : \"9\",\n",
      "          \"temperature_min\" : \"5.6\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"wind_direction_max_moment\" : \"南東\",\n",
      "          \"information_night\" : \"一時曇\",\n",
      "          \"timestamp\" : \"26\",\n",
      "          \"precipitation_max_10min\" : \"0.5\",\n",
      "          \"precipitation_max_hour\" : \"0.5\",\n",
      "          \"wind_direction\" : \"南東\",\n",
      "          \"message\" : \"26,1019.1,1022.1,0.5,0.5,0.5,9,12.9,5.6,74,52,2,3.8,南東,6,南東,4.4,,,時々晴,一時曇\\r\",\n",
      "          \"information_daytime\" : \"時々晴\",\n",
      "          \"@timestamp\" : \"2016-12-12T04:58:52.723Z\",\n",
      "          \"humidity_avg\" : \"74\",\n",
      "          \"humidity_min\" : \"52\",\n",
      "          \"atmospheric_pressure\" : \"1019.1\",\n",
      "          \"wind_speed_max_moment\" : \"6\",\n",
      "          \"snowfall_max\" : null,\n",
      "          \"sea_level_pressure\" : \"1022.1\"\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100 12573  100 12573    0     0   633k      0 --:--:-- --:--:-- --:--:--  646k\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/csv_$INDEX/_search?pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91548152-152a-11e7-b31a-0242ac12000d",
     "next": "91548328-152a-11e7-b31a-0242ac12000d",
     "previous": "91547f68-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### LTSV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91548328-152a-11e7-b31a-0242ac12000d",
     "next": "9154851c-152a-11e7-b31a-0242ac12000d",
     "previous": "91548152-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "LTSV形式のデータとして、次の内容のサンプルを登録します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154851c-152a-11e7-b31a-0242ac12000d",
     "next": "915486fc-152a-11e7-b31a-0242ac12000d",
     "previous": "91548328-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"timestamp\":\"21\"\t\"atmospheric_pressure\":\"1024.3\"\t\"sea_level_pressure\":\"1027.4\"\t\"precipitation_day\":\"4\"\t\"precipitation_max_hour\":\"1.5\"\t\"precipitation_max_10min\":\"0.5\"\t\"temperature_avg\":\"2.5\"\t\"temperature_max\":\"3.8\"\t\"temperature_min\":\"1.5\"\t\"humidity_avg\":\"69\"\t\"humidity_min\":\"38\"\t\"wind_speed_avg\":\"3.1\"\t\"wind_speed_max\":\"5\"\t\"wind_direction\":\"北西\"\t\"wind_speed_max_moment\":\"9.3\"\t\"wind_direction_max_moment\":\"北西\"\t\"sunshine_duration\":\"0\"\t\"snowfall\":\"\"\t\"snowfall_max\":\"\"\t\"information_daytime\":\"時々雪後雨\"\t\"information_night\":\"一時雨\"\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 3 sample_data/weather.ltsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915486fc-152a-11e7-b31a-0242ac12000d",
     "next": "915488d2-152a-11e7-b31a-0242ac12000d",
     "previous": "9154851c-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "confファイルの設定では、mutate filterを使って入力をタブで分割したあとに、その内容をパースします。  \n",
    "ファイル名は\"ltsv.conf\"とします。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915488d2-152a-11e7-b31a-0242ac12000d",
     "next": "91548a9e-152a-11e7-b31a-0242ac12000d",
     "previous": "915486fc-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "```ruby:ltsv.conf\n",
    "input{\n",
    "  stdin{}\n",
    "}\n",
    "filter{\n",
    "  mutate{\n",
    "    split => {\"message\" => \"    \"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"timestamp\\\":\\\"%{DATA:timestamp}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"atmospheric_pressure\\\":\\\"%{DATA:atmospheric_pressure}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"sea_level_pressure\\\":\\\"%{DATA:sea_level_pressure}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"precipitation_day\\\":\\\"%{DATA:precipitation_day}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"precipitation_max_hour\\\":\\\"%{DATA:precipitation_max_hour}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"precipitation_max_10min\\\":\\\"%{DATA:precipitation_max_10min}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"temperature_avg\\\":\\\"%{DATA:temperature_avg}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"temperature_max\\\":\\\"%{DATA:temperature_max}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"temperature_min\\\":\\\"%{DATA:temperature_min}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"humidity_avg\\\":\\\"%{DATA:humidity_avg}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"humidity_min\\\":\\\"%{DATA:humidity_min}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"wind_speed_avg\\\":\\\"%{DATA:wind_speed_avg}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"wind_speed_max\\\":\\\"%{DATA:wind_speed_max}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"wind_direction\\\":\\\"%{DATA:wind_direction}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"wind_direction_max_moment\\\":\\\"%{DATA:wind_direction_max_moment}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"sunshine_duration\\\":\\\"%{DATA:sunshine_duration}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"snowfall\\\":\\\"%{DATA:snowfall}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"snowfall_max\\\":\\\"%{DATA:snowfall_max}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"information_daytime\\\":\\\"%{DATA:information_daytime}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  grok{\n",
    "    match => {\n",
    "      \"message\" => \"\\\"information_night\\\":\\\"%{DATA:information_night}\\\"\"\n",
    "    }\n",
    "  }\n",
    "  date{\n",
    "    match => [\"timestamp\",\"ISO8601\"]\n",
    "    timezone => \"UTC\"\n",
    "    remove_field => [\"timestamp\",\"message\"]\n",
    "  }\n",
    "}\n",
    "output{\n",
    "  elasticsearch{\n",
    "    hosts => \"XXX.XXX.XXX.232\"\n",
    "    index => \"sample_index\"\n",
    "    document_type => \"sample_ltsv\"\n",
    "  }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91548a9e-152a-11e7-b31a-0242ac12000d",
     "next": "91548c74-152a-11e7-b31a-0242ac12000d",
     "previous": "915488d2-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "実際のデータ登録は次のコマンドで実行します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "lc_cell_meme": {
     "current": "91548c74-152a-11e7-b31a-0242ac12000d",
     "next": "91548e54-152a-11e7-b31a-0242ac12000d",
     "previous": "91548a9e-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\r\n",
      "Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties.\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible logstash-server -i ./hosts -m shell -a \"cat weather.ltsv | sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/ltsv.conf\" -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91548e54-152a-11e7-b31a-0242ac12000d",
     "next": "9154902a-152a-11e7-b31a-0242ac12000d",
     "previous": "91548c74-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "コマンドの実行に成功したら、次の検索APIで正しく登録されているか確認してください。  \n",
    "正しく登録されていれば、json形式で登録されたデータが出力されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154902a-152a-11e7-b31a-0242ac12000d",
     "next": "915491f6-152a-11e7-b31a-0242ac12000d",
     "previous": "91548e54-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"took\":10,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\":5,\"failed\":0},\"hits\":{\"total\":92,\"max_score\":1.0,\"hits\":[{\"_index\":\"sample_index\",\"_type\":\"sample_json\",\"_id\":\"AVjxYFq1eJQ1fjM9_cyd\",\"_score\":1.0,\"_source\":{\"precipitation_max_10min\":\"0\",\"wind_speed_avg\":\"4.4\",\"snowfall\":\"\",\"precipitation_max_hour\":\"0\",\"temperature_max\":\"8.2\",\"wind_direction\":\"西\",\"information_daytime\":\"時々晴一時雪\",\"precipitation_day\":\"0\",\"@timestamp\":\"2016-12-12T04:52:52.957Z\",\"humidity_avg\":\"41\",\"humidity_min\":\"30\",\"wind_speed_max\":\"9.4\",\"sunshine_duration\":\"1.3\",\"temperature_avg\":\"3.7\",\"temperature_min\":\"0.7\",\"@version\":\"1\",\"host\":\"cnXXXX1004.ecloud.nii.ac.jp\",\"atmospheric_pressure\":\"1000.4\",\"wind_direction_max_moment\":\"北西\",\"wind_speed_max_moment\":\"18.7\",\"snowfall_max\":\"\",\"sea_level_pressure\":\"1003.4\",\"information_night\":\"晴\",\"timestamp\":\"1\"}},{\"_index\":\"sample_index\",\"_type\":\"sample_json\",\"_id\":\"AVjxYFq1eJQ1fjM9_cyi\",\"_score\":1.0,\"_source\":{\"precipitation_max_10min\":\"1.5\",\"wind_speed_avg\":\"5.2\",\"snowfall\":\"\",\"precipitation_max_hour\":\"5\",\"temperature_max\":\"16\",\"wind_direction\":\"南西\",\"information_daytime\":\"後時々雨\",\"precipitation_day\":\"6.5\",\"@timestamp\":\"2016-12-12T04:52:53.293Z\",\"humidity_avg\":\"64\",\"humidity_min\":\"40\",\"wind_speed_max\":\"8.9\",\"sunshine_duration\":\"0.2\",\"temperature_avg\":\"10.8\",\"temperature_min\":\"5.8\",\"@version\":\"1\",\"host\":\"cnXXXX1004.ecloud.nii.ac.jp\",\"atmospheric_pressure\":\"1000.6\",\"wind_direction_max_moment\":\"\",\"wind_speed_max_moment\":\"19.3\",\"snowfall_max\":\"\",\"sea_level_pressure\":\"1003.5\",\"information_night\":\"時々晴\",\"timestamp\":\"6\"}},{\"_index\":\"sample_index\",\"_type\":\"sample_json\",\"_id\":\"AVjxYFq1eJQ1fjM9_cyj\",\"_score\":1.0,\"_source\":{\"precipitation_max_10min\":\"\",\"wind_speed_avg\":\"4.2\",\"snowfall\":\"\",\"precipitation_max_hour\":\"\",\"temperature_max\":\"10.2\",\"wind_direction\":\"北西\",\"information_daytime\":\"時々曇\",\"precipitation_day\":\"\",\"@timestamp\":\"2016-12-12T04:52:53.298Z\",\"humidity_avg\":\"35\",\"humidity_min\":\"23\",\"wind_speed_max\":\"7\",\"sunshine_duration\":\"3.3\",\"temperature_avg\":\"5.6\",\"temperature_min\":\"3\",\"@version\":\"1\",\"host\":\"cnXXXX1004.ecloud.nii.ac.jp\",\"atmospheric_pressure\":\"1000.5\",\"wind_direction_max_moment\":\"北西\",\"wind_speed_max_moment\":\"16.1\",\"snowfall_max\":\"\",\"sea_level_pressure\":\"1003.4\",\"information_night\":\"晴\",\"timestamp\":\"7\"}},{\"_index\":\"sample_index\",\"_type\":\"sample_json\",\"_id\":\"AVjxYFq1eJQ1fjM9_cyn\",\"_score\":1.0,\"_source\":{\"precipitation_max_10min\":\"\",\"wind_speed_avg\":\"2.7\",\"snowfall\":\"\",\"precipitation_max_hour\":\"\",\"temperature_max\":\"10.8\",\"wind_direction\":\"南東\",\"information_daytime\":\"晴\",\"precipitation_day\":\"\",\"@timestamp\":\"2016-12-12T04:52:53.306Z\",\"humidity_avg\":\"46\",\"humidity_min\":\"30\",\"wind_speed_max\":\"5.7\",\"sunshine_duration\":\"9.1\",\"temperature_avg\":\"6.4\",\"temperature_min\":\"0.2\",\"@version\":\"1\",\"host\":\"cnXXXX1004.ecloud.nii.ac.jp\",\"atmospheric_pressure\":\"1008.7\",\"wind_direction_max_moment\":\"北西\",\"wind_speed_max_moment\":\"10\",\"snowfall_max\":\"\",\"sea_level_pressure\":\"1011.7\",\"information_night\":\"一時曇\",\"timestamp\":\"11\"}},{\"_index\":\"sample_index\",\"_type\":\"sample_json\",\"_id\":\"AVjxYFq1eJQ1fjM9_cyr\",\"_score\":1.0,\"_source\":{\"precipitation_max_10min\":\"2.5\",\"wind_speed_avg\":\"3.2\",\"snowfall\":\"\",\"precipitation_max_hour\":\"11\",\"temperature_max\":\"8.2\",\"wind_direction\":\"北西\",\"information_daytime\":\"雨一時曇\",\"precipitation_day\":\"35.5\",\"@timestamp\":\"2016-12-12T04:52:53.310Z\",\"humidity_avg\":\"76\",\"humidity_min\":\"46\",\"wind_speed_max\":\"8.1\",\"sunshine_duration\":\"0\",\"temperature_avg\":\"5.4\",\"temperature_min\":\"3.4\",\"@version\":\"1\",\"host\":\"cnXXXX1004.ecloud.nii.ac.jp\",\"atmospheric_pressure\":\"1009.7\",\"wind_direction_max_moment\":\"西\",\"wind_speed_max_moment\":\"14.1\",\"snowfall_max\":\"\",\"sea_level_pressure\":\"1012.7\",\"information_night\":\"後晴\",\"timestamp\":\"15\"}},{\"_index\":\"sample_index\",\"_type\":\"sample_json\",\"_id\":\"AVjxYFq1eJQ1fjM9_cy2\",\"_score\":1.0,\"_source\":{\"precipitation_max_10min\":\"0.5\",\"wind_speed_avg\":\"2\",\"snowfall\":\"\",\"precipitation_max_hour\":\"0.5\",\"temperature_max\":\"12.9\",\"wind_direction\":\"南東\",\"information_daytime\":\"時々晴\",\"precipitation_day\":\"0.5\",\"@timestamp\":\"2016-12-12T04:52:53.325Z\",\"humidity_avg\":\"74\",\"humidity_min\":\"52\",\"wind_speed_max\":\"3.8\",\"sunshine_duration\":\"4.4\",\"temperature_avg\":\"9\",\"temperature_min\":\"5.6\",\"@version\":\"1\",\"host\":\"cnXXXX1004.ecloud.nii.ac.jp\",\"atmospheric_pressure\":\"1019.1\",\"wind_direction_max_moment\":\"南東\",\"wind_speed_max_moment\":\"6\",\"snowfall_max\":\"\",\"sea_level_pressure\":\"1022.1\",\"information_night\":\"一時曇\",\"timestamp\":\"26\"}},{\"_index\":\"sample_index\",\"_type\":\"sample_json\",\"_id\":\"AVjxYFq1eJQ1fjM9_cy3\",\"_score\":1.0,\"_source\":{\"precipitation_max_10min\":\"1.5\",\"wind_speed_avg\":\"3.1\",\"snowfall\":\"\",\"precipitation_max_hour\":\"6\",\"temperature_max\":\"16.4\",\"wind_direction\":\"北西\",\"information_daytime\":\"時々曇一時雨\",\"precipitation_day\":\"14.5\",\"@timestamp\":\"2016-12-12T04:52:53.325Z\",\"humidity_avg\":\"71\",\"humidity_min\":\"40\",\"wind_speed_max\":\"6.3\",\"sunshine_duration\":\"5.1\",\"temperature_avg\":\"9.8\",\"temperature_min\":\"5.8\",\"@version\":\"1\",\"host\":\"cnXXXX1004.ecloud.nii.ac.jp\",\"atmospheric_pressure\":\"1007\",\"wind_direction_max_moment\":\"西\",\"wind_speed_max_moment\":\"12.1\",\"snowfall_max\":\"\",\"sea_level_pressure\":\"1009.9\",\"information_night\":\"時々雨後一時みぞれ\",\"timestamp\":\"27\"}},{\"_index\":\"sample_index\",\"_type\":\"sample_json\",\"_id\":\"AVjxYFq1eJQ1fjM9_cy4\",\"_score\":1.0,\"_source\":{\"precipitation_max_10min\":\"0\",\"wind_speed_avg\":\"4\",\"snowfall\":\"\",\"precipitation_max_hour\":\"0\",\"temperature_max\":\"9.7\",\"wind_direction\":\"北西\",\"information_daytime\":\"時々曇一時雪\",\"precipitation_day\":\"0\",\"@timestamp\":\"2016-12-12T04:52:53.326Z\",\"humidity_avg\":\"41\",\"humidity_min\":\"20\",\"wind_speed_max\":\"7.6\",\"sunshine_duration\":\"6.2\",\"temperature_avg\":\"5.2\",\"temperature_min\":\"1.4\",\"@version\":\"1\",\"host\":\"cnXXXX1004.ecloud.nii.ac.jp\",\"atmospheric_pressure\":\"1014.7\",\"wind_direction_max_moment\":\"北西\",\"wind_speed_max_moment\":\"14.6\",\"snowfall_max\":\"\",\"sea_level_pressure\":\"1017.7\",\"information_night\":\"晴\",\"timestamp\":\"28\"}},{\"_index\":\"sample_index\",\"_type\":\"sample_multiline_json\",\"_id\":\"AVjxZOVqeJQ1fjM9_cy7\",\"_score\":1.0,\"_source\":{\"wind_speed_avg\":\"4.4\",\"snowfall\":\"\",\"temperature_max\":\"8.2\",\"precipitation_day\":\"0\",\"wind_speed_max\":\"9.4\",\"sunshine_duration\":\"1.3\",\"temperature_avg\":\"3.7\",\"temperature_min\":\"0.7\",\"@version\":\"1\",\"host\":\"cnXXXX1004.ecloud.nii.ac.jp\",\"wind_direction_max_moment\":\"北西\",\"information_night\":\"晴\",\"timestamp\":\"1\",\"precipitation_max_10min\":\"0\",\"precipitation_max_hour\":\"0\",\"wind_direction\":\"西\",\"message\":\"{\\r\\n\\\"timestamp\\\":\\\"1\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1000.4\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1003.4\\\",\\r\\n\\\"precipitation_day\\\":\\\"0\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"0\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"0\\\",\\r\\n\\\"temperature_avg\\\":\\\"3.7\\\",\\r\\n\\\"temperature_max\\\":\\\"8.2\\\",\\r\\n\\\"temperature_min\\\":\\\"0.7\\\",\\r\\n\\\"humidity_avg\\\":\\\"41\\\",\\r\\n\\\"humidity_min\\\":\\\"30\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"4.4\\\",\\r\\n\\\"wind_speed_max\\\":\\\"9.4\\\",\\r\\n\\\"wind_direction\\\":\\\"西\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"18.7\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"北西\\\",\\r\\n\\\"sunshine_duration\\\":\\\"1.3\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"時々晴一時雪\\\",\\r\\n\\\"information_night\\\":\\\"晴\\\"\\r\\n}\\r\",\"information_daytime\":\"時々晴一時雪\",\"tags\":[\"multiline\"],\"@timestamp\":\"2016-12-12T04:57:50.634Z\",\"humidity_avg\":\"41\",\"humidity_min\":\"30\",\"atmospheric_pressure\":\"1000.4\",\"wind_speed_max_moment\":\"18.7\",\"snowfall_max\":\"\",\"sea_level_pressure\":\"1003.4\"}},{\"_index\":\"sample_index\",\"_type\":\"sample_multiline_json\",\"_id\":\"AVjxZOVqeJQ1fjM9_czE\",\"_score\":1.0,\"_source\":{\"wind_speed_avg\":\"3.5\",\"snowfall\":\"\",\"temperature_max\":\"11.2\",\"precipitation_day\":\"\",\"wind_speed_max\":\"8.8\",\"sunshine_duration\":\"9.1\",\"temperature_avg\":\"6.3\",\"temperature_min\":\"1.9\",\"@version\":\"1\",\"host\":\"cnXXXX1004.ecloud.nii.ac.jp\",\"wind_direction_max_moment\":\"西\",\"information_night\":\"晴\",\"timestamp\":\"10\",\"precipitation_max_10min\":\"\",\"precipitation_max_hour\":\"\",\"wind_direction\":\"西\",\"message\":\"{\\r\\n\\\"timestamp\\\":\\\"10\\\",\\r\\n\\\"atmospheric_pressure\\\":\\\"1008.8\\\",\\r\\n\\\"sea_level_pressure\\\":\\\"1011.8\\\",\\r\\n\\\"precipitation_day\\\":\\\"\\\",\\r\\n\\\"precipitation_max_hour\\\":\\\"\\\",\\r\\n\\\"precipitation_max_10min\\\":\\\"\\\",\\r\\n\\\"temperature_avg\\\":\\\"6.3\\\",\\r\\n\\\"temperature_max\\\":\\\"11.2\\\",\\r\\n\\\"temperature_min\\\":\\\"1.9\\\",\\r\\n\\\"humidity_avg\\\":\\\"31\\\",\\r\\n\\\"humidity_min\\\":\\\"16\\\",\\r\\n\\\"wind_speed_avg\\\":\\\"3.5\\\",\\r\\n\\\"wind_speed_max\\\":\\\"8.8\\\",\\r\\n\\\"wind_direction\\\":\\\"西\\\",\\r\\n\\\"wind_speed_max_moment\\\":\\\"17.7\\\",\\r\\n\\\"wind_direction_max_moment\\\":\\\"西\\\",\\r\\n\\\"sunshine_duration\\\":\\\"9.1\\\",\\r\\n\\\"snowfall\\\":\\\"\\\",\\r\\n\\\"snowfall_max\\\":\\\"\\\",\\r\\n\\\"information_daytime\\\":\\\"晴\\\",\\r\\n\\\"information_night\\\":\\\"晴\\\"\\r\\n}\\r\",\"information_daytime\":\"晴\",\"tags\":[\"multiline\"],\"@timestamp\":\"2016-12-12T04:57:51.008Z\",\"humidity_avg\":\"31\",\"humidity_min\":\"16\",\"atmospheric_pressure\":\"1008.8\",\"wind_speed_max_moment\":\"17.7\",\"snowfall_max\":\"\",\"sea_level_pressure\":\"1011.8\"}}]}}"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100  8920  100  8920    0     0   571k      0 --:--:-- --:--:-- --:--:--  580k\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/ltsv_$INDEX/_search\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915491f6-152a-11e7-b31a-0242ac12000d",
     "next": "915493c2-152a-11e7-b31a-0242ac12000d",
     "previous": "9154902a-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### 任意のテキスト(1行完結)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915493c2-152a-11e7-b31a-0242ac12000d",
     "next": "9154958e-152a-11e7-b31a-0242ac12000d",
     "previous": "915491f6-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "ここまでの説明のように、JSONやCSVなどの定型のデータであれば設定は簡単ですが、実際に読み込みたいデータはそのように形式がそろっていないことも多くあります。\n",
    "\n",
    "次のサンプルデータのように、「名前=値」という形式がスペース区切りで1行に並んでいるようなデータを登録することを考えます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154958e-152a-11e7-b31a-0242ac12000d",
     "next": "91549796-152a-11e7-b31a-0242ac12000d",
     "previous": "915493c2-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "timestamp=2015-01-01T00:00:00 location=tokyo atmospheric_pressure=1000.4 sea_level_pressure=1003.4 precipitation_day=0 precipitation_max_hour=0 precipitation_max_10min=0 temperature_avg=3.7 temperature_max=8.2 temperature_min=0.7 humidity_avg=41.0 humidity_min=30.0 wind_speed_avg=4.4 wind_speed_max=9.4 wind_direction=西 wind_speed_max_moment=18.7 wind_direction_max_moment=北西 sunshine_duration=1.3 information_daytime=時々晴一時雪 information_night=晴"
     ]
    }
   ],
   "source": [
    "!head sample_data/single_line.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91549796-152a-11e7-b31a-0242ac12000d",
     "next": "91549962-152a-11e7-b31a-0242ac12000d",
     "previous": "9154958e-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "この形式のデータを読み込むためのconfファイルは、次のような内容になります。  \n",
    "ファイル名は\"single_line.conf\"とします。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91549962-152a-11e7-b31a-0242ac12000d",
     "next": "91549b2e-152a-11e7-b31a-0242ac12000d",
     "previous": "91549796-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "```ruby:single_line.conf\n",
    "input{\n",
    "  stdin{}\n",
    "}\n",
    "\n",
    "filter{\n",
    "  grok{\n",
    "    match => {\"message\" => \"timestamp=%{TIMESTAMP_ISO8601:timestamp}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"location=%{WORD:location}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"atmospheric_pressure=%{WORD:atmospheric_pressure}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"sea_level_pressure=%{WORD:sea_level_pressure}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"precipitation_day=%{WORD:precipitation_day}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"precipitation_max_hour=%{WORD:precipitation_max_hour}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"precipitation_max_10min=%{WORD:precipitation_max_10min}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"temperature_avg=%{WORD:temperature_avg}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"temperature_max=%{WORD:temperature_max}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"temperature_min=%{WORD:temperature_min}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"humidity_avg=%{WORD:humidity_avg}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"humidity_min=%{WORD:humidity_min}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"wind_speed_avg=%{WORD:wind_speed_avg}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"wind_speed_max=%{WORD:wind_speed_max}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"wind_direction=%{WORD:wind_direction}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"wind_speed_max_moment=%{WORD:wind_speed_max_moment}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"wind_direction_max_moment=%{WORD:wind_direction_max_moment}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"sunshine_duration=%{WORD:sunshine_duration}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"snowfall=%{WORD:snowfall}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"snowfall_max=%{WORD:snowfall_max}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"information_daytime=%{WORD:information_daytime}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"information_night=%{WORD:information_night}\"}\n",
    "  }\n",
    "  date{\n",
    "    match => [\"timestamp\",\"ISO8601\"]\n",
    "    timezone => \"UTC\"\n",
    "    remove_field => \"timestamp\"\n",
    "  }\n",
    "}\n",
    "\n",
    "output{\n",
    "  elasticsearch{\n",
    "    hosts => <ES_HOST>\n",
    "    index => <INDEX>\n",
    "  }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91549b2e-152a-11e7-b31a-0242ac12000d",
     "next": "91549cfa-152a-11e7-b31a-0242ac12000d",
     "previous": "91549962-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "この設定では、2種類の加工処理を行っています。  \n",
    "入力の特定部分をパースするための「grok filter」と  \n",
    "文字列をタイムスタンプとして認識させるための「date filter」を用いています。  \n",
    "加工処理の詳細に関しては、Logstashの[公式ページ](https://www.elastic.co/guide/en/logstash/current/filter-plugins.html)を参照してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "91549cfa-152a-11e7-b31a-0242ac12000d",
     "next": "91549ed0-152a-11e7-b31a-0242ac12000d",
     "previous": "91549b2e-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "実際のデータ登録は次のコマンドで実行します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "lc_cell_meme": {
     "current": "91549ed0-152a-11e7-b31a-0242ac12000d",
     "next": "9154a09c-152a-11e7-b31a-0242ac12000d",
     "previous": "91549cfa-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\r\n",
      "Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties.\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible logstash-server -i ./hosts -m shell -a \"cat single_line.txt | sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/single_line.conf\" -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154a09c-152a-11e7-b31a-0242ac12000d",
     "next": "9154a268-152a-11e7-b31a-0242ac12000d",
     "previous": "91549ed0-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "コマンドの実行に成功したら、次の検索APIで正しく登録されているか確認してください。  \n",
    "正しく登録されていれば、json形式で登録されたデータが出力されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154a268-152a-11e7-b31a-0242ac12000d",
     "next": "9154a434-152a-11e7-b31a-0242ac12000d",
     "previous": "9154a09c-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"took\" : 6,\n",
      "  \"timed_out\" : false,\n",
      "  \"_shards\" : {\n",
      "    \"total\" : 5,\n",
      "    \"successful\" : 5,\n",
      "    \"failed\" : 0\n",
      "  },\n",
      "  \"hits\" : {\n",
      "    \"total\" : 0,\n",
      "    \"max_score\" : null,\n",
      "    \"hits\" : [ ]\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   193  100   193    0     0  17948      0 --:--:-- --:--:-- --:--:-- 19300\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/single_line_$INDEX/_search?pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154a434-152a-11e7-b31a-0242ac12000d",
     "next": "9154a63c-152a-11e7-b31a-0242ac12000d",
     "previous": "9154a268-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### 任意のテキスト(複数行完結)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154a63c-152a-11e7-b31a-0242ac12000d",
     "next": "9154a808-152a-11e7-b31a-0242ac12000d",
     "previous": "9154a434-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "前の節の形式に加え、データが複数行にまたがっているような場合であっても、Logstashは対応が可能です。  \n",
    "次のサンプルのようなデータを登録してみましょう。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154a808-152a-11e7-b31a-0242ac12000d",
     "next": "9154a9d4-152a-11e7-b31a-0242ac12000d",
     "previous": "9154a63c-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "timestamp=2015-01-01T00:00:00 location=tokyo atmospheric_pressure=1000.4\r",
      "\r\n",
      "sea_level_pressure=1003.4 precipitation_day=0 precipitation_max_hour=0\r",
      "\r\n",
      "precipitation_max_10min=0 temperature_avg=3.7 temperature_max=8.2\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 3 sample_data/multi_line.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154a9d4-152a-11e7-b31a-0242ac12000d",
     "next": "9154aba0-152a-11e7-b31a-0242ac12000d",
     "previous": "9154a808-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "この形式のデータを読み込むためのconfファイルは、次のような内容になります。  \n",
    "ファイル名は\"multi_line.conf\"とします。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154aba0-152a-11e7-b31a-0242ac12000d",
     "next": "9154ad80-152a-11e7-b31a-0242ac12000d",
     "previous": "9154a9d4-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "```ruby:multi_line.conf\n",
    "input{\n",
    "  stdin{\n",
    "    codec => multiline{\n",
    "        pattern => \"information_night\"\n",
    "        negate => \"true\"\n",
    "        what => \"next\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "\n",
    "filter{\n",
    "  grok{\n",
    "    match => {\"message\" => \"timestamp=%{TIMESTAMP_ISO8601:timestamp}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"location=%{WORD:location}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"atmospheric_pressure=%{WORD:atmospheric_pressure}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"sea_level_pressure=%{WORD:sea_level_pressure}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"precipitation_day=%{WORD:precipitation_day}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"precipitation_max_hour=%{WORD:precipitation_max_hour}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"precipitation_max_10min=%{WORD:precipitation_max_10min}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"temperature_avg=%{WORD:temperature_avg}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"temperature_max=%{WORD:temperature_max}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"temperature_min=%{WORD:temperature_min}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"humidity_avg=%{WORD:humidity_avg}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"humidity_min=%{WORD:humidity_min}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"wind_speed_avg=%{WORD:wind_speed_avg}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"wind_speed_max=%{WORD:wind_speed_max}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"wind_direction=%{WORD:wind_direction}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"wind_speed_max_moment=%{WORD:wind_speed_max_moment}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"wind_direction_max_moment=%{WORD:wind_direction_max_moment}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"sunshine_duration=%{WORD:sunshine_duration}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"snowfall=%{WORD:snowfall}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"snowfall_max=%{WORD:snowfall_max}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"information_daytime=%{WORD:information_daytime}\"}\n",
    "  }\n",
    "  grok{\n",
    "    match => {\"message\" => \"information_night=%{WORD:information_night}\"}\n",
    "  }\n",
    "  date{\n",
    "    match => [\"timestamp\",\"ISO8601\"]\n",
    "    timezone => \"UTC\"\n",
    "    remove_field => \"timestamp\"\n",
    "  }\n",
    "}\n",
    "\n",
    "output{\n",
    "  elasticsearch{\n",
    "    hosts => \"XXX.XXX.XXX.1:9200\"\n",
    "    index => \"sample_index\"\n",
    "  }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154ad80-152a-11e7-b31a-0242ac12000d",
     "next": "9154af56-152a-11e7-b31a-0242ac12000d",
     "previous": "9154aba0-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "1行で完結するデータを読み込むときとの差分はcodecでmultilineの設定をしているところです。  \n",
    "Logstashでは通常、入力を１行単位で読み込んで処理します。  \n",
    "multilineの指定をすることで、複数行を一つのイベントとして連結して処理することができます。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154af56-152a-11e7-b31a-0242ac12000d",
     "next": "9154b122-152a-11e7-b31a-0242ac12000d",
     "previous": "9154ad80-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "実際のデータ登録は次のコマンドで実行します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154b122-152a-11e7-b31a-0242ac12000d",
     "next": "9154b2f8-152a-11e7-b31a-0242ac12000d",
     "previous": "9154af56-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\r\n",
      "Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties.\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible logstash-server -i ./hosts -m shell -a \"cat multi_line.txt | sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/multi_line.conf\" -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154b2f8-152a-11e7-b31a-0242ac12000d",
     "next": "9154b4ce-152a-11e7-b31a-0242ac12000d",
     "previous": "9154b122-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "コマンドの実行に成功したら、次の検索APIで正しく登録されているか確認してください。  \n",
    "正しく登録されていれば、json形式で登録されたデータが出力されます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154b4ce-152a-11e7-b31a-0242ac12000d",
     "next": "9154b690-152a-11e7-b31a-0242ac12000d",
     "previous": "9154b2f8-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"took\" : 12,\n",
      "  \"timed_out\" : false,\n",
      "  \"_shards\" : {\n",
      "    \"total\" : 5,\n",
      "    \"successful\" : 5,\n",
      "    \"failed\" : 0\n",
      "  },\n",
      "  \"hits\" : {\n",
      "    \"total\" : 1,\n",
      "    \"max_score\" : 1.0,\n",
      "    \"hits\" : [\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_multi_line\",\n",
      "        \"_id\" : \"AVjxaDXqeJQ1fjM9_cz5\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"precipitation_max_10min\" : \"0\",\n",
      "          \"wind_speed_avg\" : \"4\",\n",
      "          \"precipitation_max_hour\" : \"0\",\n",
      "          \"temperature_max\" : \"8\",\n",
      "          \"message\" : \"timestamp=2015-01-01T00:00:00 location=tokyo atmospheric_pressure=1000.4\\\\r\\\\nsea_level_pressure=1003.4 precipitation_day=0 precipitation_max_hour=0\\\\r\\\\nprecipitation_max_10min=0 temperature_avg=3.7 temperature_max=8.2\\\\r\\\\ntemperature_min=0.7 humidity_avg=41.0 humidity_min=30.0\\\\r\\\\nwind_speed_avg=4.4 wind_speed_max=9.4 wind_direction=\\\\x90\\\\xBC\\\\r\\\\nwind_speed_max_moment=18.7 wind_direction_max_moment=\\\\x96k\\\\x90\\\\xBC\\\\r\\\\nsunshine_duration=1.3 information_daytime=\\\\x8E\\\\x9E\\\\x81X\\\\x90\\\\xB0\\\\x88ꎞ\\\\x90\\\\xE1\\\\r\\\\ninformation_night=\\\\x90\\\\xB0\\\\r\\\\ntimestamp=2015-01-02T00:00:00 location=tokyo atmospheric_pressure=1007.4\\\\r\\\\nsea_level_pressure=1010.4 temperature_avg=2.7 temperature_max=7.9\\\\r\\\\ntemperature_min=-2.2 humidity_avg=41.0 humidity_min=19.0\\\\r\\\\nwind_speed_avg=2.1 wind_speed_max=4.2 wind_direction=\\\\x90\\\\xBC\\\\r\\\\nwind_speed_max_moment=7.8 wind_direction_max_moment=\\\\x90\\\\xBC sunshine_duration=6.0 information_daytime=\\\\x8E\\\\x9E\\\\x81X\\\\x90\\\\xB0\\\\x88ꎞ\\\\x90\\\\xE1 information_night=\\\\x90\\\\xB0\",\n",
      "          \"precipitation_day\" : \"0\",\n",
      "          \"tags\" : [\n",
      "            \"_grokparsefailure\"\n",
      "          ],\n",
      "          \"@timestamp\" : \"2015-01-01T00:00:00.000Z\",\n",
      "          \"humidity_avg\" : \"41\",\n",
      "          \"humidity_min\" : \"30\",\n",
      "          \"wind_speed_max\" : \"9\",\n",
      "          \"sunshine_duration\" : \"1\",\n",
      "          \"temperature_avg\" : \"3\",\n",
      "          \"temperature_min\" : \"0\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"host\" : \"cnXXXX1004.ecloud.nii.ac.jp\",\n",
      "          \"atmospheric_pressure\" : \"1000\",\n",
      "          \"location\" : \"tokyo\",\n",
      "          \"wind_speed_max_moment\" : \"18\",\n",
      "          \"sea_level_pressure\" : \"1003\"\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100  2175  100  2175    0     0   114k      0 --:--:-- --:--:-- --:--:--  118k\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/multi_line_$INDEX/_search?pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154b690-152a-11e7-b31a-0242ac12000d",
     "next": "9154b85c-152a-11e7-b31a-0242ac12000d",
     "previous": "9154b4ce-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## データ加工\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154b85c-152a-11e7-b31a-0242ac12000d",
     "next": "9154ba28-152a-11e7-b31a-0242ac12000d",
     "previous": "9154b690-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "ここまでの例では、読み込んだ値を特に加工せず、そのまま格納していました。  \n",
    "この章では、confファイルの\"filter\"を定義することで値を加工する方法について説明します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154ba28-152a-11e7-b31a-0242ac12000d",
     "next": "9154bbfe-152a-11e7-b31a-0242ac12000d",
     "previous": "9154b85c-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### geoipの付与"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154bbfe-152a-11e7-b31a-0242ac12000d",
     "next": "9154bdca-152a-11e7-b31a-0242ac12000d",
     "previous": "9154ba28-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "geoip filterを用いることで、ipアドレスをもとに地理情報を付与することができます。  \n",
    "利用するサンプルデータは次のコマンドを実行することで確認できます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154bdca-152a-11e7-b31a-0242ac12000d",
     "next": "9154bf96-152a-11e7-b31a-0242ac12000d",
     "previous": "9154bbfe-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r",
      "\r\n",
      "  \"recordtime\":\"2016-12-02\",\r",
      "\r\n",
      "  \"clientip\":\"XXX.XXX.XXX.134\"\r",
      "\r\n",
      "}\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 4 sample_data/geoip.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154bf96-152a-11e7-b31a-0242ac12000d",
     "next": "9154c162-152a-11e7-b31a-0242ac12000d",
     "previous": "9154bdca-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "具体的な設定を次に示します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154c162-152a-11e7-b31a-0242ac12000d",
     "next": "9154c32e-152a-11e7-b31a-0242ac12000d",
     "previous": "9154bf96-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input{\r",
      "\r\n",
      "  stdin{\r",
      "\r\n",
      "    codec => multiline{\r",
      "\r\n",
      "      pattern => \"}\"\r",
      "\r\n",
      "      negate => \"true\"\r",
      "\r\n",
      "      what => \"next\"\r",
      "\r\n",
      "    }\r",
      "\r\n",
      "  }\r",
      "\r\n",
      "}\r",
      "\r\n",
      "\r",
      "\r\n",
      "filter{\r",
      "\r\n",
      "  json{\r",
      "\r\n",
      "    source => \"message\"\r",
      "\r\n",
      "  }\r",
      "\r\n",
      "  date{\r",
      "\r\n",
      "    match => [\"recordtime\",\"yyyy-MM-dd\"]\r",
      "\r\n",
      "    remove_field => [\"recordtime\",\"message\",\"host\"]\r",
      "\r\n",
      "  }\r",
      "\r\n",
      "  geoip{\r",
      "\r\n",
      "    source => \"clientip\"\r",
      "\r\n",
      "  }\r",
      "\r\n",
      "}\r",
      "\r\n",
      "\r",
      "\r\n",
      "output{\r",
      "\r\n",
      "  elasticsearch{\r",
      "\r\n",
      "    hosts => \"XXX.XXX.XXX.232\"\r",
      "\r\n",
      "    index => \"sample_index\"\r",
      "\r\n",
      "    document_type => \"sample_geoip\"\r",
      "\r\n",
      "  }\r",
      "\r\n",
      "}"
     ]
    }
   ],
   "source": [
    "!cat logstash_conf/geoip.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154c32e-152a-11e7-b31a-0242ac12000d",
     "next": "9154c504-152a-11e7-b31a-0242ac12000d",
     "previous": "9154c162-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "geoip{}の中のsourceという項目に、地理情報を取得したいipアドレスを指定します。  \n",
    "\n",
    "実際のデータ登録は次のセルのコマンドを実行してください。  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154c504-152a-11e7-b31a-0242ac12000d",
     "next": "9154c6ee-152a-11e7-b31a-0242ac12000d",
     "previous": "9154c32e-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\r\n",
      "Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties.\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible logstash-server -i ./hosts -m shell -a \"cat geoip.json | sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/geoip.conf\" -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154c6ee-152a-11e7-b31a-0242ac12000d",
     "next": "9154c8ba-152a-11e7-b31a-0242ac12000d",
     "previous": "9154c504-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "コマンドの実行に成功したら、次の検索APIで正しく登録されているか確認してください。  \n",
    "もとのデータに含まれている**recordtime**,**clientip**以外に国名や緯度・経度などが追加されています。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154c8ba-152a-11e7-b31a-0242ac12000d",
     "next": "9154ca90-152a-11e7-b31a-0242ac12000d",
     "previous": "9154c6ee-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"took\" : 6,\n",
      "  \"timed_out\" : false,\n",
      "  \"_shards\" : {\n",
      "    \"total\" : 5,\n",
      "    \"successful\" : 5,\n",
      "    \"failed\" : 0\n",
      "  },\n",
      "  \"hits\" : {\n",
      "    \"total\" : 13,\n",
      "    \"max_score\" : 1.0,\n",
      "    \"hits\" : [\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_geoip\",\n",
      "        \"_id\" : \"AVjxaOLceJQ1fjM9_c0G\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"@timestamp\" : \"2016-12-03T15:00:00.000Z\",\n",
      "          \"geoip\" : {\n",
      "            \"timezone\" : \"America/New_York\",\n",
      "            \"ip\" : \"134.65.66.21\",\n",
      "            \"latitude\" : 40.9777,\n",
      "            \"continent_code\" : \"NA\",\n",
      "            \"city_name\" : \"Rye\",\n",
      "            \"country_code2\" : \"US\",\n",
      "            \"country_name\" : \"United States\",\n",
      "            \"dma_code\" : 501,\n",
      "            \"country_code3\" : \"US\",\n",
      "            \"region_name\" : \"New York\",\n",
      "            \"location\" : [\n",
      "              -73.6935,\n",
      "              40.9777\n",
      "            ],\n",
      "            \"postal_code\" : \"10580\",\n",
      "            \"longitude\" : -73.6935,\n",
      "            \"region_code\" : \"NY\"\n",
      "          },\n",
      "          \"clientip\" : \"134.65.66.21\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"tags\" : [\n",
      "            \"multiline\"\n",
      "          ]\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100  1247  100  1156  100    91   100k   8112 --:--:-- --:--:-- --:--:--  102k\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/geoip_$INDEX/_search?pretty\" -H 'Content-Type:application/json' -d @- <<EOF\n",
    "{\n",
    "    \"size\":\"1\"\n",
    "}\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154ca90-152a-11e7-b31a-0242ac12000d",
     "next": "9154cc52-152a-11e7-b31a-0242ac12000d",
     "previous": "9154c8ba-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### useragentの解析"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154cc52-152a-11e7-b31a-0242ac12000d",
     "next": "9154ce32-152a-11e7-b31a-0242ac12000d",
     "previous": "9154ca90-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "useragentの情報を解析するためには、useragent filterを利用します。  \n",
    "利用するサンプルデータは次のコマンドを実行することで確認できます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154ce32-152a-11e7-b31a-0242ac12000d",
     "next": "9154cffe-152a-11e7-b31a-0242ac12000d",
     "previous": "9154cc52-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8) AppleWebKit/536.25 (KHTML, like Gecko) Version/6.0 Safari/536.25\r\n",
      "\r\n",
      "Mozilla/4.0 (compatible; MSIE 6.0; Nitro) Opera 8.50 [ja]\r",
      "\r\n",
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:9.0.1) Gecko/20100101 Firefox/9.0.1\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 3 sample_data/useragent.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154cffe-152a-11e7-b31a-0242ac12000d",
     "next": "9154d1e8-152a-11e7-b31a-0242ac12000d",
     "previous": "9154ce32-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "具体的な設定を次に示します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154d1e8-152a-11e7-b31a-0242ac12000d",
     "next": "9154d3be-152a-11e7-b31a-0242ac12000d",
     "previous": "9154cffe-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input{\r",
      "\r\n",
      "  stdin{}\r",
      "\r\n",
      "}\r",
      "\r\n",
      "filter{\r",
      "\r\n",
      "  useragent{\r",
      "\r\n",
      "    source => \"message\"\r",
      "\r\n",
      "    remove_field => [\"message\",\"host\"]\r",
      "\r\n",
      "  }\r",
      "\r\n",
      "}\r",
      "\r\n",
      "output{\r",
      "\r\n",
      "  elasticsearch{\r",
      "\r\n",
      "    hosts => \"XXX.XXX.XXX.232\"\r",
      "\r\n",
      "    index => \"sample_index\"\r",
      "\r\n",
      "    document_type => \"sample_useragent\"\r",
      "\r\n",
      "  }\r",
      "\r\n",
      "}"
     ]
    }
   ],
   "source": [
    "!cat logstash_conf/useragent.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154d3be-152a-11e7-b31a-0242ac12000d",
     "next": "9154d580-152a-11e7-b31a-0242ac12000d",
     "previous": "9154d1e8-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "useragent filterを利用すると、自動的にuseragentの情報をパースすることができます。  \n",
    "\n",
    "実際のデータ登録は次のセルのコマンドを実行してください。  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154d580-152a-11e7-b31a-0242ac12000d",
     "next": "9154d760-152a-11e7-b31a-0242ac12000d",
     "previous": "9154d3be-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32mXXX.XXX.XXX.232 | SUCCESS | rc=0 >>\r\n",
      "Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties.\r\n",
      "\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!ansible logstash-server -i ./hosts -m shell -a \"cat useragent.txt | sudo /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/useragent.conf\" -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154d760-152a-11e7-b31a-0242ac12000d",
     "next": "9154d92c-152a-11e7-b31a-0242ac12000d",
     "previous": "9154d580-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "コマンドの実行に成功したら、次の検索APIで正しく登録されているか確認してください。  \n",
    "もとのデータに含まれているrecordtime,clientip以外に国名や緯度・経度などが追加されています。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154d92c-152a-11e7-b31a-0242ac12000d",
     "next": "9154db02-152a-11e7-b31a-0242ac12000d",
     "previous": "9154d760-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"took\" : 6,\n",
      "  \"timed_out\" : false,\n",
      "  \"_shards\" : {\n",
      "    \"total\" : 5,\n",
      "    \"successful\" : 5,\n",
      "    \"failed\" : 0\n",
      "  },\n",
      "  \"hits\" : {\n",
      "    \"total\" : 4,\n",
      "    \"max_score\" : 1.0,\n",
      "    \"hits\" : [\n",
      "      {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"sample_useragent\",\n",
      "        \"_id\" : \"AVjxa3nceJQ1fjM9_c0H\",\n",
      "        \"_score\" : 1.0,\n",
      "        \"_source\" : {\n",
      "          \"@timestamp\" : \"2016-12-12T05:05:01.656Z\",\n",
      "          \"os\" : \"Mac OS X 10.8\",\n",
      "          \"major\" : \"6\",\n",
      "          \"minor\" : \"0\",\n",
      "          \"os_minor\" : \"8\",\n",
      "          \"os_major\" : \"10\",\n",
      "          \"@version\" : \"1\",\n",
      "          \"name\" : \"Safari\",\n",
      "          \"os_name\" : \"Mac OS X\",\n",
      "          \"device\" : \"Other\"\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "bash: line 7: warning: here-document at line 1 delimited by end-of-file (wanted `EOF')\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   790  100   695  100    95  57457   7853 --:--:-- --:--:-- --:--:-- 63181\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/useragent_$INDEX/_search?pretty\" -H 'Content-Type:application/json' -d @- <<EOF\n",
    "{\n",
    "    \"size\":\"1\"\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154db02-152a-11e7-b31a-0242ac12000d",
     "next": "9154dcce-152a-11e7-b31a-0242ac12000d",
     "previous": "9154d92c-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## Ingest Nodeの利用方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154dcce-152a-11e7-b31a-0242ac12000d",
     "next": "9154df12-152a-11e7-b31a-0242ac12000d",
     "previous": "9154db02-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Ingest Nodeを利用するためには、まずpipelineを作成する必要があります。  \n",
    "\n",
    "![images/03_pipeline.png](images/03_pipeline.png)\n",
    "\n",
    "\n",
    "Node種類の説明で記述したように、pipelineはIngest Nodeが受け取ったドキュメントに対して、いくつかの加工処理(processor)を順次実行するものとなります。  \n",
    "\n",
    "pipelineの作成には、Put Pipeline APIを用います。pipelineには次の二つの要素を定義します。  \n",
    "* description : 処理内容の説明\n",
    "* processors : 実際の処理\n",
    "\n",
    "ここでは受け取ったドキュメントに対して新しいフィールドを追加するためのpipelineを作成します。  \n",
    "ここで扱う例では、\"country\"というフィールドを追加し、値を\"japan\"に設定しています。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154df12-152a-11e7-b31a-0242ac12000d",
     "next": "9154e0f2-152a-11e7-b31a-0242ac12000d",
     "previous": "9154dcce-152a-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"acknowledged\":true}"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   158  100    21  100   137    419   2738 --:--:-- --:--:-- --:--:--  2795\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XPUT \"http://$ES_HOST/_ingest/pipeline/my-pipeline\" -H 'Content-Type:application/json' -d @- << EOF\n",
    "{\n",
    "  \"description\":\"describe pipeline\",\n",
    "  \"processors\":[\n",
    "    {\n",
    "      \"set\":{\n",
    "        \"field\":\"country\",\n",
    "        \"value\": \"japan\"\n",
    "      }\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154e0f2-152a-11e7-b31a-0242ac12000d",
     "next": "9154e2d2-152a-11e7-b31a-0242ac12000d",
     "previous": "9154df12-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "作成済みpipelineの確認にはGet Ppeline APIを用います。次のコマンドで作成済みpipelineの確認が可能です。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154e2d2-152a-11e7-b31a-0242ac12000d",
     "next": "9154e4a8-152a-11e7-b31a-0242ac12000d",
     "previous": "9154e0f2-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"my-pipeline\" : {\n",
      "    \"description\" : \"describe pipeline\",\n",
      "    \"processors\" : [\n",
      "      {\n",
      "        \"set\" : {\n",
      "          \"field\" : \"country\",\n",
      "          \"value\" : \"japan\"\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   199  100   199    0     0  35183      0 --:--:-- --:--:-- --:--:-- 39800\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/_ingest/pipeline?pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154e4a8-152a-11e7-b31a-0242ac12000d",
     "next": "9154e66a-152a-11e7-b31a-0242ac12000d",
     "previous": "9154e2d2-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "pipelineの動作をシミュレートするためには、Simulate Pipeline APIを用います。  \n",
    "pipelineの定義と、ドキュメントの内容を記述することで、実際にpipeline処理が行われた時の出力結果を確認することができます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "lc_cell_meme": {
     "current": "9154e66a-152a-11e7-b31a-0242ac12000d",
     "next": "9154e840-152a-11e7-b31a-0242ac12000d",
     "previous": "9154e4a8-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"docs\" : [\n",
      "    {\n",
      "      \"doc\" : {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"logs\",\n",
      "        \"_id\" : \"id\",\n",
      "        \"_source\" : {\n",
      "          \"country\" : \"japan\",\n",
      "          \"wind_speed_avg\" : \"4.4\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"8.2\",\n",
      "          \"precipitation_day\" : \"0\",\n",
      "          \"wind_speed_max\" : \"9.4\",\n",
      "          \"sunshine_duration\" : \"1.3\",\n",
      "          \"temperature_avg\" : \"3.7\",\n",
      "          \"temperature_min\" : \"0.7\",\n",
      "          \"wind_direction_max_moment\" : \"北西\",\n",
      "          \"information_night\" : \"晴\",\n",
      "          \"timestamp\" : \"1\",\n",
      "          \"precipitation_max_10min\" : \"0\",\n",
      "          \"precipitation_max_hour\" : \"0\",\n",
      "          \"wind_direction\" : \"西\",\n",
      "          \"information_daytime\" : \"時々晴一時雪\",\n",
      "          \"humidity_avg\" : \"41\",\n",
      "          \"humidity_min\" : \"30\",\n",
      "          \"atmospheric_pressure\" : \"1000.4\",\n",
      "          \"wind_speed_max_moment\" : \"18.7\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1003.4\"\n",
      "        },\n",
      "        \"_ingest\" : {\n",
      "          \"timestamp\" : \"2018-02-15T03:51:26.510Z\"\n",
      "        }\n",
      "      }\n",
      "    },\n",
      "    {\n",
      "      \"doc\" : {\n",
      "        \"_index\" : \"sample_index\",\n",
      "        \"_type\" : \"_type\",\n",
      "        \"_id\" : \"id\",\n",
      "        \"_source\" : {\n",
      "          \"country\" : \"japan\",\n",
      "          \"wind_speed_avg\" : \"2.1\",\n",
      "          \"snowfall\" : \"\",\n",
      "          \"temperature_max\" : \"7.9\",\n",
      "          \"precipitation_day\" : \"\",\n",
      "          \"wind_speed_max\" : \"4.2\",\n",
      "          \"sunshine_duration\" : \"6\",\n",
      "          \"temperature_avg\" : \"2.7\",\n",
      "          \"temperature_min\" : \"-2.2\",\n",
      "          \"wind_direction_max_moment\" : \"西\",\n",
      "          \"information_night\" : \"一時曇\",\n",
      "          \"timestamp\" : \"2\",\n",
      "          \"precipitation_max_10min\" : \"\",\n",
      "          \"precipitation_max_hour\" : \"\",\n",
      "          \"wind_direction\" : \"西\",\n",
      "          \"information_daytime\" : \"時々曇\",\n",
      "          \"humidity_avg\" : \"41\",\n",
      "          \"humidity_min\" : \"19\",\n",
      "          \"atmospheric_pressure\" : \"1007.4\",\n",
      "          \"wind_speed_max_moment\" : \"7.8\",\n",
      "          \"snowfall_max\" : \"\",\n",
      "          \"sea_level_pressure\" : \"1010.4\"\n",
      "        },\n",
      "        \"_ingest\" : {\n",
      "          \"timestamp\" : \"2018-02-15T03:51:26.510Z\"\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100  4371  100  2163  100  2208  20471  20897 --:--:-- --:--:-- --:--:-- 21028\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XPOST \"http://$ES_HOST/_ingest/pipeline/_simulate?pretty\" -H 'Content-Type:application/json' -d @- << EOF\n",
    "{\n",
    "    \"pipeline\":{\n",
    "        \"description\":\"_description\",\n",
    "        \"processors\":[\n",
    "            {\n",
    "                \"set\":{\n",
    "                    \"field\":\"country\",\n",
    "                    \"value\":\"japan\"\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    \"docs\":[\n",
    "        {\n",
    "            \"_index\":\"$INDEX\",\n",
    "            \"_type\":\"$TYPE\",\n",
    "            \"_id\":\"id\",\n",
    "            \"_source\":{\n",
    "                \"timestamp\":\"1\",\n",
    "                \"atmospheric_pressure\":\"1000.4\",\n",
    "                \"sea_level_pressure\":\"1003.4\",\n",
    "                \"precipitation_day\":\"0\",\n",
    "                \"precipitation_max_hour\":\"0\",\n",
    "                \"precipitation_max_10min\":\"0\",\n",
    "                \"temperature_avg\":\"3.7\",\n",
    "                \"temperature_max\":\"8.2\",\n",
    "                \"temperature_min\":\"0.7\",\n",
    "                \"humidity_avg\":\"41\",\n",
    "                \"humidity_min\":\"30\",\n",
    "                \"wind_speed_avg\":\"4.4\",\n",
    "                \"wind_speed_max\":\"9.4\",\n",
    "                \"wind_direction\":\"西\",\n",
    "                \"wind_speed_max_moment\":\"18.7\",\n",
    "                \"wind_direction_max_moment\":\"北西\",\n",
    "                \"sunshine_duration\":\"1.3\",\n",
    "                \"snowfall\":\"\",\n",
    "                \"snowfall_max\":\"\",\n",
    "                \"information_daytime\":\"時々晴一時雪\",\n",
    "                \"information_night\":\"晴\"\n",
    "            }\n",
    "        },\n",
    "        {\n",
    "            \"_index\":\"$INDEX\",\n",
    "            \"_type\":\"$TYPE\",\n",
    "            \"_id\":\"id\",\n",
    "            \"_source\":{\n",
    "                \"timestamp\":\"2\",\n",
    "                \"atmospheric_pressure\":\"1007.4\",\n",
    "                \"sea_level_pressure\":\"1010.4\",\n",
    "                \"precipitation_day\":\"\",\n",
    "                \"precipitation_max_hour\":\"\",\n",
    "                \"precipitation_max_10min\":\"\",\n",
    "                \"temperature_avg\":\"2.7\",\n",
    "                \"temperature_max\":\"7.9\",\n",
    "                \"temperature_min\":\"-2.2\",\n",
    "                \"humidity_avg\":\"41\",\n",
    "                \"humidity_min\":\"19\",\n",
    "                \"wind_speed_avg\":\"2.1\",\n",
    "                \"wind_speed_max\":\"4.2\",\n",
    "                \"wind_direction\":\"西\",\n",
    "                \"wind_speed_max_moment\":\"7.8\",\n",
    "                \"wind_direction_max_moment\":\"西\",\n",
    "                \"sunshine_duration\":\"6\",\n",
    "                \"snowfall\":\"\",\n",
    "                \"snowfall_max\":\"\",\n",
    "                \"information_daytime\":\"時々曇\",\n",
    "                \"information_night\":\"一時曇\"\n",
    "            }\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154e840-152a-11e7-b31a-0242ac12000d",
     "next": "9154ea0c-152a-11e7-b31a-0242ac12000d",
     "previous": "9154e66a-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## トラブルシューティング"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154ea0c-152a-11e7-b31a-0242ac12000d",
     "next": "9154ebd8-152a-11e7-b31a-0242ac12000d",
     "previous": "9154e840-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### 文字コード指定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154ebd8-152a-11e7-b31a-0242ac12000d",
     "next": "9154ed9a-152a-11e7-b31a-0242ac12000d",
     "previous": "9154ea0c-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Logstashでデータを扱う際には、文字コードを指定することができます。  \n",
    "codecプラグインの中でcharsetという項目を設定することで文字コードが指定されます。  \n",
    "デフォルト設定はutf-8になっています。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154ed9a-152a-11e7-b31a-0242ac12000d",
     "next": "9154ef8e-152a-11e7-b31a-0242ac12000d",
     "previous": "9154ebd8-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "文字コードをshift-jisに指定する場合の設定例を示します。"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154ef8e-152a-11e7-b31a-0242ac12000d",
     "next": "9154f182-152a-11e7-b31a-0242ac12000d",
     "previous": "9154ed9a-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "input{\n",
    "  stdin{\n",
    "    codec => plain{\n",
    "      charset => \"Shift_JIS\"\n",
    "    }\n",
    "  }\n",
    "}\n",
    "output{\n",
    "  stdout{\n",
    "    codec => \"rubydebug\"\n",
    "  }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154f182-152a-11e7-b31a-0242ac12000d",
     "next": "9154f34e-152a-11e7-b31a-0242ac12000d",
     "previous": "9154ef8e-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### 自動で入るフィールドの削除"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154f34e-152a-11e7-b31a-0242ac12000d",
     "next": "9154f51a-152a-11e7-b31a-0242ac12000d",
     "previous": "9154f182-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Elasticsearchにドキュメントを登録する際に、自動で付加されるフィールドがあります。  \n",
    "例えば\\_sourceフィールドなどが自動的に付加されます。  \n",
    "これらのフィールドは高速に検索を行う時などに有用なフィールドですが、削除することでindexのサイズを少なくすることもできます。  \n",
    "_sourceの詳細に関してはElastic Referenceの[Meta-Fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-fields.html)を参照してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154f51a-152a-11e7-b31a-0242ac12000d",
     "next": "9154f6dc-152a-11e7-b31a-0242ac12000d",
     "previous": "9154f34e-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "上記の_sourceは、Elasticsearchへの登録時に作成されるフィールドですが、  \n",
    "hostや@timestampなどのようにLogstashで自動的に作成されるフィールドもあります。  \n",
    "これらのフィールドはLogstashの設定ファイルに設定を記述することで削除することができます。  \n",
    "詳しくはElastic Referenceの[mutate](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-remove_field)を参照してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154f6dc-152a-11e7-b31a-0242ac12000d",
     "next": "9154f8a8-152a-11e7-b31a-0242ac12000d",
     "previous": "9154f51a-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### ドキュメントIDの重要性と付与方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154f8a8-152a-11e7-b31a-0242ac12000d",
     "next": "9154fa7e-152a-11e7-b31a-0242ac12000d",
     "previous": "9154f6dc-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "#### ドキュメントID の重要性"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154fa7e-152a-11e7-b31a-0242ac12000d",
     "next": "9154fc40-152a-11e7-b31a-0242ac12000d",
     "previous": "9154f8a8-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Elasticsearchに登録されるドキュメントは、ドキュメントごとに固有のドキュメントIDを持ちます。  \n",
    "Logstashを用いてElasticsearchへのドキュメント登録を行う場合、ドキュメントIDを明示的に指定しなければランダムなドキュメントIDが付与されるようになっています。  \n",
    "  \n",
    "既にElasticsearchに登録されているものと重複するドキュメントIDを持ったドキュメントを新たに登録(index)しようとした場合には、ドキュメントは更新されません。  \n",
    "新しいドキュメントで上書きする場合は、更新(update)する必要があります。  \n",
    "この振る舞いは、index時の設定によって変更可能です。  \n",
    "設定の変更方法については、Logstash ReferenceのOutputプラグインelasticsearchの[actionパラメータの説明](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-action)を参照してください。\n",
    "  \n",
    "ドキュメントIDを明示的に指定することの利点として挙げられるのが、ドキュメントの再登録時の重複防止です。  \n",
    "明示的に指定しなかった場合、内容が同一のドキュメントを複数回Elasticsearchに登録すると、その分だけ重複して登録されてしまいます。  \n",
    "ドキュメントIDを指定しておくことで、ドキュメント登録の処理が中断してしまった場合でも重複を気にせず再登録することができます。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "lc_cell_meme": {
     "current": "9154fc40-152a-11e7-b31a-0242ac12000d",
     "next": "9154fe0c-152a-11e7-b31a-0242ac12000d",
     "previous": "9154fa7e-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "#### ドキュメントIDの付与方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9154fe0c-152a-11e7-b31a-0242ac12000d",
     "next": "91550014-152a-11e7-b31a-0242ac12000d",
     "previous": "9154fc40-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "LogstashでドキュメントIDを明示的に付与するにはLogstashのoutputにある、elasticsearchプラグインの設定を利用します。  \n",
    "ドキュメントIDという項目で直接指定することができます。  \n",
    "他のフィールドの値を用いることもできるので、例えばmessageの内容をもとにドキュメントIDを付与すれば、同一内容のドキュメントに関して常に一意なidを付与することができます。  \n",
    "具体的な設定は次の通りです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "lc_cell_meme": {
     "current": "91550014-152a-11e7-b31a-0242ac12000d",
     "next": "915501f4-152a-11e7-b31a-0242ac12000d",
     "previous": "9154fe0c-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input{\r",
      "\r\n",
      "  stdin{\r",
      "\r\n",
      "  }\r",
      "\r\n",
      "}\r",
      "\r\n",
      "\r",
      "\r\n",
      "filter{\r",
      "\r\n",
      "  csv{\r",
      "\r\n",
      "    columns => [\"timestamp\",\"location\",\"atmospheric_pressure\",\"sea_level_pressure\",\"precipitation_day\",\"precipitation_max_hour\",\"precipitation_max_10min\",\"temperature_avg\",\"temperature_max\",\"temperature_min\",\"humidity_avg\",\"humidity_min\",\"wind_speed_avg\",\"wind_speed_max\",\"wind_direction\",\"wind_speed_max_moment\",\"wind_direction_max_moment\",\"sunshine_duration\",\"snowfall\",\"snowfall_max\",\"information_daytime\",\"information_night\"]\r",
      "\r\n",
      "  }\r",
      "\r\n",
      "  date{\r",
      "\r\n",
      "    match => [\"timestamp\",\"ISO8601\"]\r",
      "\r\n",
      "    remove_field => \"timestamp\"\r",
      "\r\n",
      "  }\r",
      "\r\n",
      "}\r",
      "\r\n",
      "\r",
      "\r\n",
      "output{\r",
      "\r\n",
      "  elasticsearch{\r",
      "\r\n",
      "    index => \"document_id_test\"\r",
      "\r\n",
      "    document_id => \"%{location}-%{@timestamp}\"\r",
      "\r\n",
      "    hosts => \"localhost\"\r",
      "\r\n",
      "  }\r",
      "\r\n",
      "}"
     ]
    }
   ],
   "source": [
    "!cat logstash_conf/document_id.conf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915501f4-152a-11e7-b31a-0242ac12000d",
     "next": "915503c0-152a-11e7-b31a-0242ac12000d",
     "previous": "91550014-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### 処理時間算出"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "915503c0-152a-11e7-b31a-0242ac12000d",
     "next": "91550596-152a-11e7-b31a-0242ac12000d",
     "previous": "915501f4-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "データ投入作業時にかかる時間を、概算で算出する方法を示します。  \n",
    "やり方として、定期的にElasticsearchへクエリを送り、単位時間あたりどのくらいデータ投入ができているかを計測し算出します。  \n",
    "次の全件検索コマンドを実行します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "lc_cell_meme": {
     "current": "91550596-152a-11e7-b31a-0242ac12000d",
     "next": "9155076c-152a-11e7-b31a-0242ac12000d",
     "previous": "915503c0-152a-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"took\" : 5,\n",
      "  \"timed_out\" : false,\n",
      "  \"_shards\" : {\n",
      "    \"total\" : 5,\n",
      "    \"successful\" : 5,\n",
      "    \"failed\" : 0\n",
      "  },\n",
      "  \"hits\" : {\n",
      "    \"total\" : 110,\n",
      "    \"max_score\" : 0.0,\n",
      "    \"hits\" : [ ]\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   247  100   194  100    53  19088   5214 --:--:-- --:--:-- --:--:-- 21555\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/*$INDEX/_search?pretty\" -H 'Content-Type:application/json' -d @- << EOF\n",
    "{\n",
    "  \"size\": 0,\n",
    "  \"query\" : { \n",
    "    \"match_all\" : {} \n",
    "  } \n",
    "}\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9155076c-152a-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "9155076c-152a-11e7-b31a-0242ac12000d",
       "next": null,
       "previous": "91550596-152a-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "a828c718-1830-11e7-b31a-0242ac12000d",
     "previous": "91550596-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "\"hits\"配下の\"total\"が投入されたデータ数になり、この値を使用します。  \n",
    "例えば、ある程度の間隔でコマンドを実行し、totalの変化量を知ることでおおまかにですが全体の処理時間を算出することができます。  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "a828c718-1830-11e7-b31a-0242ac12000d",
     "next": "a828cbc8-1830-11e7-b31a-0242ac12000d",
     "previous": "9155076c-152a-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### タイムゾーンについての注意"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "a828cbc8-1830-11e7-b31a-0242ac12000d",
     "next": "a828ce0c-1830-11e7-b31a-0242ac12000d",
     "previous": "a828c718-1830-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Elasticsearchの内部ではデータの時間をUTCで持っています。  \n",
    "例えば、インダックス名に\"some_index-2015.01.01\"のような日付を使用していた場合、日本時間で1月1日8時のデータは\"some_index-2014.12.31\"の23時のデータとして持ちます。9時間ずれた結果、前日のインデックスに登録されます。インデックスを指定して検索する場合には注意が必要です。  \n",
    "  \n",
    "データ投入時に使用したいタイムゾーンに合わせる必要があります。  \n",
    "データ投入後にタイムゾーンを変更する場合、再投入が必要となります。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "a828ce0c-1830-11e7-b31a-0242ac12000d",
     "next": null,
     "previous": "a828cbc8-1830-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "タイムゾーンを指定したクエリについてはElasticsearch Referenceの[Time Zone](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html#_time_zone_2)を参照してください。"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  },
  "lc_notebook_meme": {
   "current": "91542554-152a-11e7-b31a-0242ac12000d",
   "lc_server_signature": {
    "history": []
   }
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "389px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_position": {
    "height": "610px",
    "left": "0px",
    "right": "1169.11px",
    "top": "107px",
    "width": "212px"
   },
   "toc_section_display": "block",
   "toc_window_display": true,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}