01_52_Backup_Restore.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530584e-1509-11e7-b31a-0242ac12000d",
     "next": "95305a9c-1509-11e7-b31a-0242ac12000d",
     "previous": null
    }
   },
   "source": [
    "# バックアップ＆リストア\n",
    "----\n",
    "Data&nbsp;Node上にあるデータのバックアップと リストアについて説明します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "95305a9c-1509-11e7-b31a-0242ac12000d",
     "next": "95305cae-1509-11e7-b31a-0242ac12000d",
     "previous": "9530584e-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## 準備"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "95305cae-1509-11e7-b31a-0242ac12000d",
     "next": "95305e8e-1509-11e7-b31a-0242ac12000d",
     "previous": "95305a9c-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "本章のコマンドを実行するための設定を行います。  \n",
    "\n",
    "各サーバにAnsibleコマンドを発行するため、  \n",
    "Ansibleを実行可能なユーザアカウントとそのSSHキーを次のセルで設定してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "deletable": false,
    "editable": false,
    "lc_cell_meme": {
     "current": "a71ead0e-12ff-11e8-8ae7-0242ac110002",
     "next": "95305cae-1509-11e7-b31a-0242ac12000d",
     "previous": "e56420d2-1510-11e7-b31a-0242ac12000d"
    },
    "run_through_control": {
     "frozen": true
    }
   },
   "outputs": [],
   "source": [
    "#elasticsearchをインストールするサーバーでansibleコマンドを実行するユーザー\n",
    "USER='ansible'\n",
    "\n",
    "#公開鍵認証を行う場合の秘密鍵のパス\n",
    "#KEYPATH='/notebooks/etc/key/id_rsa'\n",
    "KEYPATH = '~/.ssh/ansible_id_rsa'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "95305cae-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "95305cae-1509-11e7-b31a-0242ac12000d",
       "next": "95305e8e-1509-11e7-b31a-0242ac12000d",
       "previous": "95305a9c-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "95305e8e-1509-11e7-b31a-0242ac12000d",
     "previous": "a71ead0e-12ff-11e8-8ae7-0242ac110002"
    }
   },
   "source": [
    "\n",
    "\n",
    "セットアップ済みの環境のうちコマンドを送る先であるCoordinating(Client) Nodeのホストアドレスを、次のセルに記入して実行し、保存してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "lc_cell_meme": {
     "current": "95305e8e-1509-11e7-b31a-0242ac12000d",
     "next": "9530606e-1509-11e7-b31a-0242ac12000d",
     "previous": "95305cae-1509-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: ES_HOST=XXX.XXX.XXX.233:9200\n"
     ]
    }
   ],
   "source": [
    "%env ES_HOST=XXX.XXX.XXX.233:9200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530606e-1509-11e7-b31a-0242ac12000d",
     "next": "95306244-1509-11e7-b31a-0242ac12000d",
     "previous": "95305e8e-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "データを蓄積する先のインデックスを、次のセルを実行して保存してください。  \n",
    "\n",
    "※特にセルの内容の書き換えは必要ありません。  \n",
    "　前の章と異なるインデックス名にしているのは、この章のコマンドで投入する内容が前の章までのものと混ざらないようにするためです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "lc_cell_meme": {
     "current": "95306244-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "95306244-1509-11e7-b31a-0242ac12000d",
       "next": "9530642e-1509-11e7-b31a-0242ac12000d",
       "previous": "9530606e-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "95305cae-1509-11e7-b31a-0242ac12000d",
     "previous": "9530606e-1509-11e7-b31a-0242ac12000d"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: INDEX=meteorological-data-*\n"
     ]
    }
   ],
   "source": [
    "%env INDEX=meteorological-data-*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "95305cae-1509-11e7-b31a-0242ac12000d",
     "next": "91c52538-12f9-11e8-8ae7-0242ac110002",
     "previous": "95306244-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "バックアップファイルの保存先ディレクトリを、次のセルに記入して実行し、保存してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "deletable": false,
    "editable": false,
    "lc_cell_meme": {
     "current": "91c52538-12f9-11e8-8ae7-0242ac110002",
     "next": "95308d96-1509-11e7-b31a-0242ac12000d",
     "previous": "95305cae-1509-11e7-b31a-0242ac12000d"
    },
    "run_through_control": {
     "frozen": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: BACKUP_DIR=/mnt/backup/my_backup\n"
     ]
    }
   ],
   "source": [
    "backup_dir=\"/mnt/backup/my_backup\"\n",
    "%env BACKUP_DIR=backup_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "95308d96-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "95308d96-1509-11e7-b31a-0242ac12000d",
       "next": "95308f62-1509-11e7-b31a-0242ac12000d",
       "previous": "95308bc0-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "95308f62-1509-11e7-b31a-0242ac12000d",
     "previous": "95306244-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## バックアップ\n",
    "バックアップ方法について説明します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "95308f62-1509-11e7-b31a-0242ac12000d",
     "next": "9530912e-1509-11e7-b31a-0242ac12000d",
     "previous": "95308d96-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Elasticsearchのバックアップはクラスタ・index単位で取得することができます。   \n",
    "詳細はElasticsearch Referenceの[Snapshot And Restore](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html)を参照してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530912e-1509-11e7-b31a-0242ac12000d",
     "next": "953092fa-1509-11e7-b31a-0242ac12000d",
     "previous": "95308f62-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### バックアップ対象indexの容量確認\n",
    "バックアップの保存前に、対象となるindexが保存先のストレージに収まる容量かを確認します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "953092fa-1509-11e7-b31a-0242ac12000d",
     "next": "953094d0-1509-11e7-b31a-0242ac12000d",
     "previous": "9530912e-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "indexの容量はcat APIにより確認できます。  \n",
    "詳細はElasticsearch Referenceの[cat-indices](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html)を参照してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "lc_cell_meme": {
     "current": "953094d0-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "953094d0-1509-11e7-b31a-0242ac12000d",
       "next": "95309692-1509-11e7-b31a-0242ac12000d",
       "previous": "953092fa-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "7b88a19c-183f-11e7-b31a-0242ac12000d",
     "previous": "953092fa-1509-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "health status index          uuid                   pri rep docs.count docs.deleted store.size pri.store.size\n",
      "green  open   sample_index   NCA-vz62SfisSEoiBBdyHA   5   1        110            0    907.1kb        453.5kb\n",
      "green  open   reindex_source FTQ0QcHxR9qVpt_sART-YQ   5   1          1            0      8.9kb          4.4kb\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   330  100   330    0     0  24422      0 --:--:-- --:--:-- --:--:-- 25384\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/_cat/indices?v&s=index\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "7b88a19c-183f-11e7-b31a-0242ac12000d",
     "next": "7b88a5f2-183f-11e7-b31a-0242ac12000d",
     "previous": "953094d0-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### リポジトリの作成（初回のみ）\n",
    "保存先であるリポジトリを作成します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "545e860e-12ee-11e8-8ae7-0242ac110002",
     "next": "80bcb810-12ee-11e8-8ae7-0242ac110002",
     "previous": "7b88a19c-183f-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "バックアップ用ディレクトリを作成します。\n",
    "オーナーとグループは \"elasticsearch\"を指定します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "lc_cell_meme": {
     "current": "80bcb810-12ee-11e8-8ae7-0242ac110002",
     "next": "80bcbafe-12ee-11e8-8ae7-0242ac110002",
     "previous": "545e860e-12ee-11e8-8ae7-0242ac110002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [all] *********************************************************************\n",
      "\n",
      "TASK [Gathering Facts] *********************************************************\n",
      "\u001b[0;32mok: [192.168.122.80]\u001b[0m\n",
      "\n",
      "TASK [cd_mnt] ******************************************************************\n",
      "\u001b[0;33mchanged: [192.168.122.80]\u001b[0m\n",
      "\n",
      "TASK [mkdir] *******************************************************************\n",
      "\u001b[1;35m [WARNING]: Consider using file module with state=directory rather than running\u001b[0m\n",
      "\u001b[1;35mmkdir\u001b[0m\n",
      "\u001b[1;35m\u001b[0m\n",
      "\u001b[0;33mchanged: [192.168.122.80]\u001b[0m\n",
      "\n",
      "TASK [chown] *******************************************************************\n",
      "\u001b[1;35m [WARNING]: Consider using file module with owner rather than running chown\u001b[0m\n",
      "\u001b[1;35m\u001b[0m\n",
      "\u001b[0;33mchanged: [192.168.122.80]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33m192.168.122.80\u001b[0m             : \u001b[0;32mok=4   \u001b[0m \u001b[0;33mchanged=3   \u001b[0m unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook playbooks/make_backup_directory.yml --extra-vars \"backup_dir=$backup_dir\" -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "80bcbafe-12ee-11e8-8ae7-0242ac110002",
     "next": "ab2ab304-12ee-11e8-8ae7-0242ac110002",
     "previous": "80bcb810-12ee-11e8-8ae7-0242ac110002"
    }
   },
   "source": [
    "elasticsearch.ymlにバックアップ用ディレクトリパスを設定します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "lc_cell_meme": {
     "current": "ab2ab304-12ee-11e8-8ae7-0242ac110002",
     "next": "5bc2db02-12fd-11e8-8ae7-0242ac110002",
     "previous": "80bcbafe-12ee-11e8-8ae7-0242ac110002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [all] *********************************************************************\n",
      "\n",
      "TASK [Gathering Facts] *********************************************************\n",
      "\u001b[0;32mok: [192.168.122.80]\u001b[0m\n",
      "\n",
      "TASK [set_shared file system repository] ***************************************\n",
      "\u001b[0;32mok: [192.168.122.80]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;32m192.168.122.80\u001b[0m             : \u001b[0;32mok=2   \u001b[0m changed=0    unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook playbooks/set_backup_directory.yml --extra-vars \"backup_dir=$backup_dir\" -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "5bc2db02-12fd-11e8-8ae7-0242ac110002",
     "next": "5bc2dde6-12fd-11e8-8ae7-0242ac110002",
     "previous": "ab2ab304-12ee-11e8-8ae7-0242ac110002"
    }
   },
   "source": [
    "elasticsearchを再起動します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "lc_cell_meme": {
     "current": "5bc2dde6-12fd-11e8-8ae7-0242ac110002",
     "history": [
      {
       "current": "5bc2dde6-12fd-11e8-8ae7-0242ac110002",
       "next": "76322c42-12fb-11e8-8ae7-0242ac110002",
       "previous": "5bc2db02-12fd-11e8-8ae7-0242ac110002"
      }
     ],
     "next": "7b88a5f2-183f-11e7-b31a-0242ac12000d",
     "previous": "5bc2db02-12fd-11e8-8ae7-0242ac110002"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "PLAY [all] *********************************************************************\n",
      "\n",
      "TASK [Gathering Facts] *********************************************************\n",
      "\u001b[0;32mok: [192.168.122.80]\u001b[0m\n",
      "\n",
      "TASK [restart elasticsearch] ***************************************************\n",
      "\u001b[0;33mchanged: [192.168.122.80]\u001b[0m\n",
      "\n",
      "PLAY RECAP *********************************************************************\n",
      "\u001b[0;33m192.168.122.80\u001b[0m             : \u001b[0;32mok=2   \u001b[0m \u001b[0;33mchanged=1   \u001b[0m unreachable=0    failed=0   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "!ansible-playbook playbooks/restart_cluster.yml -i ./hosts -u $USER --private-key=$KEYPATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "7b88a5f2-183f-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "7b88a5f2-183f-11e7-b31a-0242ac12000d",
       "next": "95309854-1509-11e7-b31a-0242ac12000d",
       "previous": "7b88a19c-183f-11e7-b31a-0242ac12000d"
      },
      {
       "current": "7b88a5f2-183f-11e7-b31a-0242ac12000d",
       "next": "95309a48-1509-11e7-b31a-0242ac12000d",
       "previous": "7b88a19c-183f-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "95309c28-1509-11e7-b31a-0242ac12000d",
     "previous": "7b88a19c-183f-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "Elasticsearchでバックアップを保存するには、データのスナップショットを保存する先であるリポジトリを作成する必要があります。  \n",
    "my_backupという名前のリポジトリを作るコマンドは次の通りです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "lc_cell_meme": {
     "current": "95309c28-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "95309c28-1509-11e7-b31a-0242ac12000d",
       "next": "95309e12-1509-11e7-b31a-0242ac12000d",
       "previous": "95309a48-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "95309e12-1509-11e7-b31a-0242ac12000d",
     "previous": "7b88a5f2-183f-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"acknowledged\" : true\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   137  100    28  100   109    166    648 --:--:-- --:--:-- --:--:--   652\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XPUT http://$ES_HOST/_snapshot/my_backup?pretty -H 'Content-Type:application/json' -d @- << EOF\n",
    "{\n",
    "    \"type\": \"fs\",\n",
    "    \"settings\": {\n",
    "        \"location\": \"$BACKUP_DIR\",\n",
    "        \"compress\": true\n",
    "    }\n",
    "}\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "95309e12-1509-11e7-b31a-0242ac12000d",
     "next": "95309fde-1509-11e7-b31a-0242ac12000d",
     "previous": "95309c28-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "my_backupリポジトリが作成されたことを確認します"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "lc_cell_meme": {
     "current": "95309fde-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "95309fde-1509-11e7-b31a-0242ac12000d",
       "next": "9530a1aa-1509-11e7-b31a-0242ac12000d",
       "previous": "95309e12-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "95309692-1509-11e7-b31a-0242ac12000d",
     "previous": "95309e12-1509-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"my_backup\" : {\n",
      "    \"type\" : \"fs\",\n",
      "    \"settings\" : {\n",
      "      \"compress\" : \"true\",\n",
      "      \"location\" : \"/mnt/backup/my_backup\"\n",
      "    }\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   140  100   140    0     0   8829      0 --:--:-- --:--:-- --:--:--  9333\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/_snapshot/my_backup?pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "95309692-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "95309692-1509-11e7-b31a-0242ac12000d",
       "next": "95309854-1509-11e7-b31a-0242ac12000d",
       "previous": "953094d0-1509-11e7-b31a-0242ac12000d"
      },
      {
       "current": "95309692-1509-11e7-b31a-0242ac12000d",
       "next": "9530a1aa-1509-11e7-b31a-0242ac12000d",
       "previous": "95309fde-1509-11e7-b31a-0242ac12000d"
      },
      {
       "current": "95309692-1509-11e7-b31a-0242ac12000d",
       "next": "4a33ce5e-1840-11e7-b31a-0242ac12000d",
       "previous": "95309fde-1509-11e7-b31a-0242ac12000d"
      },
      {
       "current": "95309692-1509-11e7-b31a-0242ac12000d",
       "next": "4a33d228-1840-11e7-b31a-0242ac12000d",
       "previous": "95309fde-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "99a770ee-1840-11e7-b31a-0242ac12000d",
     "previous": "95309fde-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### バックアップ実行\n",
    "実際にデータをバックアップします。  \n",
    "クラスタ全体で取るのか、index単位で取るのかでコマンドを切り替えてください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99a770ee-1840-11e7-b31a-0242ac12000d",
     "next": "4a33d228-1840-11e7-b31a-0242ac12000d",
     "previous": "95309692-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "\n",
    "オプションについては、Elasticsearch Referenceの[Snapshot And Restore](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html)を参照してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "4a33d228-1840-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "4a33d228-1840-11e7-b31a-0242ac12000d",
       "next": "9530a1aa-1509-11e7-b31a-0242ac12000d",
       "previous": "4a33ce5e-1840-11e7-b31a-0242ac12000d"
      },
      {
       "current": "4a33d228-1840-11e7-b31a-0242ac12000d",
       "next": "9530a1aa-1509-11e7-b31a-0242ac12000d",
       "previous": "95309692-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "9530a1aa-1509-11e7-b31a-0242ac12000d",
     "previous": "99a770ee-1840-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■クラスタ全体をバックアップ**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530a1aa-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "9530a1aa-1509-11e7-b31a-0242ac12000d",
       "next": "9530a380-1509-11e7-b31a-0242ac12000d",
       "previous": "95309fde-1509-11e7-b31a-0242ac12000d"
      },
      {
       "current": "9530a1aa-1509-11e7-b31a-0242ac12000d",
       "next": "9530a380-1509-11e7-b31a-0242ac12000d",
       "previous": "95309692-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "9530a380-1509-11e7-b31a-0242ac12000d",
     "previous": "4a33d228-1840-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "クラスタ全体のバックアップを、my_backupリポジトリにsnapshot_1という名前で保存するコマンドは次の通りです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "lc_cell_meme": {
     "current": "9530a380-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "9530a380-1509-11e7-b31a-0242ac12000d",
       "next": "9530a54c-1509-11e7-b31a-0242ac12000d",
       "previous": "9530a1aa-1509-11e7-b31a-0242ac12000d"
      },
      {
       "current": "9530a380-1509-11e7-b31a-0242ac12000d",
       "next": "9530aaa6-1509-11e7-b31a-0242ac12000d",
       "previous": "9530a1aa-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "99a775a8-1840-11e7-b31a-0242ac12000d",
     "previous": "9530a1aa-1509-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"snapshot\" : {\n",
      "    \"snapshot\" : \"snapshot_1\",\n",
      "    \"uuid\" : \"3T5V4TwsQ7C1a_O0cqO7NA\",\n",
      "    \"version_id\" : 5000099,\n",
      "    \"version\" : \"5.0.0\",\n",
      "    \"indices\" : [\n",
      "      \"sample_index\",\n",
      "      \"meteorological-data-2015.07\",\n",
      "      \"meteorological-data-2015.08\",\n",
      "      \"meteorological-data-2015.03\",\n",
      "      \"meteorological-data-2015.09\",\n",
      "      \"meteorological-data-2015.05\",\n",
      "      \"meteorological-data-2015.01\",\n",
      "      \"meteorological-data-2015.04\",\n",
      "      \"meteorological-data-2015.12\",\n",
      "      \"meteorological-data-2015.11\",\n",
      "      \"meteorological-data-2015.10\",\n",
      "      \"meteorological-data-2015.02\",\n",
      "      \"meteorological-data-2015.06\",\n",
      "      \"meteorological-data-2014.12\"\n",
      "    ],\n",
      "    \"state\" : \"SUCCESS\",\n",
      "    \"start_time\" : \"2016-12-13T04:31:41.470Z\",\n",
      "    \"start_time_in_millis\" : 1481603501470,\n",
      "    \"end_time\" : \"2016-12-13T04:31:42.079Z\",\n",
      "    \"end_time_in_millis\" : 1481603502079,\n",
      "    \"duration_in_millis\" : 609,\n",
      "    \"failures\" : [ ],\n",
      "    \"shards\" : {\n",
      "      \"total\" : 70,\n",
      "      \"failed\" : 0,\n",
      "      \"successful\" : 70\n",
      "    }\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100  1019  100  1019    0     0   1555      0 --:--:-- --:--:-- --:--:--  1555\r",
      "100  1019  100  1019    0     0   1555      0 --:--:-- --:--:-- --:--:--  1555\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XPUT \"http://$ES_HOST/_snapshot/my_backup/snapshot_1?wait_for_completion=true&pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "99a775a8-1840-11e7-b31a-0242ac12000d",
     "next": "9530aaa6-1509-11e7-b31a-0242ac12000d",
     "previous": "9530a380-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "**■インデックスを指定してバックアップ**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530aaa6-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "9530aaa6-1509-11e7-b31a-0242ac12000d",
       "next": "9530ac7c-1509-11e7-b31a-0242ac12000d",
       "previous": "9530a8da-1509-11e7-b31a-0242ac12000d"
      },
      {
       "current": "9530aaa6-1509-11e7-b31a-0242ac12000d",
       "next": "9530b3a2-1509-11e7-b31a-0242ac12000d",
       "previous": "9530a8da-1509-11e7-b31a-0242ac12000d"
      },
      {
       "current": "9530aaa6-1509-11e7-b31a-0242ac12000d",
       "next": "9530b3a2-1509-11e7-b31a-0242ac12000d",
       "previous": "9530a380-1509-11e7-b31a-0242ac12000d"
      },
      {
       "current": "9530aaa6-1509-11e7-b31a-0242ac12000d",
       "next": "9530b3a2-1509-11e7-b31a-0242ac12000d",
       "previous": "99a775a8-1840-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "9530b56e-1509-11e7-b31a-0242ac12000d",
     "previous": "99a775a8-1840-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "meteorological-data-\\* indexのバックアップを、my_backupリポジトリにsnapshot_2という名前で保存するコマンドは次の通りです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "lc_cell_meme": {
     "current": "9530b56e-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "9530b56e-1509-11e7-b31a-0242ac12000d",
       "next": "9530b73a-1509-11e7-b31a-0242ac12000d",
       "previous": "9530b3a2-1509-11e7-b31a-0242ac12000d"
      },
      {
       "current": "9530b56e-1509-11e7-b31a-0242ac12000d",
       "next": "4a33d48a-1840-11e7-b31a-0242ac12000d",
       "previous": "9530b3a2-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "4a33d48a-1840-11e7-b31a-0242ac12000d",
     "previous": "9530aaa6-1509-11e7-b31a-0242ac12000d"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"accepted\":true}"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   116  100    17  100    99    153    891 --:--:-- --:--:-- --:--:--   900\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XPUT \"http://$ES_HOST/_snapshot/my_backup/snapshot_2\" -H 'Content-Type:application/json' -d @- << EOF\n",
    "{\n",
    "  \"indices\": \"$INDEX\",\n",
    "  \"ignore_unavailable\": true,\n",
    "  \"include_global_state\": false\n",
    "}\n",
    "EOF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "4a33d48a-1840-11e7-b31a-0242ac12000d",
     "next": "9530a54c-1509-11e7-b31a-0242ac12000d",
     "previous": "9530b56e-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### バックアップできたかの確認\n",
    "リポジトリの内容を閲覧して、バックアップが取れたかを確認します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530a54c-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "9530a54c-1509-11e7-b31a-0242ac12000d",
       "next": "9530a70e-1509-11e7-b31a-0242ac12000d",
       "previous": "9530a380-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "9530a70e-1509-11e7-b31a-0242ac12000d",
     "previous": "4a33d48a-1840-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "snapshot_1という名前のバックアップ内容を確認するコマンドは次の通りです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "lc_cell_meme": {
     "current": "9530a70e-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "9530a70e-1509-11e7-b31a-0242ac12000d",
       "next": "9530a8da-1509-11e7-b31a-0242ac12000d",
       "previous": "9530a54c-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "9530bad2-1509-11e7-b31a-0242ac12000d",
     "previous": "9530a54c-1509-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"snapshots\" : [\n",
      "    {\n",
      "      \"snapshot\" : \"snapshot_1\",\n",
      "      \"uuid\" : \"3T5V4TwsQ7C1a_O0cqO7NA\",\n",
      "      \"version_id\" : 5000099,\n",
      "      \"version\" : \"5.0.0\",\n",
      "      \"indices\" : [\n",
      "        \"sample_index\",\n",
      "        \"meteorological-data-2015.07\",\n",
      "        \"meteorological-data-2015.08\",\n",
      "        \"meteorological-data-2015.03\",\n",
      "        \"meteorological-data-2015.09\",\n",
      "        \"meteorological-data-2015.05\",\n",
      "        \"meteorological-data-2015.01\",\n",
      "        \"meteorological-data-2015.04\",\n",
      "        \"meteorological-data-2015.12\",\n",
      "        \"meteorological-data-2015.11\",\n",
      "        \"meteorological-data-2015.10\",\n",
      "        \"meteorological-data-2015.02\",\n",
      "        \"meteorological-data-2015.06\",\n",
      "        \"meteorological-data-2014.12\"\n",
      "      ],\n",
      "      \"state\" : \"SUCCESS\",\n",
      "      \"start_time\" : \"2016-12-13T04:31:41.470Z\",\n",
      "      \"start_time_in_millis\" : 1481603501470,\n",
      "      \"end_time\" : \"2016-12-13T04:31:42.079Z\",\n",
      "      \"end_time_in_millis\" : 1481603502079,\n",
      "      \"duration_in_millis\" : 609,\n",
      "      \"failures\" : [ ],\n",
      "      \"shards\" : {\n",
      "        \"total\" : 70,\n",
      "        \"failed\" : 0,\n",
      "        \"successful\" : 70\n",
      "      }\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100  1096  100  1096    0     0  72823      0 --:--:-- --:--:-- --:--:-- 78285\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/_snapshot/my_backup/snapshot_1?pretty\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530bad2-1509-11e7-b31a-0242ac12000d",
     "history": [
      {
       "current": "9530bad2-1509-11e7-b31a-0242ac12000d",
       "next": "9530bc9e-1509-11e7-b31a-0242ac12000d",
       "previous": "9530b8fc-1509-11e7-b31a-0242ac12000d"
      }
     ],
     "next": "9530bc9e-1509-11e7-b31a-0242ac12000d",
     "previous": "9530a70e-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "## リストア\n",
    "リストア方法について説明します。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530bc9e-1509-11e7-b31a-0242ac12000d",
     "next": "9530be74-1509-11e7-b31a-0242ac12000d",
     "previous": "9530bad2-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### リストア先のindex状況の確認\n",
    "データリストアはindexが空のところに入れることを前提としています。  \n",
    "リストア実施前に、同名indexが存在しないことを確認してください。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530be74-1509-11e7-b31a-0242ac12000d",
     "next": "9530c036-1509-11e7-b31a-0242ac12000d",
     "previous": "9530bc9e-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "現状存在するindexを確認するコマンドは、次の通りです。  \n",
    "詳細はElasticsearch Referenceの[cat-indices](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html)を参照してください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "lc_cell_meme": {
     "current": "9530c036-1509-11e7-b31a-0242ac12000d",
     "next": "9530c202-1509-11e7-b31a-0242ac12000d",
     "previous": "9530be74-1509-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "health status index          uuid                   pri rep docs.count docs.deleted store.size pri.store.size\n",
      "green  open   reindex_dest   hm0itWDWTP-h05dMnJSRAQ   5   1          1            0      8.8kb          4.4kb\n",
      "green  open   reindex_source 0CVwNJ0XQievecjHPq5kNQ   5   1          1            0      8.9kb          4.4kb\n",
      "green  open   sample_index   NCA-vz62SfisSEoiBBdyHA   5   1        110            0    907.1kb        453.5kb\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100   440  100   440    0     0  28299      0 --:--:-- --:--:-- --:--:-- 29333\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XGET \"http://$ES_HOST/_cat/indices?v&s=index\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530c202-1509-11e7-b31a-0242ac12000d",
     "next": "9530c3ce-1509-11e7-b31a-0242ac12000d",
     "previous": "9530c036-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "また、snapshotから一部のindexをリストアすることやindex名を変更しながらリストアすることも可能ですので、  \n",
    "状況に応じて必要なindexをリストアしてください。    \n",
    "詳細はElasticsearch Referenceの[Restoring from a Snapshotedit](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html)を参照してください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530c3ce-1509-11e7-b31a-0242ac12000d",
     "next": "9530c5ae-1509-11e7-b31a-0242ac12000d",
     "previous": "9530c202-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "### リストア実行\n",
    "実際にデータをリストアします。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lc_cell_meme": {
     "current": "9530c5ae-1509-11e7-b31a-0242ac12000d",
     "next": "9530c77a-1509-11e7-b31a-0242ac12000d",
     "previous": "9530c3ce-1509-11e7-b31a-0242ac12000d"
    }
   },
   "source": [
    "例として、[バックアップ実行](#バックアップ実行)で取得したバックアップ(snapshot_1)からリストアするコマンドは次の通りです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "lc_cell_meme": {
     "current": "9530c77a-1509-11e7-b31a-0242ac12000d",
     "next": null,
     "previous": "9530c5ae-1509-11e7-b31a-0242ac12000d"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"accepted\" : true\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "\r",
      "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r",
      "100    24  100    24    0     0    239      0 --:--:-- --:--:-- --:--:--   242\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "curl -XPOST \"http://$ES_HOST/_snapshot/my_backup/snapshot_1/_restore?pretty\""
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  },
  "lc_notebook_meme": {
   "current": "9530542a-1509-11e7-b31a-0242ac12000d",
   "lc_server_signature": {
    "history": []
   }
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "154px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_position": {
    "height": "754px",
    "left": "0px",
    "right": "1088px",
    "top": "106px",
    "width": "248px"
   },
   "toc_section_display": "block",
   "toc_window_display": true,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}