0_build_rgb_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Land Use Classification | Prepare Dataset\n",
    "\n",
    "The data for this project comes from the German Reseach Center for Artificial Intelligence's open source EuroSAT Sentinel-2 satellite image [dataset](http://madm.dfki.de/downloads). The data was downloaded locally and split into train, validation, and test folders suitable for keras using this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "import random\n",
    "from osgeo import gdal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = \"/Users/limesncoconuts2/datasets/\"\n",
    "SEED = random.seed(123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Folders\n",
    "\n",
    "In order to be compatible with the keras function `flow_from_directory`, I created train, validation, and test set folders for the data. Within each folder, the data was separated into more folders by category. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get names of categories\n",
    "categories = []\n",
    "tif_files = []\n",
    "for (dirpath, dirnames, filenames) in os.walk(path + 'land-use-tif/'):\n",
    "    categories.extend(dirnames)\n",
    "    tif_files += filenames\n",
    "tif_files.remove('.DS_Store')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create new folder with test train and valid sets with folders of all categories\n",
    "\n",
    "# make train, test, valid folders\n",
    "split_names = ['train', 'test', 'valid']\n",
    "for sp_name in split_names:\n",
    "    directory = path + 'land-use-jpeg/' + sp_name + '/'\n",
    "    if not os.path.exists(directory):\n",
    "        os.makedirs(directory)\n",
    "    # make category folders\n",
    "    for category in categories:\n",
    "        dir_cat = directory + category + '/'\n",
    "        if not os.path.exists(dir_cat):\n",
    "            os.makedirs(dir_cat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Translating File Type\n",
    "\n",
    "Compatibility with keras/tensorflow also required me to translate the files from tif to jpeg format. I found that this was easiest using the `gdal_translate` function in the command line, which I accessed within the notebook using the `!` notation. \n",
    "\n",
    "When translating, I selected bands 2, 3, and 4, which are the red, green, and blue bands. Choosing the RGB bands makes for a traditional image, rather than a specialized satellite image which may have extra near-infrared, red-edge, or short-wave infrared bands. This makes the model more accessible to the average, everyday image of the outdoors.\n",
    "\n",
    "After translating a file, I moved it to the training data folder. When all files of a certain category were trained and moved, I separated the jpeg files into the validation and test data folders with a train:validation:test separation of 80:10:10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "for category in categories:\n",
    "    directory = path + 'land-use-tif/' + category + '/'\n",
    "    directory_train = path + 'land-use-jpeg/train/' + category + '/'\n",
    "    directory_valid = path + 'land-use-jpeg/test/' + category + '/'\n",
    "    directory_test = path + 'land-use-jpeg/valid/' + category + '/'\n",
    "    \n",
    "    cat_files = os.listdir(directory)\n",
    "    \n",
    "    if '.DS_Store' in cat_files:\n",
    "        cat_files.remove('.DS_Store')\n",
    "        \n",
    "    # translate files from .tif to .jpeg\n",
    "    for file in cat_files:\n",
    "        file_no_ext = file.split('.')[0] # separate file name from extension\n",
    "        img_in = directory + file\n",
    "        img_out = directory + file_no_ext + '.jpeg'\n",
    "            \n",
    "        if not os.path.exists(directory_train + file_no_ext + '.jpeg'):\n",
    "            !gdal_translate -of JPEG $img_in $img_out -b 2 -b 3 -b 4 -scale # RGB bands\n",
    "            if os.path.exists(img_out):\n",
    "                shutil.move(img_out, directory_train + file_no_ext + '.jpeg')\n",
    "                \n",
    "    # remove .xml files that come from translation\n",
    "    for item in cat_files:\n",
    "        if item.endswith(\".xml\"):\n",
    "            os.remove(os.path.join(directory, item))\n",
    "                \n",
    "    # sort files into test and valid folders\n",
    "    filenames = os.listdir(directory_train)\n",
    "    filenames.sort()\n",
    "    if '.DS_Store' in filenames:\n",
    "        filenames.remove('.DS_Store')\n",
    "    random.shuffle(filenames)\n",
    "    split_1 = int(0.8 * len(filenames))\n",
    "    split_2 = int(0.9 * len(filenames))\n",
    "    train_filenames = filenames[:split_1]\n",
    "    valid_filenames = filenames[split_1:split_2]\n",
    "    test_filenames = filenames[split_2:]\n",
    "        \n",
    "    for file in filenames:\n",
    "        if file in valid_filenames:\n",
    "            shutil.move(directory_train + file, directory_valid + file)\n",
    "        elif file in test_filenames:\n",
    "            shutil.move(directory_train + file, directory_test + file)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}