\n",
+ "Question 9: Describe the various steps you have taken to make sure that the windstorm map is now exactly the same extent as the corine land cover map. Feel free to include lines of code in your answer and also describe the different functions you have used along the way.\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4959f306-df08-4548-8470-9083c814b203",
+ "metadata": {
+ "id": "4959f306-df08-4548-8470-9083c814b203"
+ },
+ "source": [
+ "Now its finally time to do our damage assessment! To do so, we need to convert our data to `numpy.arrays()` to do our calculation:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 54,
+ "id": "f0f82a7c-f2b1-43dd-bcb6-4a9d24738d75",
+ "metadata": {
+ "id": "f0f82a7c-f2b1-43dd-bcb6-4a9d24738d75"
+ },
+ "outputs": [],
+ "source": [
+ "landuse_map = CLC_region_wind['band_data'].to_numpy()[0,:,:]\n",
+ "wind_map = windstorm['FX'].to_numpy()[0,:,:]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "aca9bbe2-95f2-4b0d-9fb7-bb0ffc94ecef",
+ "metadata": {
+ "id": "aca9bbe2-95f2-4b0d-9fb7-bb0ffc94ecef"
+ },
+ "source": [
+ "And remember that our windstorm data was stored in **m/s**. Hence, we need to convert it to **km/h**:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 56,
+ "id": "11ddca33-7360-4efb-84f2-7613fd360ca3",
+ "metadata": {
+ "id": "11ddca33-7360-4efb-84f2-7613fd360ca3"
+ },
+ "outputs": [],
+ "source": [
+ "wind_map_kmh = wind_map* #convert to km/h"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "53b5ac7e-e817-49e2-b588-614e997adf8a",
+ "metadata": {
+ "id": "53b5ac7e-e817-49e2-b588-614e997adf8a"
+ },
+ "source": [
+ "And now let's run the DamageScanner to obtain the damage results"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 57,
+ "id": "31da4031-fdd4-485f-af99-30fa49fdabe0",
+ "metadata": {
+ "id": "31da4031-fdd4-485f-af99-30fa49fdabe0",
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "wind_damage_CLC = DamageScanner(landuse_map,wind_map_kmh,wind_curves,maxdam)[1]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a717a4a2-8d6b-436b-b855-bebc7835d0b3",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1000
+ },
+ "id": "a717a4a2-8d6b-436b-b855-bebc7835d0b3",
+ "outputId": "1f8aaf42-367a-4073-bb05-0342e2a42898"
+ },
+ "outputs": [],
+ "source": [
+ "wind_damage_CLC"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5UNySYvk-g4J",
+ "metadata": {
+ "id": "5UNySYvk-g4J",
+ "tags": []
+ },
+ "source": [
+ "#### Flood Damage\n",
+ "---\n",
+ "To Assess the flood damage, we are again going to make use of the [DamageScanner](https://damagescanner.readthedocs.io/en/latest/). The Corine Land Cover data is widely used in European flood risk assessments. As such, we can simply make use of pre-developed curves. We are using the damage curves as developed by Huizinga et al. (2007). Again, let's first load the maximum damages and the depth-damage curves:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 86,
+ "id": "ua2xyAGW-g4J",
+ "metadata": {
+ "id": "ua2xyAGW-g4J"
+ },
+ "outputs": [],
+ "source": [
+ "flood_curves = pd.read_excel(\"https://github.com/ElcoK/BigData_AED/raw/main/week5/damage_curves.xlsx\",sheet_name='flood_curves',engine='openpyxl')\n",
+ "maxdam = pd.read_excel(\"https://github.com/ElcoK/BigData_AED/raw/main/week5/damage_curves.xlsx\",sheet_name='maxdam')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "HT54wRvs-g4K",
+ "metadata": {
+ "id": "HT54wRvs-g4K"
+ },
+ "source": [
+ "And convert our data to `numpy.arrays()` to do our calculation:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 87,
+ "id": "qzXKNmg2-g4K",
+ "metadata": {
+ "id": "qzXKNmg2-g4K"
+ },
+ "outputs": [],
+ "source": [
+ "landuse_map = CLC_region['band_data'].to_numpy()\n",
+ "flood_map = flood_map_area['band_data'].to_numpy()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ttGra99k-g4K",
+ "metadata": {
+ "id": "ttGra99k-g4K"
+ },
+ "source": [
+ "And now let's run the DamageScanner to obtain the damage results"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 88,
+ "id": "2qL8UATu-g4K",
+ "metadata": {
+ "id": "2qL8UATu-g4K"
+ },
+ "outputs": [],
+ "source": [
+ "flood_damage_CLC = DamageScanner(landuse_map,flood_map,flood_curves,maxdam)[1]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "WnWu6AMUeFA2",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1000
+ },
+ "id": "WnWu6AMUeFA2",
+ "outputId": "9b3076fd-a548-462d-a9e1-f532d644ed04"
+ },
+ "outputs": [],
+ "source": [
+ "flood_damage_CLC"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "61359310",
+ "metadata": {},
+ "source": [
+ "\n",
+ "Question 10: Create a plot yourself to compare the results of both the windstorm and flood damage results. This could be a simple barplot, for example.\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "tpS3A5DA5NiA",
+ "metadata": {
+ "id": "tpS3A5DA5NiA"
+ },
+ "source": [
+ "Now let's try to do this again with the OpenStreetMap data. It would be most convenient to use the same damage curves, so we want to couple our OSM land-use information to the flood curves.\n",
+ "\n",
+ "Let's first have a look at the column values:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "V9W8ngNm5dI_",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "V9W8ngNm5dI_",
+ "outputId": "71a6bdf8-69d1-4f12-f0c6-02e6d185ab9c"
+ },
+ "outputs": [],
+ "source": [
+ "flood_curves.columns"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "Q_FkDCtN5se3",
+ "metadata": {
+ "id": "Q_FkDCtN5se3"
+ },
+ "source": [
+ "Ok this does not say anything to us yet. So let's have a look at the list of classes they are refering to:\n",
+ "\n",
+ "\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "C7zUFEoZ6Yzn",
+ "metadata": {
+ "id": "C7zUFEoZ6Yzn"
+ },
+ "source": [
+ "So the first column of our damage curves relates to \"111: Continuous urban fabric\", and so on. Now let's have a look at the land uses we have within our OSM data. Can you find all the unique values in our 'landuse' column of the **landuse** dataframe?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "wPX9jZWd6sa-",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "wPX9jZWd6sa-",
+ "outputId": "ba5fea0e-2791-4c81-cdc0-001a6703251f"
+ },
+ "outputs": [],
+ "source": []
+ },
+ {
+ "cell_type": "markdown",
+ "id": "qUJ7GwF_6vGf",
+ "metadata": {
+ "id": "qUJ7GwF_6vGf"
+ },
+ "source": [
+ "Our next step would be to attach a value to each of the land-use classes that corresponds to the value of each column (to make sure we have a damage curve linked to each land-use class)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 115,
+ "id": "5pGFDGjP6ujm",
+ "metadata": {
+ "id": "5pGFDGjP6ujm"
+ },
+ "outputs": [],
+ "source": [
+ "landuse_value_clc = [0, ## complete the list, where each value corresponds to a landuse class in the flood_curves dataframe]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 116,
+ "id": "IpudoRfT7pru",
+ "metadata": {
+ "id": "IpudoRfT7pru"
+ },
+ "outputs": [],
+ "source": [
+ "value_dict_clc = dict(zip(landuse.landuse.unique(),landuse_value_clc))\n",
+ "value_dict_clc['nodata'] = ## add a nodata value\n",
+ "landuse['landuse_value_clc'] = landuse.landuse.apply(lambda x: value_dict_clc[x])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "24d86032",
+ "metadata": {},
+ "source": [
+ "and now rasterize the OSM landuse data again to make it matching!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 132,
+ "id": "38d46699-d759-48e3-a5ef-09d45ec8e41a",
+ "metadata": {
+ "id": "38d46699-d759-48e3-a5ef-09d45ec8e41a"
+ },
+ "outputs": [],
+ "source": [
+ "landuse_clc = make_geocube(\n",
+ " vector_data= # add landuse,\n",
+ " output_crs= # add CRS,\n",
+ " resolution= # add resolution (this should be somewhat similar as the resolution of the flood map),\n",
+ " categorical_enums={'landuse_value': landuse.landuse_value.drop_duplicates().values.tolist()\n",
+ "}\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "126d4106",
+ "metadata": {},
+ "source": [
+ "And run the **match_rasters** function now with the new landuse map"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 134,
+ "id": "PL1evdBM4rex",
+ "metadata": {
+ "id": "PL1evdBM4rex"
+ },
+ "outputs": [],
+ "source": [
+ "landuse_osm_clc, flood_map_osm = match_rasters(# add landuse_clc,\n",
+ " ## add landuse_clc,\n",
+ " haz_crs=3035,\n",
+ " lu_crs=# add landuse CRS,\n",
+ " resolution= #add desired resolution (I would pick the resolution of the flood map),\n",
+ " hazard_col=['band_data'])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b70655e2",
+ "metadata": {},
+ "source": [
+ "And now turn them into numpy arrays again"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 139,
+ "id": "2VNtw7HW9RXG",
+ "metadata": {
+ "id": "2VNtw7HW9RXG"
+ },
+ "outputs": [],
+ "source": [
+ "osm_landuse_map = landuse_osm_clc['band_data'] #complete function\n",
+ "osm_flood_map = flood_map_osm['band_data'] #complete function"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8398430d",
+ "metadata": {},
+ "source": [
+ "And run the DamageScanner"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "91662ad6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "flood_damage_CLC = DamageScanner(# fill the dammagescanner function)[1]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5626418e",
+ "metadata": {},
+ "source": [
+ "\n",
+ "Question 11: How do the results compare to the flood damages estimated with Corine Land Cover? Are you able to visualise the differences?\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "28e483d0-06b1-432f-86e6-d7a68e9811a4",
+ "metadata": {
+ "id": "28e483d0-06b1-432f-86e6-d7a68e9811a4"
+ },
+ "source": [
+ "### 6. Extracting high-resolution data from OpenStreetMap\n",
+ "" + ] + }, + { + "cell_type": "markdown", + "id": "502b596d-ad9f-4e42-bc2f-e889d216f0e7", + "metadata": { + "id": "502b596d-ad9f-4e42-bc2f-e889d216f0e7" + }, + "source": [ + "#### Extracting buildings from OpenStreetMap\n", + "---" + ] + }, + { + "cell_type": "markdown", + "id": "4f42ba03-9c17-465d-97eb-77df502dd8ee", + "metadata": { + "id": "4f42ba03-9c17-465d-97eb-77df502dd8ee" + }, + "source": [ + "There is a lot more data to extract from OpenStreetMap besides land-use information. Let's extract some building data. To do so, we use the *\"building\"* tag." + ] + }, + { + "cell_type": "code", + "execution_count": 60, + "id": "92f86ddc-ce27-4753-9e70-af6df03d7850", + "metadata": { + "id": "92f86ddc-ce27-4753-9e70-af6df03d7850" + }, + "outputs": [], + "source": [ + "tags = {\"building\": True}\n", + "buildings = ox.features_from_place(place_name, tags)" + ] + }, + { + "cell_type": "markdown", + "id": "772d95a6-c2ce-48a6-a7d0-bc1f4b77c323", + "metadata": { + "id": "772d95a6-c2ce-48a6-a7d0-bc1f4b77c323" + }, + "source": [ + "In case the above does not work, you can continue the assignment by using the code below (make sure you remove the hashtags to run it). If you decide to use the data as specified below, also change the map at the start to 'Kampen'." + ] + }, + { + "cell_type": "code", + "execution_count": 61, + "id": "9a7e508a-b20a-416c-8f2a-c8dd36a60389", + "metadata": { + "id": "9a7e508a-b20a-416c-8f2a-c8dd36a60389" + }, + "outputs": [], + "source": [ + "# remote_url = https://github.com/VU-IVM/UNIGIS_ProgrammingGIS/tree/main/TAA2'\n", + "# file = 'kampen_buildings.gpkg'\n", + "#\n", + "# #request.urlretrieve(remote_url, file)\n", + "# buildings = gpd.GeoDataFrame.from_file('kampen_buildings.gpkg')" + ] + }, + { + "cell_type": "markdown", + "id": "53e45bdb-36a3-4efd-b0e4-111f5fd858fe", + "metadata": { + "id": "53e45bdb-36a3-4efd-b0e4-111f5fd858fe" + }, + "source": [ + "Now let's see what information is actually extracted:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "26a60806-c927-4d0c-ab04-dc07c9c7a688", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 460 + }, + "id": "26a60806-c927-4d0c-ab04-dc07c9c7a688", + "outputId": "c4e83f03-b5cf-4421-836b-4262a804f6d1" + }, + "outputs": [], + "source": [ + "buildings.head()" + ] + }, + { + "cell_type": "markdown", + "id": "2f7f7ff1-ed38-4dca-b2ad-f09472566879", + "metadata": { + "id": "2f7f7ff1-ed38-4dca-b2ad-f09472566879" + }, + "source": [ + "As you notice in the output of the cell above, there are many columns which just contain \"NaN\". And there even seem to be to many columns to even visualize properly in one view.\n", + "\n", + "Let's check what information is collected for the different buildings:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b78b1253-16b0-4513-8903-966b1816715f", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "b78b1253-16b0-4513-8903-966b1816715f", + "outputId": "b00fd26f-4ab5-4acb-8e8a-df34bc2b9fc3" + }, + "outputs": [], + "source": [ + "buildings.columns" + ] + }, + { + "cell_type": "markdown", + "id": "772be8a9-7ab6-4996-9311-e165fcc6c8ed", + "metadata": { + "id": "772be8a9-7ab6-4996-9311-e165fcc6c8ed" + }, + "source": [ + "
\n",
+ "Question 12: Let's have a look at the extracted building information. Please describe in your own words the information it contains. Is there specific information that suprises you to see, and do you think anything is missing that you expected?\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d7077034-4cfe-436a-b5b2-64fc77f24ae5",
+ "metadata": {
+ "id": "d7077034-4cfe-436a-b5b2-64fc77f24ae5"
+ },
+ "source": [
+ "One interesting column is called `start_date`. This shows the building year per building.\n",
+ "\n",
+ "Let's explore this year of building a bit more.\n",
+ "\n",
+ "First, it would be interesting to get an idea how many buildings are build in each year through using the `value_counts()` function. Normally, that functions ranks the values in descending order (high to low). We are more interested in how this has developed over time. So we use the `sort_index()` function to sort the values by year. Add these two functions in the cell below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 64,
+ "id": "e27c9a71-68b4-4adc-a1b4-52ff94275a6f",
+ "metadata": {
+ "id": "e27c9a71-68b4-4adc-a1b4-52ff94275a6f"
+ },
+ "outputs": [],
+ "source": [
+ "building_year = buildings. #complete function"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e9e9c436-fd01-46f7-8f73-44eccafe7816",
+ "metadata": {
+ "id": "e9e9c436-fd01-46f7-8f73-44eccafe7816"
+ },
+ "source": [
+ "There is not better way to further explore this years than through plotting it. Don't forget to add things such as a x label, y label and title. Have a look at some of the matplotlib [tutorials](https://matplotlib.org/stable/tutorials/introductory/quick_start.html). Note that you need to look at the code that also uses subplots and where they use the `ax` option. Perhaps also consider setting a threshold to make the figure more easy to visualise?"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "35a6ee4b-f664-4955-9781-86d56920b3e4",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1000
+ },
+ "id": "35a6ee4b-f664-4955-9781-86d56920b3e4",
+ "outputId": "f51f5e22-aeda-4ec6-c0d4-7be7b6dbdd42"
+ },
+ "outputs": [],
+ "source": [
+ "fig,ax = plt.subplots(1,1,figsize=(5,18))\n",
+ "\n",
+ "building_year.plot(kind='barh',ax=ax)\n",
+ "\n",
+ "ax.tick_params(axis='y', which='major', labelsize=7)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7e5ce5b6-f7b6-47a4-b4b7-2cac7cc933bf",
+ "metadata": {
+ "id": "7e5ce5b6-f7b6-47a4-b4b7-2cac7cc933bf"
+ },
+ "source": [
+ "\n",
+ "Question 13: Please upload a figure that shows the development of building stock over the years in your region of interest. Make sure it contains all the necessary elements (labels on the axis, title, etc.)\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "089f6011-87fd-464a-ad8b-52ad810d8b75",
+ "metadata": {
+ "id": "089f6011-87fd-464a-ad8b-52ad810d8b75"
+ },
+ "source": [
+ "What we also noticed is that quite some buildings are identified as 'yes'. This is not very useful as it does not really say much about the use of the building.\n",
+ "\n",
+ "Let's see for how many buildings this is the case. Use the .value_counts function:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ca163a97-54bd-4050-bea6-2721d024a1eb",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 993
+ },
+ "id": "ca163a97-54bd-4050-bea6-2721d024a1eb",
+ "outputId": "94a378d4-bd6a-4efb-94e6-076ed1f1afca"
+ },
+ "outputs": [],
+ "source": [
+ "buildings.building. #complete function"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "502fcb4a-cc93-46fa-bc1a-5dad7f4438bc",
+ "metadata": {
+ "id": "502fcb4a-cc93-46fa-bc1a-5dad7f4438bc"
+ },
+ "source": [
+ "As you have seen from the `value_counts` function, there are quite a few buildings with only very few tags. You could either consider to not include them in your plot at all (for example by using the `isin` function or the `query` function, see also [here](https://stackoverflow.com/questions/12096252/use-a-list-of-values-to-select-rows-from-a-pandas-dataframe)), or rename them, similar to how you named the natural land cover classes for the land-use map. Here, we filter the dataframe to include only specific classes of buildings."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 67,
+ "id": "6ffd00f1-e13c-4c60-885e-d7fb6ebe895b",
+ "metadata": {
+ "id": "6ffd00f1-e13c-4c60-885e-d7fb6ebe895b"
+ },
+ "outputs": [],
+ "source": [
+ "buildings = buildings[buildings['building'].isin(['yes', 'house', 'industrial', 'apartment'])]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "22598d8d-a73c-41b7-befd-f8abafc86162",
+ "metadata": {
+ "id": "22598d8d-a73c-41b7-befd-f8abafc86162"
+ },
+ "source": [
+ "Now let's visualize the buildings again. We need to create a similar color dictionary as we did for the land-use categories. Now its up to you to make it!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 68,
+ "id": "739a2389-8302-4e4f-8624-2beb0225ecfd",
+ "metadata": {
+ "id": "739a2389-8302-4e4f-8624-2beb0225ecfd"
+ },
+ "outputs": [],
+ "source": [
+ "color_dict = { 'yes' : \"#f1134b\",\n",
+ " 'house':'#f13013',\n",
+ " 'industrial':'#0f045c',\n",
+ " 'apartment':'#fcfcb9' }"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "54672469-3961-4b67-860e-17f9320d1a62",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "54672469-3961-4b67-860e-17f9320d1a62",
+ "outputId": "c172f78b-7643-4c4a-f0c6-655dcc2da355"
+ },
+ "outputs": [],
+ "source": [
+ "map_dict = dict(zip(color_dict.keys(),[x for x in range(len(color_dict))]))\n",
+ "buildings['col_landuse'] =buildings.building.apply(lambda x: color_dict[x])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0bbaef6d-6b28-45f4-ace9-6a5c9dec0f2f",
+ "metadata": {
+ "id": "0bbaef6d-6b28-45f4-ace9-6a5c9dec0f2f"
+ },
+ "source": [
+ "And plot the figure in the same manner!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "7dad1b9f-855a-468e-9090-25d7ace2990d",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 847
+ },
+ "id": "7dad1b9f-855a-468e-9090-25d7ace2990d",
+ "outputId": "68968e35-8949-4abd-c593-2ef1254d81d8"
+ },
+ "outputs": [],
+ "source": [
+ "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n",
+ "\n",
+ "# add color scheme\n",
+ "color_scheme_map = list(color_dict.values())\n",
+ "cmap = LinearSegmentedColormap.from_list(name='landuse',\n",
+ " colors=color_scheme_map)\n",
+ "\n",
+ "# and plot the land-use map.\n",
+ "buildings.plot(color=buildings['col_landuse'],ax=ax,linewidth=0)\n",
+ "\n",
+ "# remove the ax labels\n",
+ "ax.set_xticks([])\n",
+ "ax.set_yticks([])\n",
+ "ax.set_axis_off()\n",
+ "\n",
+ "# add a legend:\n",
+ "legend_elements = []\n",
+ "for iter_,item in enumerate(color_dict):\n",
+ " legend_elements.append(Patch(facecolor=color_scheme_map[iter_],label=item))\n",
+ "\n",
+ "ax.legend(handles=legend_elements,edgecolor='black',facecolor='#fefdfd',prop={'size':12},loc=(1.02,0.2))\n",
+ "\n",
+ "# add a title\n",
+ "ax.set_title(place_name,fontweight='bold')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7fc1437d-8deb-4569-a3ee-9270676f0e41",
+ "metadata": {
+ "id": "7fc1437d-8deb-4569-a3ee-9270676f0e41"
+ },
+ "source": [
+ "\n",
+ "Question 14: Please upload a figure of your building stock map of your region of interest. Make sure that the interpretation is clear. If necessary, merge multiple categories into one (i.e., when some categories only contain 1 or 2 buildings).\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "508022f5-46d7-4c69-91d2-46711d6a514c",
+ "metadata": {
+ "id": "508022f5-46d7-4c69-91d2-46711d6a514c"
+ },
+ "source": [
+ "### 7. Perform a damage assessment of the road network using OpenStreetMap\n",
+ "" + ] + }, + { + "cell_type": "markdown", + "id": "b8d7623d-40ee-4262-9781-0c2ed3ea1889", + "metadata": { + "id": "b8d7623d-40ee-4262-9781-0c2ed3ea1889" + }, + "source": [ + "Generally, wind damage does not cause much damage to roads. There will be clean-up cost of the trees that will fall on the roads, but structural damage is rare. As such, we will only do a flood damage assessment for the road network of our region.\n", + "\n", + "To do so, we first need to extract the roads again. We will use the `graph_from_place()` function again to do so. However, the area will be to large to extract roads, so we will focus our analysis on the main network." + ] + }, + { + "cell_type": "code", + "execution_count": 71, + "id": "abacf8e1-b27c-4156-8521-87f38004ac2e", + "metadata": { + "id": "abacf8e1-b27c-4156-8521-87f38004ac2e" + }, + "outputs": [], + "source": [ + "cf = '[\"highway\"~\"trunk|motorway|primary|secondary\"]'\n", + "G = ox.graph_from_place(place_name, network_type=\"drive\", custom_filter=cf)" + ] + }, + { + "cell_type": "markdown", + "id": "87de9a96-e3fc-4ab5-b090-be6412a0d0a1", + "metadata": { + "id": "87de9a96-e3fc-4ab5-b090-be6412a0d0a1" + }, + "source": [ + "Now we convert the road network to a `geodataframe`." + ] + }, + { + "cell_type": "code", + "execution_count": 72, + "id": "90acac15-3b1e-4c0a-b0a4-6033d243f9ea", + "metadata": { + "id": "90acac15-3b1e-4c0a-b0a4-6033d243f9ea" + }, + "outputs": [], + "source": [ + "roads = gpd.GeoDataFrame(nx.to_pandas_edgelist(G))\n", + "roads.highway = roads.highway.astype('str')" + ] + }, + { + "cell_type": "markdown", + "id": "56f95cc0-2611-472f-92fa-c256f368c33c", + "metadata": { + "id": "56f95cc0-2611-472f-92fa-c256f368c33c" + }, + "source": [ + "In case the above does not work, you can continue the assignment by using the code below (make sure you remove the hashtags to run it)." + ] + }, + { + "cell_type": "code", + "execution_count": 73, + "id": "55752d12-81de-4eb5-b849-90473f88478a", + "metadata": { + "id": "55752d12-81de-4eb5-b849-90473f88478a" + }, + "outputs": [], + "source": [ + "#from urllib import request\n", + "# remote_url = 'https://github.com/VU-IVM/UNIGIS_ProgrammingGIS/tree/main/TAA2'\n", + "# file = 'kampen_roads.gpkg'\n", + "#\n", + "# #request.urlretrieve(remote_url, file)\n", + "# roads = gpd.GeoDataFrame.from_file('kampen_roads.gpkg')" + ] + }, + { + "cell_type": "markdown", + "id": "73f4a201-87b1-4cd8-a206-6b965c9b4107", + "metadata": { + "id": "73f4a201-87b1-4cd8-a206-6b965c9b4107" + }, + "source": [ + "And lets have a look at the data:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "91a864f8-6af7-454f-869c-197e5e47151b", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 807 + }, + "id": "91a864f8-6af7-454f-869c-197e5e47151b", + "outputId": "01bfca83-8de6-4d76-90df-4702886e66fb" + }, + "outputs": [], + "source": [ + "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n", + "\n", + "\n", + "roads.plot(column='highway',legend=True,ax=ax,legend_kwds={'loc': 'lower right'});\n", + "\n", + "\n", + "# remove the ax labels\n", + "ax.set_xticks([])\n", + "ax.set_yticks([])\n", + "ax.set_axis_off()" + ] + }, + { + "cell_type": "markdown", + "id": "4789ae59-e6c9-4a1c-a1ad-44e94c9172ca", + "metadata": { + "id": "4789ae59-e6c9-4a1c-a1ad-44e94c9172ca" + }, + "source": [ + "Dependening on the region you have selected, you may have lists in your data. It is actually quite inconvenient to have all these lists in the data for when we want to do the damage assessment. Let's clean this up a bit. To do so, we first make sure that all the lists are represented as actual lists, and not lists wrapped within a string." + ] + }, + { + "cell_type": "code", + "execution_count": 75, + "id": "51c4746a-6efd-4b59-8b6b-5fe9a8c42372", + "metadata": { + "id": "51c4746a-6efd-4b59-8b6b-5fe9a8c42372" + }, + "outputs": [], + "source": [ + "roads.highway = roads.highway.apply(lambda x: x.strip('][').split(', '))" + ] + }, + { + "cell_type": "markdown", + "id": "eb311b10-0a1c-4e0a-ad1f-cef37aa40ad7", + "metadata": { + "id": "eb311b10-0a1c-4e0a-ad1f-cef37aa40ad7" + }, + "source": [ + "Now we just need to grab the first element of each of the lists." + ] + }, + { + "cell_type": "code", + "execution_count": 76, + "id": "9e449847-c279-479e-bf3b-382b8c50e018", + "metadata": { + "id": "9e449847-c279-479e-bf3b-382b8c50e018" + }, + "outputs": [], + "source": [ + "roads.highway = roads.highway.apply(lambda x: x[0] if isinstance(x, list) else x)\n", + "roads.highway = roads.highway.str.replace(\"'\",\"\")" + ] + }, + { + "cell_type": "markdown", + "id": "5b3f95ad-9a1f-43e1-b098-45255c9d5dd2", + "metadata": { + "id": "5b3f95ad-9a1f-43e1-b098-45255c9d5dd2" + }, + "source": [ + "And let's have a look whether this worked:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "21dc3a62-c266-4d0e-bd03-ad1d1b4b22e8", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 807 + }, + "id": "21dc3a62-c266-4d0e-bd03-ad1d1b4b22e8", + "outputId": "0361fc7e-ad9c-40f4-a767-aac6ac1d84cc" + }, + "outputs": [], + "source": [ + "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n", + "\n", + "roads.plot(column='highway',legend=True,ax=ax,legend_kwds={'loc': 'upper left','ncol':1});\n", + "\n", + "# remove the ax labels\n", + "ax.set_xticks([])\n", + "ax.set_yticks([])\n", + "ax.set_axis_off()" + ] + }, + { + "cell_type": "markdown", + "id": "d784f6dc-7091-4e60-b7df-58682efb66ef", + "metadata": { + "id": "d784f6dc-7091-4e60-b7df-58682efb66ef" + }, + "source": [ + "Nice! now let's start with the damage calculation. As you already have may have noticed, our data is now not stored in raster format, but in vector format. One way to deal with this issue is to convert our vector data to raster data, but we will lose a lot of information and detail. As such, we will perform the damage assessment on the road elements, using the xarray flood map.\n", + "\n", + "Let's start with preparing the flood data into vector format:" + ] + }, + { + "cell_type": "code", + "execution_count": 78, + "id": "af68dc43-fd65-416c-bc7a-a1b1231187d3", + "metadata": { + "id": "af68dc43-fd65-416c-bc7a-a1b1231187d3" + }, + "outputs": [], + "source": [ + "# get the mean values\n", + "flood_map_vector = flood_map_area['band_data'].to_dataframe().reset_index()\n", + "\n", + "# create geometry values and drop lat lon columns\n", + "flood_map_vector['geometry'] = [shapely.points(x) for x in list(zip(flood_map_vector['x'],flood_map_vector['y']))]\n", + "flood_map_vector = flood_map_vector.drop(['x','y','band','spatial_ref'],axis=1)\n", + "\n", + "# drop all non values to reduce size\n", + "flood_map_vector = flood_map_vector.loc[~flood_map_vector['band_data'].isna()].reset_index(drop=True)\n", + "\n", + "# and turn them into squares again:\n", + "flood_map_vector.geometry= shapely.buffer(flood_map_vector.geometry,distance=100/2,cap_style='square').values" + ] + }, + { + "cell_type": "markdown", + "id": "9894f1ba-e637-4ede-b46c-5aa4747eaea0", + "metadata": { + "id": "9894f1ba-e637-4ede-b46c-5aa4747eaea0" + }, + "source": [ + "And let's plot the results:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5f6ab83a-62af-4142-bb26-157e077d868b", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 481 + }, + "id": "5f6ab83a-62af-4142-bb26-157e077d868b", + "outputId": "adbb0e5e-7487-4890-f767-3bbd11ded3ae" + }, + "outputs": [], + "source": [ + "gpd.GeoDataFrame(flood_map_vector.copy()).plot(column='band_data',cmap='Blues',vmax=5,linewidth=0)" + ] + }, + { + "cell_type": "markdown", + "id": "1cdbc0d8-beba-4b37-be49-f5226953f19b", + "metadata": { + "id": "1cdbc0d8-beba-4b37-be49-f5226953f19b" + }, + "source": [ + "We will need a bunch of functions to make sure we can do our calculations. They are specified below." + ] + }, + { + "cell_type": "code", + "execution_count": 80, + "id": "d470c747-42aa-4f20-bbf3-90773c3d4cc8", + "metadata": { + "id": "d470c747-42aa-4f20-bbf3-90773c3d4cc8" + }, + "outputs": [], + "source": [ + "def reproject(df_ds, current_crs=\"epsg:4326\", approximate_crs=\"epsg:3035\"):\n", + " \"\"\"\n", + " Reproject geometries in a DataFrame from one coordinate reference system (CRS) to another.\n", + "\n", + " Parameters:\n", + " df_ds : pandas.DataFrame\n", + " A DataFrame containing a 'geometry' column with geometries to reproject.\n", + " current_crs : str or pyproj.CRS, optional\n", + " The current coordinate reference system of the geometries. Default is \"epsg:4326\".\n", + " approximate_crs : str or pyproj.CRS, optional\n", + " The target coordinate reference system to reproject the geometries into. Default is \"epsg:3035\".\n", + "\n", + " Returns:\n", + " shapely.GeometryArray\n", + " A Shapely GeometryArray containing the reprojected geometries.\n", + " \"\"\"\n", + " geometries = df_ds['geometry']\n", + " coords = shapely.get_coordinates(geometries)\n", + " transformer = pyproj.Transformer.from_crs(current_crs, approximate_crs, always_xy=True)\n", + " new_coords = transformer.transform(coords[:, 0], coords[:, 1])\n", + "\n", + " return shapely.set_coordinates(geometries.copy(), np.array(new_coords).T)\n", + "\n", + "\n", + "def buffer_assets(assets, buffer_size=100):\n", + " \"\"\"\n", + " Create a buffer around each geometry in the assets DataFrame.\n", + "\n", + " Parameters:\n", + " assets : pandas.DataFrame\n", + " A DataFrame containing a 'geometry' column with geometries to buffer.\n", + " buffer_size : float, optional\n", + " The distance to buffer around each geometry. Default is 100 units.\n", + "\n", + " Returns:\n", + " pandas.DataFrame\n", + " The input DataFrame with an additional 'buffered' column containing the buffered geometries.\n", + " \"\"\"\n", + " assets['buffered'] = shapely.buffer(assets.geometry.values, buffer_size)\n", + " return assets\n", + "\n", + "\n", + "def overlay_hazard_assets(df_ds, assets):\n", + " \"\"\"\n", + " Find the indices of hazards that overlay or intersect with assets.\n", + "\n", + " Parameters:\n", + " df_ds : pandas.DataFrame\n", + " A DataFrame containing hazard geometries in a 'geometry' column.\n", + " assets : pandas.DataFrame\n", + " A DataFrame containing asset geometries in a 'geometry' column.\n", + "\n", + " Returns:\n", + " numpy.ndarray\n", + " An array of indices of hazards that intersect with the assets.\n", + " \"\"\"\n", + " # Build a spatial index for the hazard geometries\n", + " hazard_tree = shapely.STRtree(df_ds.geometry.values)\n", + " # Determine geometry type of the first asset\n", + " asset_geom_type = shapely.get_type_id(assets.iloc[0].geometry)\n", + " # If the asset geometry is a polygon or multipolygon\n", + " if (asset_geom_type == 3) or (asset_geom_type == 6):\n", + " return hazard_tree.query(assets.geometry, predicate='intersects')\n", + " else:\n", + " # If the asset geometry is not polygon/multipolygon, use buffered geometries\n", + " return hazard_tree.query(assets.buffered, predicate='intersects')\n", + "\n", + "\n", + "def get_damage_per_asset(asset, df_ds, assets):\n", + " \"\"\"\n", + " Calculate the total damage for a single asset based on overlapping hazards.\n", + "\n", + " Parameters:\n", + " asset : tuple\n", + " A tuple containing the asset index and a DataFrame with hazard points intersecting the asset.\n", + " df_ds : pandas.DataFrame\n", + " A DataFrame containing hazard data with a 'geometry' column.\n", + " assets : pandas.DataFrame\n", + " A DataFrame containing asset data with a 'geometry' column.\n", + "\n", + " Returns:\n", + " tuple\n", + " A tuple containing the asset index and the calculated damage.\n", + " \"\"\"\n", + " # Find the exact hazard overlays\n", + " get_hazard_points = df_ds.iloc[asset[1]['hazard_point'].values].reset_index()\n", + " # Select hazard points that intersect with the asset geometry\n", + " asset_geom = assets.iloc[asset[0]].geometry\n", + " get_hazard_points = get_hazard_points.loc[shapely.intersects(get_hazard_points.geometry.values, asset_geom)]\n", + "\n", + " # Parameters for damage calculation\n", + " maxdam_asset = 100 # Maximum damage per asset\n", + " hazard_intensity = np.arange(0, 10, 0.1) # Hazard intensity levels\n", + " fragility_values = np.arange(0, 1, 0.01) # Fragility values corresponding to hazard intensity\n", + "\n", + " if len(get_hazard_points) == 0:\n", + " return asset[0], 0\n", + " else:\n", + " # Calculate the length of the intersection between hazard and asset geometries\n", + " get_hazard_points['overlay_meters'] = shapely.length(\n", + " shapely.intersection(get_hazard_points.geometry.values, asset_geom))\n", + " # Interpolate fragility values based on hazard intensity\n", + " damage = np.sum(\n", + " (np.interp(get_hazard_points.band_data.values, hazard_intensity, fragility_values))\n", + " * get_hazard_points.overlay_meters * maxdam_asset)\n", + " return asset[0], damage" + ] + }, + { + "cell_type": "markdown", + "id": "622a243c-dd99-44c1-975c-7cb4eab99359", + "metadata": { + "id": "622a243c-dd99-44c1-975c-7cb4eab99359" + }, + "source": [ + "Now we need to make sure that the road data is the same coordinate system." + ] + }, + { + "cell_type": "code", + "execution_count": 81, + "id": "484c68a5-90ae-4394-9e4f-4ed5ba0c665d", + "metadata": { + "id": "484c68a5-90ae-4394-9e4f-4ed5ba0c665d" + }, + "outputs": [], + "source": [ + "roads.geometry = reproject(roads)" + ] + }, + { + "cell_type": "markdown", + "id": "efebb735-7955-4a75-bedf-1b213067fe20", + "metadata": { + "id": "efebb735-7955-4a75-bedf-1b213067fe20" + }, + "source": [ + "And we can now overlay the roads with the flood data" + ] + }, + { + "cell_type": "code", + "execution_count": 82, + "id": "60de298f-9e06-40c3-b3a2-eecf2aa7205d", + "metadata": { + "id": "60de298f-9e06-40c3-b3a2-eecf2aa7205d" + }, + "outputs": [], + "source": [ + "overlay_roads = pd.DataFrame(overlay_hazard_assets(flood_map_vector,buffer_assets(roads)).T,columns=['asset','hazard_point'])" + ] + }, + { + "cell_type": "markdown", + "id": "53aeaec5-4ea3-48a5-93b5-109a1578c77e", + "metadata": { + "id": "53aeaec5-4ea3-48a5-93b5-109a1578c77e" + }, + "source": [ + "And estimate the damages" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "daa56503-90e8-4b7f-a5dc-82a4b235e95b", + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "daa56503-90e8-4b7f-a5dc-82a4b235e95b", + "outputId": "059ff95d-6e35-4a68-ebcf-31ee13910468" + }, + "outputs": [], + "source": [ + "collect_output = []\n", + "for asset in tqdm(overlay_roads.groupby('asset'),total=len(overlay_roads.asset.unique()),\n", + " desc='polyline damage calculation for'):\n", + " collect_output.append(get_damage_per_asset(asset,flood_map_vector,roads))\n", + "\n", + "damaged_roads = roads.merge(pd.DataFrame(collect_output,columns=['index','damage']),\n", + " left_index=True,right_on='index')[['highway','geometry','damage']]" + ] + }, + { + "cell_type": "markdown", + "id": "580663ca-b83f-4724-a699-b6a4790678ad", + "metadata": { + "id": "580663ca-b83f-4724-a699-b6a4790678ad" + }, + "source": [ + "
\n",
+ "Question 15: Describe the various steps we have taken to perform the damage assessment on the road network. How is this approach different compared to the raster-based approach? Highlight the differences you find most important. Include any line of code you may want to include to make your story clear.\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2e42c7f3-c868-400c-858e-7ac0c848fd4c",
+ "metadata": {
+ "id": "2e42c7f3-c868-400c-858e-7ac0c848fd4c"
+ },
+ "source": [
+ "And let's plot the results"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ef910b4e-201e-444b-92b6-826be82a5164",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 864
+ },
+ "id": "ef910b4e-201e-444b-92b6-826be82a5164",
+ "outputId": "eead7d6c-01ea-4851-d8d0-b756fea0798e"
+ },
+ "outputs": [],
+ "source": [
+ "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n",
+ "\n",
+ "damaged_roads.plot(column='damage',cmap='Reds',ax=ax)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 84,
+ "id": "fe92fb12-9049-4e15-8992-b3d0cbd459b5",
+ "metadata": {
+ "id": "fe92fb12-9049-4e15-8992-b3d0cbd459b5"
+ },
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "colab": {
+ "provenance": []
+ },
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.12.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/TAA2/tutorial.ipynb b/TAA2/tutorial.ipynb
index ffe4bc0..da93bba 100644
--- a/TAA2/tutorial.ipynb
+++ b/TAA2/tutorial.ipynb
@@ -3,7 +3,9 @@
{
"cell_type": "markdown",
"id": "cbd68ab3-f04b-434f-a956-e730535c9ee5",
- "metadata": {},
+ "metadata": {
+ "id": "cbd68ab3-f04b-434f-a956-e730535c9ee5"
+ },
"source": [
"## TAA2: Natural Hazard Risk Assessment using Open Data"
]
@@ -11,26 +13,32 @@
{
"cell_type": "markdown",
"id": "4c8cc742-23a7-4895-8d2a-7e826d8a23b8",
- "metadata": {},
+ "metadata": {
+ "id": "4c8cc742-23a7-4895-8d2a-7e826d8a23b8"
+ },
"source": [
"Within this tutorial, we are going to use publicly available hazard data and exposure data to do a risk assessment for the Netherlands. More specifically, we will look at damage due to wind storms and flooding. We will use both Copernicus Land Cover data and OpenStreetMap to estimate the potential damage of natural hazards to the built environment.\n",
- " \n",
- "We will first download, access and explore hazard data retrieved from the Copernicus Climate Data Copernicus Store and the European Commission Joint Research Centre. After this, we will learn how to download and access Copernicus Land Cover data. We will also explore the power of OpenStreetMap that provides vector data. We will learn how to extract information from OpenStreetMap, how you can explore and visualize this. Lastly, we will use Copernicus Land Cover data to estimate the damage to specific land-uses, whereas we will use OpenStreetMap to assess the potential damage to the road system."
+ "\n",
+ "We will first download, access and explore hazard data retrieved from the Copernicus Climate Data Copernicus Store and the European Commission Joint Research Centre. We will also explore the power of OpenStreetMap that provides vector data. We will learn how to extract information from OpenStreetMap, how you can explore and visualize this. Lastly, we will use Copernicus Land Cover data to estimate the damage to specific land-uses, whereas we will use OpenStreetMap to assess the potential damage to the road system."
]
},
{
"cell_type": "markdown",
"id": "4355fca9-3e21-4556-a5d0-3e1577c68643",
- "metadata": {},
+ "metadata": {
+ "id": "4355fca9-3e21-4556-a5d0-3e1577c68643"
+ },
"source": [
- "## Learning Objectives\n",
+ "### Learning Objectives\n",
"" ] }, { "cell_type": "markdown", "id": "04dd18de-459a-4cb4-891e-cc3c98e76e7a", - "metadata": {}, + "metadata": { + "id": "04dd18de-459a-4cb4-891e-cc3c98e76e7a" + }, "source": [ "- To understand the use of **OSMnx** to extract geospatial data from OpenStreetmap.\n", "- To know how to download data from the Copernicus Climate Data Store using the `cdsapi` and access it through Python.\n", @@ -49,18 +57,22 @@ { "cell_type": "markdown", "id": "88a7cd45-4394-44fb-ba1b-e52464223d42", - "metadata": {}, + "metadata": { + "id": "88a7cd45-4394-44fb-ba1b-e52464223d42" + }, "source": [ - "## 1. Introducing the packages\n", + "### 1. Introducing the packages\n", "
" ] }, { "cell_type": "markdown", "id": "961d01ad-b5d4-4486-9d00-04fd00142a19", - "metadata": {}, + "metadata": { + "id": "961d01ad-b5d4-4486-9d00-04fd00142a19" + }, "source": [ - "Within this tutorial, we are going to make use of the following packages: \n", + "Within this tutorial, we are going to make use of the following packages:\n", "\n", "[**GeoPandas**](https://geopandas.org/) is a Python package that extends the datatypes used by pandas to allow spatial operations on geometric types.\n", "\n", @@ -79,14 +91,15 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "id": "b1d683ee-d2b5-49a5-9b31-9ab50039f428", - "metadata": {}, + "metadata": { + "id": "b1d683ee-d2b5-49a5-9b31-9ab50039f428" + }, "outputs": [], "source": [ - "import os\n", "import cdsapi\n", - "import shapely \n", + "import shapely\n", "import matplotlib\n", "import urllib3\n", "import pyproj\n", @@ -99,8 +112,11 @@ "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import networkx as nx\n", + "import cdsapi\n", "\n", - "from matplotlib.colors import ListedColormap\n", + "from matplotlib.colors import LinearSegmentedColormap,ListedColormap\n", + "from matplotlib.patches import Patch\n", + "from geocube.api.core import make_geocube\n", "from zipfile import ZipFile\n", "from io import BytesIO\n", "from urllib.request import urlopen\n", @@ -113,7 +129,9 @@ { "cell_type": "markdown", "id": "d3208479-d07e-4d6d-afd7-a9944b9630c0", - "metadata": {}, + "metadata": { + "id": "d3208479-d07e-4d6d-afd7-a9944b9630c0" + }, "source": [ "Import error? Not all of the packages were installed already. Make sure to install the missing packages using pip install in the cell below and then run the cell above again:" ] @@ -122,57 +140,73 @@ "cell_type": "code", "execution_count": null, "id": "d395181b-53d5-48de-84ff-55e27da494a2", - "metadata": {}, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "d395181b-53d5-48de-84ff-55e27da494a2", + "outputId": "b5418cb6-c160-4ec0-ed2a-f539eb9d5568" + }, "outputs": [], "source": [ - " # provide code to pip install missing packages" + " !pip install cdsapi\n", + " !pip install geocube\n", + " !pip install contextily\n", + " !pip install --pre osmnx\n", + " !pip install 'cdsapi>=0.7.0'" ] }, { "cell_type": "markdown", "id": "981898d2-5f8f-4990-a236-b1cfe2c7008e", - "metadata": {}, + "metadata": { + "id": "981898d2-5f8f-4990-a236-b1cfe2c7008e" + }, "source": [ - "## 2. Downloading and accessing natural hazard data\n", + "### 2. Downloading and accessing natural hazard data\n", "
" ] }, { "cell_type": "markdown", "id": "d7693fcc-a0cf-4ed4-a0d0-e3a00b917547", - "metadata": {}, + "metadata": { + "id": "d7693fcc-a0cf-4ed4-a0d0-e3a00b917547" + }, "source": [ - "We will first download and explore windstorm and flood data for the Netherlands. \n", + "We will first download and explore windstorm and flood data for the Netherlands.\n", "\n", - "### Windstorm Data\n", + "#### Windstorm Data\n", "
\n", "\n", - "The windstorm data will be downloaded from the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/). As we have seen during the lecture, and as you can also see by browsing on this website, there is an awful lot of climate data available through this Data Store. As such, it is very valuable to understand how to access and download this information to use within an analysis. To keep things simple, we only download one dataset today: [A winter windstorm](https://cds.climate.copernicus.eu/cdsapp#!/dataset/sis-european-wind-storm-indicators?tab=overview). \n", + "The windstorm data will be downloaded from the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/). As we have seen during the lecture, and as you can also see by browsing on this website, there is an awful lot of climate data available through this Data Store. As such, it is very valuable to understand how to access and download this information to use within an analysis. To keep things simple, we only download one dataset today: [A winter windstorm](https://cds.climate.copernicus.eu/cdsapp#!/dataset/sis-european-wind-storm-indicators?tab=overview).\n", "\n", "We will do so using an **API**, which is the acronym for application programming interface. It is a software intermediary that allows two applications to talk to each other. APIs are an accessible way to extract and share data within and across organizations. APIs are all around us. Every time you use a rideshare app, send a mobile payment, or change the thermostat temperature from your phone, you’re using an API.\n", "\n", - "However, before we can access this **API**, we need to take a few steps. Most importantly, we need to register ourselves on the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/) portal. To do so, we need to register, as explained in the video clip below:\n", + "However, before we can access this **API**, we need to take a few steps which can be found on the [CDSAPI setup webpage of the Copernicus Climate Data Store](https://cds-beta.climate.copernicus.eu/how-to-api/). The first step is to register yourself on the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/) portal. \n", "\n", - "
![](\"https://github.com/ElcoK/BigData_AED/blob/main/_static/images/CDS_registration.gif?raw=1\")
\n",
- "\n", + "Now, the next step is to request access to the dataset. As you can see in the cell below, we download a specific windstorm that has occured on the 28th of October in 2013. This is storm [Carmen (also called St Jude)](https://en.wikipedia.org/wiki/St._Jude_storm). To download the relevant windstorm data, fill out the associated [dataset form](https://cds-beta.climate.copernicus.eu/datasets/sis-european-wind-storm-indicators?tab=download) and make sure to **agree to the Terms of Use**. \n", "\n", - "Now, the next step is to access the API. You can now login on the website of the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/). After you login, you can click on your name in the top right corner of the webpage (next to the login button). On the personal page that has just opened, you will find your user ID (**uid**) and your personal **API**. You need to add those in the cell below to be able to download the windstorm.\n", - "\n", - "As you can see in the cell below, we download a specific windstorm that has occured on the 28th of October in 2013. This is storm [Carmen (also called St Jude)](https://en.wikipedia.org/wiki/St._Jude_storm). " + "The last step is to access the API. You can now login on the website of the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/). After you login, you can click on your name in the top right corner of the webpage (next to the login button). On the personal page that has just opened, you will find your personal access token **API**. You need to add this in the cell below to be able to download the windstorm." ] }, { "cell_type": "code", "execution_count": null, "id": "4736833f-c0ec-48f4-8c29-1721ba2ecb7d", - "metadata": {}, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "4736833f-c0ec-48f4-8c29-1721ba2ecb7d", + "outputId": "2cd3a582-faa3-4293-be54-ff38771e4605" + }, "outputs": [], "source": [ - "uid = XXX\n", - "apikey = 'XXX'\n", - "\n", - "c = cdsapi.Client(key=f\"{uid}:{apikey}\", url=\"https://cds.climate.copernicus.eu/api/v2\")\n", - "\n", + "apikey = '' #add your personal API\n", + " \n", + "c = cdsapi.Client(key=f\"{apikey}\", url=\"https://cds-beta.climate.copernicus.eu/api\")\n", + " \n", "c.retrieve(\n", " 'sis-european-wind-storm-indicators',\n", " {\n", @@ -189,26 +223,33 @@ { "cell_type": "markdown", "id": "0ec192f1-8ac1-4e61-ac8f-c01d496e157f", - "metadata": {}, + "metadata": { + "id": "0ec192f1-8ac1-4e61-ac8f-c01d496e157f" + }, "source": [ - "### Flood Data\n", + "#### Flood Data\n", "
\n", "\n", - "The flood data we will extract from a repository maintained by the European Commission Joint Research Centre. We will download river flood hazard maps from their [Flood Data Collection](https://data.jrc.ec.europa.eu/dataset/1d128b6c-a4ee-4858-9e34-6210707f3c81). \n", + "The flood data we will extract from a repository maintained by the European Commission Joint Research Centre. We will download river flood hazard maps from their [Flood Data Collection](https://data.jrc.ec.europa.eu/dataset/1d128b6c-a4ee-4858-9e34-6210707f3c81).\n", "\n", "Here we do not need to use an API and we also do not need to register ourselves, so we can download any of the files directly. To do so, we use the `urllib` package." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, "id": "c30f3b19-cba4-4f4a-a902-082e396b7ac4", - "metadata": {}, + "metadata": { + "id": "c30f3b19-cba4-4f4a-a902-082e396b7ac4" + }, "outputs": [], "source": [ "## this is the link to the 1/100 flood map for Europe\n", "zipurl = 'https://jeodpp.jrc.ec.europa.eu/ftp/jrc-opendata/FLOODS/EuropeanMaps/floodMap_RP100.zip'\n", "\n", + "# The path where the downloaded flood map will be extracted, this is the folder of this Google Collaboratory instance. NOTE: a new instance will have this directory be cleared.\n", + "data_path = \"\"\n", + "\n", "# and now we open and extract the data\n", "with urlopen(zipurl) as zipresp:\n", " with ZipFile(BytesIO(zipresp.read())) as zfile:\n", @@ -218,7 +259,9 @@ { "cell_type": "markdown", "id": "e8baa489-3b9c-4d16-afdf-359651d2f6ba", - "metadata": {}, + "metadata": { + "id": "e8baa489-3b9c-4d16-afdf-359651d2f6ba" + }, "source": [ "The download and zip in the cell above sometimes does not work. If that is indeed the case (e.g., when it seems to remain stuck), download the files manually through the link and upload them in the data folder for this week (as explained at the start of this tutorial.)" ] @@ -226,70 +269,74 @@ { "cell_type": "markdown", "id": "94cb3103-bc54-4670-9936-f299561190f6", - "metadata": {}, + "metadata": { + "id": "94cb3103-bc54-4670-9936-f299561190f6" + }, "source": [ - "### Set location to explore\n", + "#### Set location to explore\n", "---\n", "Before we continue, we need to specify our location of interest. This should be a province that will have some flooding and relative high wind speeds occuring (else we will find zero damage). We specify the region of interest in the cell below by using the `geocode_to_gdf()` function." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "id": "edb7b64e-de75-4490-ac72-bad321d4da2c", - "metadata": {}, + "metadata": { + "id": "edb7b64e-de75-4490-ac72-bad321d4da2c" + }, "outputs": [], "source": [ - "place_name = \"Gelderland, The Netherlands\" ### But you could also consider Zeeland, for example.\n", + "place_name = \"Kampen, The Netherlands\" ### But you could also consider a city in Zeeland, for example.\n", "area = ox.geocode_to_gdf(place_name)" ] }, { "cell_type": "markdown", "id": "9bd3821f-16dd-4e7b-8ac6-a46475704afd", - "metadata": {}, + "metadata": { + "id": "9bd3821f-16dd-4e7b-8ac6-a46475704afd" + }, "source": [ - "## 3. Exploring the natural hazard data\n", + "### 3. Exploring the natural hazard data\n", "
" ] }, - { - "cell_type": "markdown", - "id": "92e203b2-e4c0-458a-a91e-9218614baeab", - "metadata": {}, - "source": [ - "Now we will explore our natural hazard data." - ] - }, { "cell_type": "markdown", "id": "424d77ba-32df-44c4-a565-b197b7e4cefc", - "metadata": {}, + "metadata": { + "id": "424d77ba-32df-44c4-a565-b197b7e4cefc" + }, "source": [ - "### Windstorm Data\n", + "#### Windstorm Data\n", "---" ] }, { "cell_type": "markdown", "id": "8b379bcc-84a2-4bb6-86a6-d0e3ad0d1c84", - "metadata": {}, + "metadata": { + "id": "8b379bcc-84a2-4bb6-86a6-d0e3ad0d1c84" + }, "source": [ "As you can see in the section above, we have downloaded the storm footprint in a zipfile. Let's open the zipfile and load the dataset using the `xarray` package through the `open_dataset()` function." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, "id": "64d33537-8db8-4def-89ee-2387ec993831", - "metadata": {}, + "metadata": { + "id": "64d33537-8db8-4def-89ee-2387ec993831" + }, "outputs": [], "source": [ "with ZipFile('Carmen.zip') as zf:\n", - " \n", + "\n", " # Let's get the filename first\n", " file = zf.namelist()[0]\n", - " \n", + "\n", " # And now we can open and select the file within Python\n", " with zf.open(file) as f:\n", " windstorm_europe = xr.open_dataset(f)" @@ -298,7 +345,9 @@ { "cell_type": "markdown", "id": "375410e2-ebe5-4f4d-b924-ad15e8fb9d59", - "metadata": {}, + "metadata": { + "id": "375410e2-ebe5-4f4d-b924-ad15e8fb9d59" + }, "source": [ "Let's have a look at the storm we have downloaded!" ] @@ -307,28 +356,27 @@ "cell_type": "code", "execution_count": null, "id": "7d35bb91-0705-4d80-b063-adabc74cc353", - "metadata": {}, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 490 + }, + "id": "7d35bb91-0705-4d80-b063-adabc74cc353", + "outputId": "f6fbb7c5-9649-41db-ca5f-0251faa6357a" + }, "outputs": [], "source": [ "windstorm_europe['FX'].plot()" ] }, - { - "cell_type": "markdown", - "id": "092e7f97-cf7a-4000-818f-1a73e578b0ec", - "metadata": {}, - "source": [ - "
\n",
- "Question 1: Describe windstorm Carmen. When did this event happen, which areas were most affected? Can you say something about the maximum wind speeds in different areas, based on the plot? And what does FX mean?\n",
- "
"
- ]
- },
{
"cell_type": "markdown",
"id": "13e8caf3-c5c3-4970-b923-b58d1fe007f8",
- "metadata": {},
+ "metadata": {
+ "id": "13e8caf3-c5c3-4970-b923-b58d1fe007f8"
+ },
"source": [
- "Unfortunately, our data does not have a proper coordinate system defined yet. As such, we will need to use the `rio.write_crs()` function to set the coordinate system to **EPSG:4326** (the standard global coordinate reference system). \n",
+ "Unfortunately, our data does not have a proper coordinate system defined yet. As such, we will need to use the `rio.write_crs()` function to set the coordinate system to **EPSG:4326** (the standard global coordinate reference system).\n",
"\n",
"We also need to make sure that the functions will know what the exact parameters are that we have to use for our spatial dimenions (e.g. longitude and latitude). It prefers to be named `x` and `y`. So we use the `rename()` function before we use the `set_spatial_dims()` function."
]
@@ -337,10 +385,17 @@
"cell_type": "code",
"execution_count": null,
"id": "5ca7b6ec-a394-4e3c-82dd-8a59ce029563",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 291
+ },
+ "id": "5ca7b6ec-a394-4e3c-82dd-8a59ce029563",
+ "outputId": "8bd1709a-1525-4d4d-e349-84ef40dce012"
+ },
"outputs": [],
"source": [
- "windstorm_europe.rio.write_crs(4326, inplace=True)\n",
+ "windstorm_europe. #add CRS\n",
"windstorm_europe = windstorm_europe.rename({'Latitude': 'y','Longitude': 'x'})\n",
"windstorm_europe.rio.set_spatial_dims(x_dim=\"x\",y_dim=\"y\", inplace=True)"
]
@@ -348,44 +403,54 @@
{
"cell_type": "markdown",
"id": "08e41966-7273-4521-9b27-4c9498a66e0b",
- "metadata": {},
+ "metadata": {
+ "id": "08e41966-7273-4521-9b27-4c9498a66e0b"
+ },
"source": [
"\n",
- "Question 2: Climate data is often stored as a netCDF file. Please describe what a netCDF file is. Which information is stored in the netCDF file we have downloaded for the windstorm? What type of metadata does it contain?\n",
+ "Question 1: Climate data is often stored as a netCDF file. Please describe what a netCDF file is. Which information is stored in the netCDF file we have downloaded for the windstorm? What type of metadata does it contain?\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "8ba5e684-f658-42e9-bded-bbb47ced24e4",
- "metadata": {},
+ "metadata": {
+ "id": "8ba5e684-f658-42e9-bded-bbb47ced24e4"
+ },
"source": [
"Following, we also make sure it will be in the European coordinate system **EPSG:3035** to ensure we can easily use it together with the other data. To do so, we use the `rio.reproject()` function. You can simple add the number of the coordinate system."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 9,
"id": "8b276083-7825-4576-832b-129566f65222",
- "metadata": {},
+ "metadata": {
+ "id": "8b276083-7825-4576-832b-129566f65222"
+ },
"outputs": [],
"source": [
- "windstorm_europe = windstorm_europe. [add function]"
+ "windstorm_europe = windstorm_europe. # add reproject"
]
},
{
"cell_type": "markdown",
"id": "0650103f-eed7-43f6-b9ef-a221781e356a",
- "metadata": {},
+ "metadata": {
+ "id": "0650103f-eed7-43f6-b9ef-a221781e356a"
+ },
"source": [
"Now we have all the information to clip the windstorm data to our area of interest:"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 10,
"id": "86dbfe52-1484-48fd-9dee-fc6064fe343b",
- "metadata": {},
+ "metadata": {
+ "id": "86dbfe52-1484-48fd-9dee-fc6064fe343b"
+ },
"outputs": [],
"source": [
"windstorm_map = windstorm_europe.rio.clip(area.envelope.values, area.crs)"
@@ -394,7 +459,9 @@
{
"cell_type": "markdown",
"id": "237c142f-2741-41bf-bace-96b6fede47f2",
- "metadata": {},
+ "metadata": {
+ "id": "237c142f-2741-41bf-bace-96b6fede47f2"
+ },
"source": [
"And let's have a look as well by using the `plot()` function. Please note that the legend is in meters per second."
]
@@ -403,34 +470,47 @@
"cell_type": "code",
"execution_count": null,
"id": "f2b8e8ca-6a58-44df-8981-b040cc55b3b0",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 506
+ },
+ "id": "f2b8e8ca-6a58-44df-8981-b040cc55b3b0",
+ "outputId": "f34f6dcf-303a-4c67-d18b-2d604df8ec5b"
+ },
"outputs": [],
"source": [
- "windstorm_map['FX']. [add function]"
+ "windstorm_map['FX'] # add plot function"
]
},
{
"cell_type": "markdown",
"id": "4bded755-8d9b-41a0-bd6e-a25000bc7486",
- "metadata": {},
+ "metadata": {
+ "id": "4bded755-8d9b-41a0-bd6e-a25000bc7486"
+ },
"source": [
- "### Flood Data\n",
+ "#### Flood Data\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "6ef96754-17bc-4d02-92ab-bb06280015a5",
- "metadata": {},
+ "metadata": {
+ "id": "6ef96754-17bc-4d02-92ab-bb06280015a5"
+ },
"source": [
"And similarly, we want to open the flood map. But now we do not have to unzip the file anymore and we can directly open it through using `xarray`:"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 12,
"id": "b78db556-c66c-46a0-9615-b98e26b9e57f",
- "metadata": {},
+ "metadata": {
+ "id": "b78db556-c66c-46a0-9615-b98e26b9e57f"
+ },
"outputs": [],
"source": [
"flood_map_path = 'floodmap_EFAS_RP100_C.tif'"
@@ -440,7 +520,14 @@
"cell_type": "code",
"execution_count": null,
"id": "555fab92-0c74-426e-bcfe-655cf6b25682",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 291
+ },
+ "id": "555fab92-0c74-426e-bcfe-655cf6b25682",
+ "outputId": "90525d96-e592-4918-9fc0-ab4b9dc95784"
+ },
"outputs": [],
"source": [
"flood_map = xr.open_dataset(flood_map_path, engine=\"rasterio\")\n",
@@ -450,7 +537,9 @@
{
"cell_type": "markdown",
"id": "5eb3c24d-4ed2-4717-befc-67e52e1bbbea",
- "metadata": {},
+ "metadata": {
+ "id": "5eb3c24d-4ed2-4717-befc-67e52e1bbbea"
+ },
"source": [
"And let's make sure we set all the variables and the CRS correctly again to be able to open the data properly. Note that we should now use **EPSG:3035**. This is the standard coordinate system for Europe, in meters (instead of degrees)."
]
@@ -459,31 +548,40 @@
"cell_type": "code",
"execution_count": null,
"id": "b7e6de85-b165-4b7e-85ea-d09fe6bf9024",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 291
+ },
+ "id": "b7e6de85-b165-4b7e-85ea-d09fe6bf9024",
+ "outputId": "14296427-2a11-432e-a17b-c3a2f163b201"
+ },
"outputs": [],
"source": [
- "flood_map.rio.write_crs( , inplace=True)\n",
+ "flood_map.rio.write_crs(3035, inplace=True)\n",
"flood_map.rio.set_spatial_dims(x_dim=\"x\",y_dim=\"y\", inplace=True)"
]
},
{
"cell_type": "markdown",
"id": "9ebad344-dd4b-4c37-bafe-2464f11a79b2",
- "metadata": {},
+ "metadata": {
+ "id": "9ebad344-dd4b-4c37-bafe-2464f11a79b2"
+ },
"source": [
"Now it is pretty difficult to explore the data for our area of interest, so let's clip the flood data. \n",
"\n",
"We want to clip our flood data to our chosen area. The code, however, is very inefficient and will run into memories issues on Google Colab. As such, we first need to clip it by using a bounding box, followed by the actual clip.\n",
"\n",
"\n",
- "Question 3: Please provide the lines of code below in which you show how you have clipped the flood map to your area.\n",
+ "Question 2: Please provide the lines of code below in which you show how you have clipped the flood map to your area.\n",
"
\n",
"\n",
"*A few hints*:\n",
"\n",
- "* carefully read the documentation of the `.clip_box()` function of rioxarray. Which information do you need? \n",
- "* is the GeoDataFrame of your region (the area GeoDataframe) in the same coordinate system? Perhaps you need to convert it using the `.to_crs()` function. \n",
- "* how do you get the bounds from your area GeoDataFrame? \n",
+ "* carefully read the documentation of the `.clip_box()` function of rioxarray. Which information do you need?\n",
+ "* is the GeoDataFrame of your region (the area GeoDataframe) in the same coordinate system? Perhaps you need to convert it using the `.to_crs()` function.\n",
+ "* how do you get the bounds from your area GeoDataFrame?\n",
"* The final step of the clip would be to use the `.rio.clip()` function, using the actual area file and the flood map clipped to the bounding box. Please note that you should **not** use the envelope here, like we did in the previous clip. Here we really want to use the exact geometry values.\n",
"\n",
"As you will see, we first clip it very efficiently using the bounding box. After that, we do an exact clip."
@@ -491,24 +589,28 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 15,
"id": "12a22299-6507-45b3-b4ef-1321a193ad8f",
- "metadata": {},
+ "metadata": {
+ "id": "12a22299-6507-45b3-b4ef-1321a193ad8f"
+ },
"outputs": [],
"source": [
"min_lon = area.to_crs(epsg=3035).bounds.minx.values[0]\n",
- "min_lat = area.to_crs(epsg=3035).bounds.miny\n",
- "max_lon = area.to_crs(epsg=3035).bounds\n",
- "max_lat = area.to_crs(epsg=3035).\n",
+ "min_lat = area.to_crs(epsg=3035).bounds.miny #complete function\n",
+ "max_lon = area.to_crs(epsg=3035).bounds #complete function\n",
+ "max_lat = area.to_crs(epsg=3035). #complete function\n",
"\n",
- "flood_map_area = flood_map.rio.clip_box(minx=.... )\n",
- "flood_map_area = flood_map_area.rio.clip(area.XXXX.values, area.crs)"
+ "flood_map_area = flood_map.rio.clip_box #complete function\n",
+ "flood_map_area = flood_map_area.rio.clip( #add geometry values, area.crs)"
]
},
{
"cell_type": "markdown",
"id": "c744f4ff-e7c2-4b1b-b366-9ebd577f76bd",
- "metadata": {},
+ "metadata": {
+ "id": "c744f4ff-e7c2-4b1b-b366-9ebd577f76bd"
+ },
"source": [
"And let's have a look as well. Please note that the legend is in meters."
]
@@ -517,25 +619,48 @@
"cell_type": "code",
"execution_count": null,
"id": "58f23727-c8c0-4379-95f7-bc41ba5b3c6e",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 506
+ },
+ "id": "58f23727-c8c0-4379-95f7-bc41ba5b3c6e",
+ "outputId": "da748cee-b5d4-46c2-e8fb-36ce6c41385d"
+ },
"outputs": [],
"source": [
"flood_map_area['band_data'].plot(cmap='Blues',vmax=10)"
]
},
+ {
+ "cell_type": "markdown",
+ "id": "250b28b1-6da5-4700-9a17-5aa18967c120",
+ "metadata": {
+ "id": "250b28b1-6da5-4700-9a17-5aa18967c120"
+ },
+ "source": [
+ "\n",
+ "Question 3: Now that we have both wind and flood maps, comment on their spatial resolution. What would the impact of the different in resolution for the outcome of your risk assessment?\n",
+ "
"
+ ]
+ },
{
"cell_type": "markdown",
"id": "c2c98ee1-d8f8-4eba-bfd2-36f26e849279",
- "metadata": {},
+ "metadata": {
+ "id": "c2c98ee1-d8f8-4eba-bfd2-36f26e849279"
+ },
"source": [
- "## 4. Downloading and exploring Land Cover data and Land Use data\n",
+ "### 4. Downloading and exploring Land Cover data and Land Use data\n",
"" ] }, { "cell_type": "markdown", "id": "d0402ee8-6460-4db0-b216-b16b88ff2c56", - "metadata": {}, + "metadata": { + "id": "d0402ee8-6460-4db0-b216-b16b88ff2c56" + }, "source": [ "We will explore rasterized Corine Land Cover data and land use data retrieved from OpenStreetMap." ] @@ -543,33 +668,35 @@ { "cell_type": "markdown", "id": "8ccebb6a-8c7b-4c7f-b974-899e15c9631b", - "metadata": {}, + "metadata": { + "id": "8ccebb6a-8c7b-4c7f-b974-899e15c9631b" + }, "source": [ - "### Download and access Copernicus Land Cover data\n", + "#### Download and access Copernicus Land Cover data\n", "---" ] }, { "cell_type": "markdown", "id": "672472d8-aff1-498f-ac36-41082ac32c4e", - "metadata": {}, + "metadata": { + "id": "672472d8-aff1-498f-ac36-41082ac32c4e" + }, "source": [ - "Unfortunately, there is no API option to download the [Corine Land Cover](https://land.copernicus.eu/pan-european/corine-land-cover) data. We will have to download the data from the website first.\n", - "\n", - "To do so, we will first have to register ourselves again on the website. Please find in the video clip below how to register yourself on the website of the [Copernicus Land Monitoring Service](https://land.copernicus.eu/):\n", + "We will now download the [Corine Land Cover](https://land.copernicus.eu/pan-european/corine-land-cover) data.\n", "\n", - "
![](\"https://github.com/ElcoK/BigData_AED/blob/main/_static/images/CLMS_registration.gif?raw=1\")
\n",
- "\n",
- "Now click on the Login button in the top right corner to login on the website. There are many interesting datasets on this website, but we just want to download the Corine Land Cover data, and specifically the latest version: [Corine Land Cover 2018](https://land.copernicus.eu/pan-european/corine-land-cover/clc2018?tab=download). To do so, please select the **Corine Land Cover - 100 meter**. Now click on the large green Download button. Your download should start any minute.\n",
+ "To do so, we will first have to register ourselves again on the website. Now click on the Login button in the top right corner to login on the website. There are many interesting datasets on this website, but we just want to download the Corine Land Cover data, and specifically the latest version: [Corine Land Cover 2018](https://land.copernicus.eu/pan-european/corine-land-cover/clc2018?tab=download). To do so, click on the large green Download button. Now please select the **Corine Land Cover - 100 meter** and add this to your cart. Next, go to your cart and click on Process download request. After this, the requested data can be downloaded via the 'downloading process page'. After hitting Download file, your download should start any minute.\n",
"\n",
"Slightly annoying, the file you have downloaded is double zipped. Its slightly inconvenient to open this through Python and within Google Drive. So let's unzip it twice outside of Python (on your local machine) and then direct yourself to the `DATA` directory within the unzipped file. Here you can find a file called `U2018_CLC2018_V2020_20u1.tif`. Drop this file into this week's data directory, as specified at the start of this tutorial when we mounted our Google Drive."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 17,
"id": "5e1c9079-cde0-4be2-8b72-f7403d5f0e4b",
- "metadata": {},
+ "metadata": {
+ "id": "5e1c9079-cde0-4be2-8b72-f7403d5f0e4b"
+ },
"outputs": [],
"source": [
"CLC_location = 'U2018_CLC2018_V2020_20u1.tif'"
@@ -577,9 +704,11 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 18,
"id": "baaa699e-1ba0-4ed8-9996-38d8cc81578b",
- "metadata": {},
+ "metadata": {
+ "id": "baaa699e-1ba0-4ed8-9996-38d8cc81578b"
+ },
"outputs": [],
"source": [
"CLC = xr.open_dataset(CLC_location, engine=\"rasterio\")"
@@ -588,27 +717,38 @@
{
"cell_type": "markdown",
"id": "d263e20d-d88b-4cf3-ad5c-c1ff2d93ad6b",
- "metadata": {},
+ "metadata": {
+ "id": "d263e20d-d88b-4cf3-ad5c-c1ff2d93ad6b"
+ },
"source": [
"Similarly to the flood map data, we need to do a two-stage clip again (like we did before in this tutorial to ensure we get only our area of interest without exceeding our RAM."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 19,
"id": "8299869a-2724-44c6-b5af-d134f7b9aece",
- "metadata": {},
+ "metadata": {
+ "id": "8299869a-2724-44c6-b5af-d134f7b9aece"
+ },
"outputs": [],
"source": [
- "CLC_region = CLC.rio.clip_box(\n",
- "CLC_region = CLC_region.rio.clip("
+ "CLC_region = CLC # complete function\n",
+ "CLC_region = CLC_region # complete function"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8706a9da-4336-4f06-a8a7-4c825639dae4",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 291
+ },
+ "id": "8706a9da-4336-4f06-a8a7-4c825639dae4",
+ "outputId": "5389b953-fdf1-449e-932a-726e160b11b2"
+ },
"outputs": [],
"source": [
"CLC_region = CLC_region.rename({'x': 'lat','y': 'lon'})\n",
@@ -618,18 +758,22 @@
{
"cell_type": "markdown",
"id": "a31c0989-e96e-4809-a24d-13ab00de751f",
- "metadata": {},
+ "metadata": {
+ "id": "a31c0989-e96e-4809-a24d-13ab00de751f"
+ },
"source": [
- "Our next step is to prepare the visualisation of a map. What better way to explore land-cover information than plotting it on a map? \n",
+ "Our next step is to prepare the visualisation of a map. What better way to explore land-cover information than plotting it on a map?\n",
"\n",
- "As you will see below, we can create a dictionary with color codes that will color each land-cover class based on the color code provided in this dictionary. We use the colorscheme of Corine Land Cover. "
+ "As you will see below, we can create a dictionary with color codes that will color each land-cover class based on the color code provided in this dictionary. We use the colorscheme of Corine Land Cover. Please find the overview of classes and colors [here](https://collections.sentinel-hub.com/corine-land-cover/readme.html)."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 21,
"id": "198958ec-68a5-44d2-9b8c-91a2dd19484f",
- "metadata": {},
+ "metadata": {
+ "id": "198958ec-68a5-44d2-9b8c-91a2dd19484f"
+ },
"outputs": [],
"source": [
"CLC_values = [111, 112, 121, 122, 123, 124, 131, 132, 133, 141, 142, 211, 212, 213, 221, 222, 223, 231, 241, 242,\n",
@@ -644,16 +788,20 @@
{
"cell_type": "markdown",
"id": "6d2c9c52",
- "metadata": {},
+ "metadata": {
+ "id": "6d2c9c52"
+ },
"source": [
"The code below allows us the use the color_dict above to plot the CLC map"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 22,
"id": "59b50772-57d5-4e49-a600-00bfeaf9ffe8",
- "metadata": {},
+ "metadata": {
+ "id": "59b50772-57d5-4e49-a600-00bfeaf9ffe8"
+ },
"outputs": [],
"source": [
"color_dict_raster = dict(zip(CLC_values,CLC_colors))\n",
@@ -661,7 +809,7 @@
"# We create a colormar from our list of colors\n",
"cm = ListedColormap(CLC_colors)\n",
"\n",
- "# Let's also define the description of each category : 1 (blue) is Sea; 2 (red) is burnt, etc... Order should be respected here ! Or using another dict maybe could help.\n",
+ "# Let's also define the description of each category in the raster\n",
"labels = np.array(CLC_values)\n",
"len_lab = len(labels)\n",
"\n",
@@ -678,7 +826,9 @@
{
"cell_type": "markdown",
"id": "68baa525-98be-4069-8129-dc6c22db4793",
- "metadata": {},
+ "metadata": {
+ "id": "68baa525-98be-4069-8129-dc6c22db4793"
+ },
"source": [
"And let's plot the Corine Land Cover data for our area of interest"
]
@@ -687,7 +837,14 @@
"cell_type": "code",
"execution_count": null,
"id": "2a977453-9180-4de6-872e-1fe7818f2915",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 906
+ },
+ "id": "2a977453-9180-4de6-872e-1fe7818f2915",
+ "outputId": "b8d0a689-46b1-4a31-d9b6-592b54f216af"
+ },
"outputs": [],
"source": [
"fig, ax = plt.subplots(1, 1,figsize=(14,10))\n",
@@ -695,59 +852,39 @@
"CLC_region[\"band_data\"].plot(ax=ax,levels=len(CLC_colors),colors=CLC_colors)"
]
},
- {
- "cell_type": "markdown",
- "id": "465c4b28-3d01-4960-bb7c-8709a0c023bc",
- "metadata": {},
- "source": [
- "\n",
- "Question 4: Describe the different land-use classes within your region that you see on the Corine Land Cover map. Do you see any dominant land-use classes? \n",
- "
"
- ]
- },
{
"cell_type": "markdown",
"id": "8867d41e-2b7b-40c2-8576-611f2c04f547",
- "metadata": {},
+ "metadata": {
+ "id": "8867d41e-2b7b-40c2-8576-611f2c04f547"
+ },
"source": [
- "### Extract and visualize land-use information from OpenStreetMap\n",
+ "#### Extract and visualize land-use information from OpenStreetMap\n",
"---"
]
},
- {
- "cell_type": "markdown",
- "id": "3e0dd8ab-15e8-494d-a6fb-3861636e5ed4",
- "metadata": {},
- "source": [
- "The next step is to define which area you want to focus on. In the cell below, you will now read \"Kampen, The Netherlands\". Change this to any area or municipality in the Netherlands that (1) you can think of and (2) will work. \n",
- "\n",
- "In some cases, the function does not recognize the location. You could either try a different phrasing or try a different location. Many parts of the Netherlands should work."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "12189f0d-5f9f-4ada-922a-6fc12f9bbf2f",
- "metadata": {},
- "outputs": [],
- "source": [
- "place_name = \"Kampen, The Netherlands\"\n",
- "area = ox.geocode_to_gdf(place_name)"
- ]
- },
{
"cell_type": "markdown",
"id": "2077d677-c8c2-4fd5-b0d0-21fad4f3eec5",
- "metadata": {},
+ "metadata": {
+ "id": "2077d677-c8c2-4fd5-b0d0-21fad4f3eec5"
+ },
"source": [
- "Now let us visualize the bounding box of the area. As you will notice, we also estimate the size of the area. If the area size is above 50km2, or when you have many elements within your area (for example the amsterdam city centre), extracting the data from OpenStreetMap may take a little while. "
+ "Now let us visualize the bounding box of the area. As you will notice, we also estimate the size of the area. If the area size is above 50km2, or when you have many elements within your area (for example the amsterdam city centre), extracting the data from OpenStreetMap may take a little while."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "50447dc3-9d91-4a64-adea-90cbb0e0a6be",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 724
+ },
+ "id": "50447dc3-9d91-4a64-adea-90cbb0e0a6be",
+ "outputId": "0c90ac99-794f-435c-ad35-f96d02261392"
+ },
"outputs": [],
"source": [
"area_to_check = area.to_crs(epsg=3857)\n",
@@ -757,55 +894,53 @@
"ax.set_axis_off()\n",
"cx.add_basemap(ax, zoom=11)\n",
"\n",
- "size = int(area_to_check.area/1e6)\n",
+ "size = int(area_to_check.area.values/1e6)\n",
"\n",
"ax.set_title(\"{}. Total area: {} km2\".format(place_name,size),fontweight='bold')"
]
},
- {
- "cell_type": "markdown",
- "id": "0154abf9-c923-4253-9e97-e8c91fe9d632",
- "metadata": {},
- "source": [
- "\n",
- "Question 5: To make sure we understand which area you focus on, please submit the figure that outlines your area.\n",
- "
"
- ]
- },
{
"cell_type": "markdown",
"id": "865b5084-08f0-471d-8982-06b340e5d3f5",
- "metadata": {},
+ "metadata": {
+ "id": "865b5084-08f0-471d-8982-06b340e5d3f5"
+ },
"source": [
"Now we are satisfied with the selected area, we are going to extract the land-use information from OpenStreetMap. To find the right information from OpenStreetMap, we use **tags**.\n",
"\n",
- "As you will see in the cell below, we use the tags *\"landuse\"* and *\"natural\"*. We need to use the *\"natural\"* tag to ensure we also obtain water bodies and other natural elements. "
+ "As you will see in the cell below, we use the tags *\"landuse\"* and *\"natural\"*. We need to use the *\"natural\"* tag to ensure we also obtain water bodies and other natural elements."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 25,
"id": "5f52137d-bf2c-4405-9df1-42ac73a5d83c",
- "metadata": {},
+ "metadata": {
+ "id": "5f52137d-bf2c-4405-9df1-42ac73a5d83c"
+ },
"outputs": [],
"source": [
- "tags = {'landuse': True, 'natural': True} \n",
+ "tags = {'landuse': True, 'natural': True}\n",
"landuse = ox.features_from_place(place_name, tags)"
]
},
{
"cell_type": "markdown",
"id": "9f44095d-c47c-49d4-ac8d-6a91eae8f1ae",
- "metadata": {},
+ "metadata": {
+ "id": "9f44095d-c47c-49d4-ac8d-6a91eae8f1ae"
+ },
"source": [
"In case the above does not work, you can continue the assignment by using the code below (make sure you remove the hashtags to run it). If you decide to use the data as specified below, also change the map at the start to 'Kampen'."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 26,
"id": "13cd17e2-d990-447a-b3d4-75dbe2b93ed1",
- "metadata": {},
+ "metadata": {
+ "id": "13cd17e2-d990-447a-b3d4-75dbe2b93ed1"
+ },
"outputs": [],
"source": [
"# remote_url = 'https://github.com/ElcoK/BigData_AED/raw/main/week5/kampen_landuse.gpkg'\n",
@@ -818,7 +953,9 @@
{
"cell_type": "markdown",
"id": "2ff9e972-55b9-45a9-b727-45066032e900",
- "metadata": {},
+ "metadata": {
+ "id": "2ff9e972-55b9-45a9-b727-45066032e900"
+ },
"source": [
"To ensure we really only get the area that we want, we use geopandas's `clip` function to only keep the area we want. This function does exactly the same as the `clip` function in QGIS."
]
@@ -826,19 +963,39 @@
{
"cell_type": "markdown",
"id": "f7be7294-2cc6-4997-9c62-a5aea3fd1c29",
- "metadata": {},
+ "metadata": {
+ "id": "f7be7294-2cc6-4997-9c62-a5aea3fd1c29"
+ },
"source": [
- "When we want to visualize or analyse the data, we want all information in a single column. However, at the moment, all information that was tagged as *\"natural\"*, has no information stored in the *\"landuse\"* tags. It is, however, very convenient if we can just use a single column for further exploration of the data. \n",
+ "When we want to visualize or analyse the data, we want all information in a single column. However, at the moment, all information that was tagged as *\"natural\"*, has no information stored in the *\"landuse\"* tags. It is, however, very convenient if we can just use a single column for further exploration of the data.\n",
"\n",
- "To overcome this issue, we need to add the missing information to the landuse column, as done below. Let's first have a look which categories we have in the **natural** column. "
+ "To overcome this issue, we need to add the missing information to the landuse column, as done below. Let's first have a look which categories we have in the **natural** column."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 27,
"id": "0ed6e32f-06a7-4189-8e77-efd1a9cf77f7",
- "metadata": {},
- "outputs": [],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "0ed6e32f-06a7-4189-8e77-efd1a9cf77f7",
+ "outputId": "4aa85ca8-bb28-4943-e6aa-73cc2ba9ca46"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array(['tree', nan, 'water', 'scrub', 'grassland', 'wetland', 'beach',\n",
+ " 'tree_row', 'sand'], dtype=object)"
+ ]
+ },
+ "execution_count": 27,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
"source": [
"landuse.natural.unique()"
]
@@ -846,16 +1003,20 @@
{
"cell_type": "markdown",
"id": "b3a6a6af-c3db-450b-944c-95c7cef1fd32",
- "metadata": {},
+ "metadata": {
+ "id": "b3a6a6af-c3db-450b-944c-95c7cef1fd32"
+ },
"source": [
"And now we can add them to the **landuse** column. We made a start, but its up to you to fill in the rest."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 28,
"id": "53b14f7a-f62d-4575-ad02-39f279fde5e1",
- "metadata": {},
+ "metadata": {
+ "id": "53b14f7a-f62d-4575-ad02-39f279fde5e1"
+ },
"outputs": [],
"source": [
"landuse.loc[landuse.natural=='water','landuse'] = 'water'\n",
@@ -868,61 +1029,82 @@
{
"cell_type": "markdown",
"id": "851df40d-02d1-4409-9a89-9d1edcfa6bcd",
- "metadata": {},
+ "metadata": {
+ "id": "851df40d-02d1-4409-9a89-9d1edcfa6bcd"
+ },
"source": [
"\n",
- "Question 6: Please provide in the answer box in Canvas the code that you used to make sure that all land uses are now registered within the landuse column.\n",
+ "Question 4: Please provide in the answer box in Canvas the code that you used to make sure that all land uses are now registered within the landuse column.\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "6ead9126-edc3-41f1-80de-339e1e42dbda",
- "metadata": {},
+ "metadata": {
+ "id": "6ead9126-edc3-41f1-80de-339e1e42dbda"
+ },
"source": [
- "We now create a *color_dict* like we have also done for the visualization of the land-use information to ensure we can visualize the data properly. This time, we use our own colorscheme. "
+ "We now create a *color_dict* like we have also done for the visualization of the land-use information to ensure we can visualize the data properly. This time, we use our own colorscheme."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 29,
"id": "64197848-0d42-480e-9d7f-4a68bff63772",
- "metadata": {},
+ "metadata": {
+ "id": "64197848-0d42-480e-9d7f-4a68bff63772"
+ },
"outputs": [],
"source": [
"color_dict = { \"grass\":'#c3eead', \"railway\": \"#000000\",\n",
" \"forest\":'#1c7426', \"orchard\":'#fe6729',\n",
" \"residential\":'#f13013', \"industrial\":'#0f045c',\n",
- " \"retail\":'#b71456', \"education\":'#d61181', \n",
+ " \"retail\":'#b71456', \"education\":'#d61181',\n",
" \"commercial\":'#981cb8', \"farmland\":'#fcfcb9',\n",
- " \"cemetery\":'#c39797', \"construction\":'#c0c0c0',\n",
" \"meadow\":'#c3eead', \"farmyard\":'#fcfcb9',\n",
- " \"plant_nursery\":'#eaffe2', \"scrub\":'#98574d',\n",
- " \"allotments\":'#fbffe2', \"reservoir\":'#8af4f2',\n",
- " \"static_caravan\":'#ff3a55', \"wetlands\": \"#c9f5e5\",\n",
- " \"water\": \"#c9e5f5\", \"beach\": \"#ffeead\",\n",
" \"landfill\" : \"#B08C4D\", \"recreation_ground\" : \"#c3eead\",\n",
- " \"brownfield\" : \"#B08C4D\", \"village_green\" : \"#f13013\" ,\n",
- " \"military\": \"#52514E\", \"garden\" : '#c3eead'\n",
- " } "
+ " }"
]
},
{
"cell_type": "markdown",
"id": "408cdac8-2ca1-4eef-8997-8c83b852f5c5",
- "metadata": {},
+ "metadata": {
+ "id": "408cdac8-2ca1-4eef-8997-8c83b852f5c5"
+ },
"source": [
- "Unfortunately, OpenSteetMap very often contains elements that have a unique tag. As such, it may be the case that some of our land-use categories are not in the dictionary yet. \n",
+ "Unfortunately, OpenSteetMap very often contains elements that have a unique tag. As such, it may be the case that some of our land-use categories are not in the dictionary yet.\n",
"\n",
"Let's first create an overview of the unique land-use categories within our data through using the `.unique()` function within our dataframe:"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 30,
"id": "92c41069-fb2c-4606-adcc-e73ab95778ba",
- "metadata": {},
- "outputs": [],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "92c41069-fb2c-4606-adcc-e73ab95778ba",
+ "outputId": "19113de0-4c78-44bb-e467-db87717ac727"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array(['residential', 'water', 'grass', 'forest', 'wetlands', 'meadow',\n",
+ " 'industrial', 'retail', 'cemetery', 'farmland', 'orchard',\n",
+ " 'construction', 'scrub', 'commercial', 'education', 'farmyard',\n",
+ " 'static_caravan', 'railway', 'allotments'], dtype=object)"
+ ]
+ },
+ "execution_count": 30,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
"source": [
"landuse.landuse.unique()"
]
@@ -930,17 +1112,36 @@
{
"cell_type": "markdown",
"id": "f354fe05-b608-435f-acf4-6310886071dd",
- "metadata": {},
+ "metadata": {
+ "id": "f354fe05-b608-435f-acf4-6310886071dd"
+ },
"source": [
"Ofcourse we can visually compare the array above with our color_dict, but it is much quicker to use `Sets` to check if there is anything missing:"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 31,
"id": "cba31d28-3398-4d33-a325-eab786869020",
- "metadata": {},
- "outputs": [],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "cba31d28-3398-4d33-a325-eab786869020",
+ "outputId": "522fd89c-b76f-45d9-f635-2ad2ff30f31b"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "set()"
+ ]
+ },
+ "execution_count": 31,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
"source": [
"set(landuse.landuse.unique())-set(color_dict)"
]
@@ -948,12 +1149,14 @@
{
"cell_type": "markdown",
"id": "5d2de6c7-180e-43b3-ab32-ee7df7ff6ea2",
- "metadata": {},
+ "metadata": {
+ "id": "5d2de6c7-180e-43b3-ab32-ee7df7ff6ea2"
+ },
"source": [
- "In case anything is missing, add them to the color_dict dictionairy and re-run that cell. \n",
+ "In case anything is missing, add them to the color_dict dictionairy and re-run that cell.\n",
"\n",
"\n",
- "Question 7: Show us in Canvas (i) which land-use categories you had to add, and (ii) how your final color dictionary looks like.\n",
+ "Question 5: Show us in Canvas (i) which land-use categories you had to add, and (ii) how your final color dictionary looks like.\n",
"
\n",
"\n",
"```{tip}\n",
@@ -966,9 +1169,11 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 32,
"id": "45093fa0-b95b-49e1-bb32-d149ba424931",
- "metadata": {},
+ "metadata": {
+ "id": "45093fa0-b95b-49e1-bb32-d149ba424931"
+ },
"outputs": [],
"source": [
"color_dict = {key: color_dict[key]\n",
@@ -982,7 +1187,9 @@
{
"cell_type": "markdown",
"id": "010bec6f-fd76-40d2-a38c-be266e57d056",
- "metadata": {},
+ "metadata": {
+ "id": "010bec6f-fd76-40d2-a38c-be266e57d056"
+ },
"source": [
"Now we can plot the figure!\n",
"\n",
@@ -993,7 +1200,14 @@
"cell_type": "code",
"execution_count": null,
"id": "b8f01254-e1d7-4e3f-b40c-b2358223670c",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 847
+ },
+ "id": "b8f01254-e1d7-4e3f-b40c-b2358223670c",
+ "outputId": "c16e27d2-7da3-4716-b5ab-19a4f4926cea"
+ },
"outputs": [],
"source": [
"fig, ax = plt.subplots(1, 1,figsize=(12,10))\n",
@@ -1001,7 +1215,7 @@
"# add color scheme\n",
"color_scheme_map = list(color_dict.values())\n",
"cmap = LinearSegmentedColormap.from_list(name='landuse',\n",
- " colors=color_scheme_map) \n",
+ " colors=color_scheme_map)\n",
"\n",
"# and plot the land-use map.\n",
"landuse.plot(color=landuse['col_landuse'],ax=ax,linewidth=0)\n",
@@ -1014,9 +1228,9 @@
"# add a legend:\n",
"legend_elements = []\n",
"for iter_,item in enumerate(color_dict):\n",
- " legend_elements.append(Patch(facecolor=color_scheme_map[iter_],label=item)) \n",
+ " legend_elements.append(Patch(facecolor=color_scheme_map[iter_],label=item))\n",
"\n",
- "ax.legend(handles=legend_elements,edgecolor='black',facecolor='#fefdfd',prop={'size':12},loc=(1.02,0.2)) \n",
+ "ax.legend(handles=legend_elements,edgecolor='black',facecolor='#fefdfd',prop={'size':12},loc=(1.02,0.2))\n",
"\n",
"# add a title\n",
"ax.set_title(place_name,fontweight='bold')"
@@ -1025,28 +1239,34 @@
{
"cell_type": "markdown",
"id": "af1ca96a-2ec2-4889-b30b-1f9d23cc0d18",
- "metadata": {},
+ "metadata": {
+ "id": "af1ca96a-2ec2-4889-b30b-1f9d23cc0d18"
+ },
"source": [
"\n",
- "Question 8: Please upload a figure of your land-use map, using OpenStreetMap. \n",
+ "Question 6: Please upload a figure of your land-use map, using OpenStreetMap. Feel free to change the visual appearance of the map to your liking\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "99777639-4201-42aa-ab97-1705bed60392",
- "metadata": {},
+ "metadata": {
+ "id": "99777639-4201-42aa-ab97-1705bed60392"
+ },
"source": [
- "### Rasterize land-use information\n",
+ "#### Rasterize land-use information\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "3f8bda0f-aea9-471c-af15-31d9bf2bd191",
- "metadata": {},
+ "metadata": {
+ "id": "3f8bda0f-aea9-471c-af15-31d9bf2bd191"
+ },
"source": [
- "As you have noticed already during the lecture, and as we have seen during TAA1 with the Google Earth Engine, most land-use data is in raster format. \n",
+ "As you have noticed already during the lecture, and as we have seen during TAA1 with the Google Earth Engine, most land-use data is in raster format.\n",
"\n",
"In OpenStreetMap everything is stored in vector format. As such, the land-use information we extracted from OpenStreetMap is also in vector format. While it is not always necessary to have this information in raster format, it is useful to know how to convert your data into a raster format.\n",
"\n",
@@ -1056,16 +1276,20 @@
{
"cell_type": "markdown",
"id": "6d84eb30-2657-4cd3-8300-be6e22673d11",
- "metadata": {},
+ "metadata": {
+ "id": "6d84eb30-2657-4cd3-8300-be6e22673d11"
+ },
"source": [
"The first thing we will need to do is to define all the unique land-use classes and store them in a dictionary:"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 34,
"id": "48d1662b-23a0-443d-8465-0acad5e374c0",
- "metadata": {},
+ "metadata": {
+ "id": "48d1662b-23a0-443d-8465-0acad5e374c0"
+ },
"outputs": [],
"source": [
"categorical_enums = {'landuse': landuse.landuse.drop_duplicates().values.tolist()\n",
@@ -1075,15 +1299,17 @@
{
"cell_type": "markdown",
"id": "17845c5f-ec24-498c-a185-1cbf5d4054dd",
- "metadata": {},
+ "metadata": {
+ "id": "17845c5f-ec24-498c-a185-1cbf5d4054dd"
+ },
"source": [
- "And now we simply use the `make_geocube()` function to convert our vector data into raster data. \n",
+ "And now we simply use the `make_geocube()` function to convert our vector data into raster data.\n",
"\n",
"In the `make_geocube()` function, we have to specify several arguments:\n",
"\n",
"- Through the `vector_data` argument we have to state which dataframe we want to rasterize.\n",
"- Through the `output_crs` argument we have to state the coordinate reference system (CRS). We use the OpenStreetMap default EPSG:4326.\n",
- "- Through the `resolution` argument we have to state the resolution. In our case, we will have to set this in degrees. 0.01 degrees is equivalent to roughly 10km around the equator. \n",
+ "- Through the `resolution` argument we have to state the resolution. In our case, we will have to set this in degrees. 0.01 degrees is equivalent to roughly 10km around the equator.\n",
"- Through the `categorical_enums` argument we specify the different land-use categories.\n",
"\n",
"Play around with the different resolutions to find the level of detail. The higher the resolution (i.e., the more zeros behind the comma), the longer it will take to rasterize."
@@ -1091,23 +1317,27 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 35,
"id": "3ca109c1-5169-4833-818d-bae97f0b33b3",
- "metadata": {},
+ "metadata": {
+ "id": "3ca109c1-5169-4833-818d-bae97f0b33b3"
+ },
"outputs": [],
"source": [
"landuse_grid = make_geocube(\n",
- " vector_data=,\n",
- " output_crs=,\n",
- " resolution=(-XXXX, XXXX),\n",
- " categorical_enums=categorical_enums\n",
+ " vector_data= ,## add landuse\n",
+ " output_crs= ,## add CRS\n",
+ " resolution= ,## add resolution\n",
+ " categorical_enums= ## add categorical_enums\n",
")"
]
},
{
"cell_type": "markdown",
"id": "ba3ea735-0255-4b44-ab66-210bf3655213",
- "metadata": {},
+ "metadata": {
+ "id": "ba3ea735-0255-4b44-ab66-210bf3655213"
+ },
"source": [
"Let's explore what this function has given us:"
]
@@ -1116,7 +1346,14 @@
"cell_type": "code",
"execution_count": null,
"id": "2ca50588-074b-4217-a384-f582a57f6b35",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 389
+ },
+ "id": "2ca50588-074b-4217-a384-f582a57f6b35",
+ "outputId": "a8f53190-d7a2-4139-8a54-6a3511d51c2f"
+ },
"outputs": [],
"source": [
"landuse_grid[\"landuse\"]"
@@ -1125,11 +1362,13 @@
{
"cell_type": "markdown",
"id": "35204590-9e19-45a0-8065-4e453d024d69",
- "metadata": {},
+ "metadata": {
+ "id": "35204590-9e19-45a0-8065-4e453d024d69"
+ },
"source": [
- "The output above is a typical output of the **xarray** package. \n",
+ "The output above is a typical output of the **xarray** package.\n",
"\n",
- "- The `array` shows the numpy array with the actual values. As you can see, the rasterization process has used the value `-1` for NoData. \n",
+ "- The `array` shows the numpy array with the actual values. As you can see, the rasterization process has used the value `-1` for NoData.\n",
"- The `Coordinates` table shows the x (longitude) and y (latitude) coordinates of the array. It has the exact same size as the `array` with land-use values.\n",
"- The `Attributes` table specifies the NoData value (the `_FillValue` element, which indeed shows `-1`) and the name of the dataset.\n",
"\n",
@@ -1140,7 +1379,14 @@
"cell_type": "code",
"execution_count": null,
"id": "407f75e9-4747-43e1-93a0-d1b63d90378d",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 825
+ },
+ "id": "407f75e9-4747-43e1-93a0-d1b63d90378d",
+ "outputId": "90a878fd-5c26-49e1-c562-2430ef8d4c21"
+ },
"outputs": [],
"source": [
"fig, ax = plt.subplots(1, 1,figsize=(14,10))\n",
@@ -1160,9 +1406,11 @@
{
"cell_type": "markdown",
"id": "945b1ce7-f6e6-410d-be82-e24e9d224bd3",
- "metadata": {},
+ "metadata": {
+ "id": "945b1ce7-f6e6-410d-be82-e24e9d224bd3"
+ },
"source": [
- "As we can see in the figure above, the land-use categories have turned into numbers, instead of land-use categories described by a string value. \n",
+ "As we can see in the figure above, the land-use categories have turned into numbers, instead of land-use categories described by a string value.\n",
"\n",
"This is of course a lot harder to interpret. Let's re-do some parts to make sure we can properly link them back to the original data.\n",
"\n",
@@ -1173,9 +1421,11 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 38,
"id": "1882f6c1-9008-4815-b19f-7ec2771aaa55",
- "metadata": {},
+ "metadata": {
+ "id": "1882f6c1-9008-4815-b19f-7ec2771aaa55"
+ },
"outputs": [],
"source": [
"value_dict = dict(zip(landuse.landuse.unique(),np.arange(0,len(landuse.landuse.unique()),1)))"
@@ -1183,9 +1433,11 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 39,
"id": "27b572e0-d97b-47cb-8fa7-1a2c11afc8fa",
- "metadata": {},
+ "metadata": {
+ "id": "27b572e0-d97b-47cb-8fa7-1a2c11afc8fa"
+ },
"outputs": [],
"source": [
"value_dict['nodata'] = -1"
@@ -1194,22 +1446,48 @@
{
"cell_type": "markdown",
"id": "3cc4797d-9d48-4c83-a564-936d579dc256",
- "metadata": {},
+ "metadata": {
+ "id": "3cc4797d-9d48-4c83-a564-936d579dc256"
+ },
"source": [
"And we now use this dictionary to add a new column to the dataframe with the values:"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 40,
+ "id": "f14df583-5a87-4ea5-9712-915c2156c02e",
+ "metadata": {
+ "id": "f14df583-5a87-4ea5-9712-915c2156c02e"
+ },
+ "outputs": [],
+ "source": [
+ "landuse['landuse_value'] = landuse.landuse.apply(lambda x: value_dict[x])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "ff53e03e-058c-4397-a6ec-77ffc6f057ac",
+ "metadata": {
+ "id": "ff53e03e-058c-4397-a6ec-77ffc6f057ac"
+ },
+ "source": [
+ "Now let us use the make_geocube() function again to rasterize."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 41,
"id": "c05b7dd2-84df-428c-9c70-7d07aa745346",
- "metadata": {},
+ "metadata": {
+ "id": "c05b7dd2-84df-428c-9c70-7d07aa745346"
+ },
"outputs": [],
"source": [
"landuse_valued = make_geocube(\n",
- " vector_data=XXXX,\n",
- " output_crs=XXXX,\n",
- " resolution=(-XXXX, XXXX),\n",
+ " vector_data= ## add landuse,\n",
+ " output_crs= ## add CRS,\n",
+ " resolution= ## add resolution,\n",
" categorical_enums={'landuse_value': landuse.landuse_value.drop_duplicates().values.tolist()\n",
"}\n",
")"
@@ -1218,35 +1496,48 @@
{
"cell_type": "markdown",
"id": "510409be-3b7f-4f75-848b-4146bee15338",
- "metadata": {},
+ "metadata": {
+ "id": "510409be-3b7f-4f75-848b-4146bee15338"
+ },
"source": [
"And let's use the original `color_dict` dictionary to find the right hex codes for each of the land-use categories"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 42,
"id": "458b393b-47e8-44a3-bebc-148b4d764401",
- "metadata": {},
+ "metadata": {
+ "id": "458b393b-47e8-44a3-bebc-148b4d764401"
+ },
"outputs": [],
"source": [
"unique_classes = landuse.landuse.drop_duplicates().values.tolist()\n",
- "colormap_raster = [color_dict[lu_class] for lu_class in unique_classes] "
+ "colormap_raster = [color_dict[lu_class] for lu_class in unique_classes]"
]
},
{
"cell_type": "markdown",
"id": "8197f4b9-9c0f-4999-89f3-1ea657888193",
- "metadata": {},
+ "metadata": {
+ "id": "8197f4b9-9c0f-4999-89f3-1ea657888193"
+ },
"source": [
"To plot the new result:"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 43,
"id": "a5eb9d2c-2435-485f-8daa-6c54e29e8fd0",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 830
+ },
+ "id": "a5eb9d2c-2435-485f-8daa-6c54e29e8fd0",
+ "outputId": "2558af6a-329d-4ac5-dcc4-0793dbab498b"
+ },
"outputs": [],
"source": [
"fig, ax = plt.subplots(1, 1,figsize=(14,10))\n",
@@ -1264,23 +1555,24 @@
},
{
"cell_type": "markdown",
- "id": "9e42dd3a-c0c3-496f-bbc4-dbdf1f3901d1",
+ "id": "b999fe12",
"metadata": {},
"source": [
- "\n",
- "Question 9: In the rasterization process, we use the `.make_geocube()` function. Please elaborate on the following: i)why is it important to specify the right coordinate system? What could happen if you choose the wrong coordinate system? ii) which resolution did you choose and why? iii)Why did the first result did not give us the right output with the correct colors? How did you solve this? \n",
- "
"
+ "But this is not entirely how we want it yet. We want to make sure we assign the correct colors to the correct categories."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 44,
"id": "d8789a06-d041-4d6f-aab7-48032ac8c8d1",
- "metadata": {},
+ "metadata": {
+ "id": "d8789a06-d041-4d6f-aab7-48032ac8c8d1"
+ },
"outputs": [],
"source": [
+ "# we first identify all the unique classes in the landuse dataset and assign them a unique color\n",
"unique_classes = landuse.landuse.drop_duplicates().values.tolist()\n",
- "colormap_raster = [color_dict[lu_class] for lu_class in unique_classes] \n",
+ "colormap_raster = [color_dict[lu_class] for lu_class in unique_classes]\n",
"color_dict_raster = dict(zip(np.arange(-1,len(landuse.landuse.unique())+1,1),['#ffffff']+colormap_raster))\n",
"\n",
"# We create a colormar from our list of colors\n",
@@ -1303,7 +1595,9 @@
{
"cell_type": "markdown",
"id": "f36beb42-1da5-4da0-bb02-b61604b0852f",
- "metadata": {},
+ "metadata": {
+ "id": "f36beb42-1da5-4da0-bb02-b61604b0852f"
+ },
"source": [
"Let's plot the map again!"
]
@@ -1312,7 +1606,14 @@
"cell_type": "code",
"execution_count": null,
"id": "def51260-e57c-4957-8263-2b43a83a134f",
- "metadata": {},
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 807
+ },
+ "id": "def51260-e57c-4957-8263-2b43a83a134f",
+ "outputId": "59db0647-2d46-435e-88ab-46f85775e791"
+ },
"outputs": [],
"source": [
"fig, ax = plt.subplots(1, 1,figsize=(14,10))\n",
@@ -1335,12 +1636,26 @@
"fig.delaxes(fig.axes[1])"
]
},
+ {
+ "cell_type": "markdown",
+ "id": "9e42dd3a-c0c3-496f-bbc4-dbdf1f3901d1",
+ "metadata": {
+ "id": "9e42dd3a-c0c3-496f-bbc4-dbdf1f3901d1"
+ },
+ "source": [
+ "\n",
+ "Question 7: In the rasterization process, we use the `.make_geocube()` function. Please elaborate on the following: i)why is it important to specify the right coordinate system? What could happen if you choose the wrong coordinate system? ii) which resolution did you choose and why? iii)Why did the first result did not give us the right output with the correct colors? How did we solve this? Are you able to explain the code in your own words?\n",
+ "
"
+ ]
+ },
{
"cell_type": "markdown",
"id": "318a2267-4057-485c-85f1-1929896ad9ee",
- "metadata": {},
+ "metadata": {
+ "id": "318a2267-4057-485c-85f1-1929896ad9ee"
+ },
"source": [
- "## 5. Perform a raster-based damage assessment using OSM and Corine Land Cover\n",
+ "### 5. Perform a raster-based damage assessment Corine Land Cover\n",
"" ] }, @@ -1348,12 +1663,12 @@ "cell_type": "markdown", "id": "5dfabe80-3c8f-4e3a-a411-a29349c2d75b", "metadata": { - "id": "Agxq2HqY-g4H" + "id": "5dfabe80-3c8f-4e3a-a411-a29349c2d75b" }, "source": [ "To calculate the potential damage to both windstorms and floods, we use stage-damage curves, which relate the intensity of the hazard to the fraction of maximum damage that can be sustained by a certain land use. As you can see on the Corine Land Cover map that we just plotted, there are a lot of land use classes (44), though not all will suffer damage from either the windstorm or the flood event. For each of the land-use classes a curve and a maximum damage number are assigned.\n", "\n", - "To Assess the damage for both the flood and windstorm event, we are going to make use of the [DamageScanner](https://damagescanner.readthedocs.io/en/latest/), which is a tool to calculate potential flood damages based on inundation depth and land use using depth-damage curves in the Netherlands. The DamageScanner was originally developed for the 'Netherlands Later' project [(Klijn et al., 2007)](https://www.rivm.nl/bibliotheek/digitaaldepot/WL_rapport_Overstromingsrisicos_Nederland.pdf). The original land-use classes were based on the Land-Use Scanner in order to evaluate the effect of future land-use change on flood damages. We have tailored the input of the DamageScanner to make sure it can estimate the damages using Corine Land Cover.\n", + "To assess the damage for both the flood and windstorm event, we are going to make use of the [DamageScanner](https://damagescanner.readthedocs.io/en/latest/), which is a tool to calculate potential flood damages based on inundation depth and land use using depth-damage curves in the Netherlands. The DamageScanner was originally developed for the 'Netherlands Later' project [(Klijn et al., 2007)](https://www.rivm.nl/bibliotheek/digitaaldepot/WL_rapport_Overstromingsrisicos_Nederland.pdf). The original land-use classes were based on the Land-Use Scanner in order to evaluate the effect of future land-use change on flood damages. We have tailored the input of the DamageScanner to make sure it can estimate the damages using Corine Land Cover.\n", "\n", "Because the simplicity of the model, we can use this for any raster-based hazard map with some level of intensity. Hence, we can use it for both hazards." ] @@ -1362,45 +1677,45 @@ "cell_type": "markdown", "id": "ed5b5f52-bbad-42a6-acd5-1ea0248591f8", "metadata": { - "id": "5m_RAcp_fraF" + "id": "ed5b5f52-bbad-42a6-acd5-1ea0248591f8" }, "source": [ "
\n",
- "Question 10: Describe in your own words what the `DamageScanner()` function does. Please walk us through the different steps. Which inputs do you need to be able to run this damage assessment?\n",
+ "Question 8: Describe in your own words what the `DamageScanner()` function does. Please walk us through the different steps. Which inputs do you need to be able to run this damage assessment?\n",
"
"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 85,
"id": "94d33d66-6bf6-43b2-8c24-3e1a1069af2b",
"metadata": {
- "id": "jDrTp44Q-g4H"
+ "id": "94d33d66-6bf6-43b2-8c24-3e1a1069af2b"
},
"outputs": [],
"source": [
"def DamageScanner(landuse_map,inun_map,curve_path,maxdam_path,cellsize=100):\n",
- " \n",
+ "\n",
" # load land-use map\n",
" landuse = landuse_map.copy()\n",
- " \n",
+ "\n",
" # Load inundation map\n",
" inundation = inun_map.copy()\n",
- " \n",
- " inundation = np.nan_to_num(inundation) \n",
+ "\n",
+ " inundation = np.nan_to_num(inundation)\n",
"\n",
" # Load curves\n",
" if isinstance(curve_path, pd.DataFrame):\n",
- " curves = curve_path.values \n",
+ " curves = curve_path.values\n",
" elif isinstance(curve_path, np.ndarray):\n",
" curves = curve_path\n",
"\n",
" #Load maximum damages\n",
" if isinstance(maxdam_path, pd.DataFrame):\n",
- " maxdam = maxdam_path.values \n",
+ " maxdam = maxdam_path.values\n",
" elif isinstance(maxdam_path, np.ndarray):\n",
" maxdam = maxdam_path\n",
- " \n",
+ "\n",
" # Speed up calculation by only considering feasible points\n",
" inun = inundation * (inundation>=0) + 0\n",
" inun[inun>=curves[:,0].max()] = curves[:,0].max()\n",
@@ -1426,8 +1741,8 @@
" alldamage[landuse==n] = damage\n",
"\n",
" # create pandas dataframe with output\n",
- " loss_df = pd.DataFrame(damagebin.astype(float),columns=['landuse','losses','area','avg_depth']).groupby('landuse').sum()\n",
- " \n",
+ " loss_df = pd.DataFrame(damagebin.astype(float),columns=['landuse','losses','area','avg_intensity']).groupby('landuse').sum()\n",
+ "\n",
" # return output\n",
" return loss_df.sum().values[0],loss_df"
]
@@ -1436,38 +1751,38 @@
"cell_type": "markdown",
"id": "146c6b79-ce68-436b-a7fb-8a889528f513",
"metadata": {
- "id": "Y7PB8oJz-g4H"
+ "id": "146c6b79-ce68-436b-a7fb-8a889528f513"
},
"source": [
- "### Windstorm Damage\n",
+ "#### Windstorm Damage\n",
"---\n",
"To estimate the potential damage of our windstorm, we use the vulnerability curves developed by [Yamin et al. (2014)](https://www.sciencedirect.com/science/article/pii/S2212420914000466). Following [Yamin et al. (2014)](https://www.sciencedirect.com/science/article/pii/S2212420914000466), we will apply a sigmoidal vulnerability function satisfying two constraints: (i) a minimum threshold for the occurrence of damage with an upper bound of 100% direct damage; (ii) a high power-law function for the slope, describing an increase in damage with increasing wind speeds. Due to the limited amount of vulnerability curves available for windstorm damage, we will use the damage curve that represents low-rise *reinforced masonry* buildings for all land-use classes that may contain buildings. Obviously, this is a large oversimplification of the real world, but this should be sufficient for this exercise. When doing a proper stand-alone windstorm risk assessment, one should take more effort in collecting the right vulnerability curves for different building types. "
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 47,
"id": "e0e63daa-636b-4a1e-ba6c-5bb78e56f290",
"metadata": {
- "id": "-RxvAEQh-g4H",
+ "id": "e0e63daa-636b-4a1e-ba6c-5bb78e56f290",
"tags": []
},
"outputs": [],
"source": [
- "wind_curves = pd.read_excel(\"https://github.com/VU-IVM/UNIGIS_ProgrammingGIS/tree/main/TAA2/damage_curves.xlsx\",sheet_name='wind_curves')\n",
- "maxdam = pd.read_excel(\"https://github.com/VU-IVM/UNIGIS_ProgrammingGIS/tree/main/TAA2/damage_curves.xlsx\",sheet_name='maxdam')"
+ "wind_curves = pd.read_excel(\"https://github.com/VU-IVM/UNIGIS_ProgrammingGIS/raw/main/TAA2/damage_curves.xlsx\",sheet_name='wind_curves')\n",
+ "maxdam = pd.read_excel(\"https://github.com/VU-IVM/UNIGIS_ProgrammingGIS/raw/main/TAA2/damage_curves.xlsx\",sheet_name='maxdam')"
]
},
{
"cell_type": "markdown",
"id": "beb4f312-c4fb-4169-b563-e5878dfd95e2",
"metadata": {
- "id": "uLZ7vl1w-g4H"
+ "id": "beb4f312-c4fb-4169-b563-e5878dfd95e2"
},
"source": [
"Unfortunately, we run into a *classic* problem when we want to overlay the windstorm data with the Corine Land Cover data. The windstorm data is not only stored in a different coordinate system (we had to convert it from **EPSG:4326** to **EPSG:3035**), it is in a different resolution (**1km** instead of the **100m** of Corine Land Cover). \n",
"\n",
- "Let's first have a look how our clipped data look's like. If you have decided to use Gelderland, you will see that we have 102 columns (our Lattitude/lat) and 74 rows (our Longitude/lon). If you scroll above to our Corine Land Cover data, you see that dimensions are different: 1270 columns (Lattitude/lat/x) and 870 rows (Longitude/lon/y). "
+ "Let's first have a look how our clipped data look's like. If you have decided to use Kampen, you will see that we have 12 columns (our Lattitude/lat) and 9 rows (our Longitude/lon). If you scroll above to our Corine Land Cover data, you see that dimensions are different: 147 columns (Lattitude/lat/x) and 111 rows (Longitude/lon/y)."
]
},
{
@@ -1479,8 +1794,8 @@
"base_uri": "https://localhost:8080/",
"height": 291
},
- "id": "gG2OXOySj8Ra",
- "outputId": "67135491-52de-4571-f8c9-7b6a74e19b66"
+ "id": "57bc4345-b1e5-45c7-8aa2-95ef39b1ea78",
+ "outputId": "47728b1f-574e-4f66-d050-57b5c568b5c6"
},
"outputs": [],
"source": [
@@ -1491,7 +1806,7 @@
"cell_type": "markdown",
"id": "e1688219-1647-442e-8ed7-e4277ca75a16",
"metadata": {
- "id": "igfFBqcK-g4H"
+ "id": "e1688219-1647-442e-8ed7-e4277ca75a16"
},
"source": [
"The first thing we are going to do is try to make sure our data will be in the correct resolution (moving from **1km** to **100m**). To do so, we will use the `rio.reproject()` function. You will see that specify the resolution as **100**. Because **EPSG:3035** is a coordinate system in meters, we can simply use meters to define the resolution. We use the `rio.clip()` function to make sure we clip it again to our area of interest. The function below (`match_rasters`) will do the hard work for us. Please note all the input variables to understand what's happening."
@@ -1499,10 +1814,10 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 49,
"id": "d960af61-b324-459e-9e4c-da2f70bf7ecf",
"metadata": {
- "id": "Kud2CWEDhz1O"
+ "id": "d960af61-b324-459e-9e4c-da2f70bf7ecf"
},
"outputs": [],
"source": [
@@ -1533,7 +1848,7 @@
" - The land use variable with matching resolution and dimensions to the hazard variable.\n",
" - The hazard variable clipped to the extent of the land use variable, with matching resolution and dimensions.\n",
" \"\"\"\n",
- " \n",
+ "\n",
" # Set the crs of the hazard variable to haz_crs\n",
" hazard.rio.write_crs(haz_crs, inplace=True)\n",
"\n",
@@ -1585,7 +1900,7 @@
"cell_type": "markdown",
"id": "9f9ccb0d-2644-45ea-b435-79d83cd12a8e",
"metadata": {
- "id": "Vkf6YKPZ-g4I"
+ "id": "9f9ccb0d-2644-45ea-b435-79d83cd12a8e"
},
"source": [
"Now let's run the `match_rasters` function and let it do its magic."
@@ -1593,10 +1908,10 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 50,
"id": "d5c3409f-5c41-4ffa-b881-fa166eec2521",
"metadata": {
- "id": "v8NW3c1Q-g4I"
+ "id": "d5c3409f-5c41-4ffa-b881-fa166eec2521"
},
"outputs": [],
"source": [
@@ -1612,7 +1927,7 @@
"cell_type": "markdown",
"id": "aa2fdc04-2327-47d2-b43a-521dc95b6882",
"metadata": {
- "id": "GgcwJe_6nJip"
+ "id": "aa2fdc04-2327-47d2-b43a-521dc95b6882"
},
"source": [
"And let's have a look if the two rasters are now the same extend:\n"
@@ -1627,8 +1942,8 @@
"base_uri": "https://localhost:8080/",
"height": 291
},
- "id": "vzMbkiSLldlQ",
- "outputId": "6e73f8b1-33ad-4a7c-b95c-d320b6c75439"
+ "id": "7ca69a83-350b-4d0e-adea-d6a00b9dfc38",
+ "outputId": "9f8a60c0-ca86-4979-bd06-95c97690b8fb"
},
"outputs": [],
"source": [
@@ -1644,8 +1959,8 @@
"base_uri": "https://localhost:8080/",
"height": 291
},
- "id": "DXnxCBS_ldWg",
- "outputId": "ec49b756-0fd9-4d49-f9ff-9f68a52bf83c"
+ "id": "0b0216bd-6497-4a0a-8903-31d46599a854",
+ "outputId": "63f88a2f-4b3a-410e-94d8-9cb85e6e6719"
},
"outputs": [],
"source": [
@@ -1656,7 +1971,7 @@
"cell_type": "markdown",
"id": "50f3a8c5-aac5-4def-b79f-8e20be1cc822",
"metadata": {
- "id": "6123eX9C-g4J"
+ "id": "50f3a8c5-aac5-4def-b79f-8e20be1cc822"
},
"source": [
"It worked! And to double check, let's also plot it:"
@@ -1669,25 +1984,25 @@
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
- "height": 324
+ "height": 506
},
- "id": "Aeay_slW-g4J",
- "outputId": "11424ad3-2a00-49db-db13-b4db8e73671e"
+ "id": "58536a6a-6e5b-40cd-8d14-dc453d781405",
+ "outputId": "f5764fa2-b419-4035-d1eb-3cab1c99b316"
},
"outputs": [],
"source": [
- "windstorm.FX.plot()"
+ "# plot the data"
]
},
{
"cell_type": "markdown",
"id": "8e445352-2695-4d54-9abe-e3215817f355",
"metadata": {
- "id": "JlZF-cs4gSuu"
+ "id": "8e445352-2695-4d54-9abe-e3215817f355"
},
"source": [
"\n",
- "Question 11: Describe the various steps you have taken to make sure that the windstorm map is now exactly the same extent as the corine land cover map. Feel free to include lines of code in your answer and also describe the different functions you have used along the way.\n",
+ "Question 9: Describe the various steps you have taken to make sure that the windstorm map is now exactly the same extent as the corine land cover map. Feel free to include lines of code in your answer and also describe the different functions you have used along the way.\n",
"
"
]
},
@@ -1695,7 +2010,7 @@
"cell_type": "markdown",
"id": "4959f306-df08-4548-8470-9083c814b203",
"metadata": {
- "id": "LW158xPh-g4J"
+ "id": "4959f306-df08-4548-8470-9083c814b203"
},
"source": [
"Now its finally time to do our damage assessment! To do so, we need to convert our data to `numpy.arrays()` to do our calculation:"
@@ -1703,10 +2018,10 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 54,
"id": "f0f82a7c-f2b1-43dd-bcb6-4a9d24738d75",
"metadata": {
- "id": "QZIzWIeP-g4J"
+ "id": "f0f82a7c-f2b1-43dd-bcb6-4a9d24738d75"
},
"outputs": [],
"source": [
@@ -1714,27 +2029,11 @@
"wind_map = windstorm['FX'].to_numpy()[0,:,:]"
]
},
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "53d83a65-c63b-427b-b950-2f4674fe8b82",
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "aBqqRqbkmA1Y",
- "outputId": "709d6c91-4ad9-4e27-e6a9-e6201df32dc7"
- },
- "outputs": [],
- "source": [
- "wind_map.shape"
- ]
- },
{
"cell_type": "markdown",
"id": "aca9bbe2-95f2-4b0d-9fb7-bb0ffc94ecef",
"metadata": {
- "id": "J9QHyhSU-g4J"
+ "id": "aca9bbe2-95f2-4b0d-9fb7-bb0ffc94ecef"
},
"source": [
"And remember that our windstorm data was stored in **m/s**. Hence, we need to convert it to **km/h**:"
@@ -1742,21 +2041,21 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 56,
"id": "11ddca33-7360-4efb-84f2-7613fd360ca3",
"metadata": {
- "id": "GqdUCXD_-g4J"
+ "id": "11ddca33-7360-4efb-84f2-7613fd360ca3"
},
"outputs": [],
"source": [
- "wind_map_kmh = wind_map*XXX"
+ "wind_map_kmh = wind_map* #convert to km/h"
]
},
{
"cell_type": "markdown",
"id": "53b5ac7e-e817-49e2-b588-614e997adf8a",
"metadata": {
- "id": "ln7NqRB1-g4J"
+ "id": "53b5ac7e-e817-49e2-b588-614e997adf8a"
},
"source": [
"And now let's run the DamageScanner to obtain the damage results"
@@ -1764,10 +2063,10 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 57,
"id": "31da4031-fdd4-485f-af99-30fa49fdabe0",
"metadata": {
- "id": "y_g0pj1h-g4J",
+ "id": "31da4031-fdd4-485f-af99-30fa49fdabe0",
"tags": []
},
"outputs": [],
@@ -1784,8 +2083,8 @@
"base_uri": "https://localhost:8080/",
"height": 1000
},
- "id": "-6DbFD_JeFA1",
- "outputId": "fb251350-8885-4dde-e665-893fa04cde6e"
+ "id": "a717a4a2-8d6b-436b-b855-bebc7835d0b3",
+ "outputId": "1f8aaf42-367a-4073-bb05-0342e2a42898"
},
"outputs": [],
"source": [
@@ -1793,775 +2092,675 @@
]
},
{
- "cell_type": "code",
- "execution_count": null,
- "id": "e4c4ad75-a555-414d-bb64-6af5f10a2eb1",
- "metadata": {},
- "outputs": [],
+ "cell_type": "markdown",
+ "id": "5UNySYvk-g4J",
+ "metadata": {
+ "id": "5UNySYvk-g4J",
+ "tags": []
+ },
"source": [
- "! @TASK: Let students also perform risk assessment based on rasterized OSM land use data?? Code needs to be written for that. Or think about a really good question that we can ask here about this?"
+ "#### Flood Damage\n",
+ "---\n",
+ "To Assess the flood damage, we are again going to make use of the [DamageScanner](https://damagescanner.readthedocs.io/en/latest/). The Corine Land Cover data is widely used in European flood risk assessments. As such, we can simply make use of pre-developed curves. We are using the damage curves as developed by Huizinga et al. (2007). Again, let's first load the maximum damages and the depth-damage curves:"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "id": "38d46699-d759-48e3-a5ef-09d45ec8e41a",
- "metadata": {},
+ "execution_count": 86,
+ "id": "ua2xyAGW-g4J",
+ "metadata": {
+ "id": "ua2xyAGW-g4J"
+ },
"outputs": [],
- "source": []
- },
- {
- "cell_type": "markdown",
- "id": "28e483d0-06b1-432f-86e6-d7a68e9811a4",
- "metadata": {},
"source": [
- "## 6. Extracting high-resolution data from OpenStreetMap\n",
- "" + "flood_curves = pd.read_excel(\"https://github.com/ElcoK/BigData_AED/raw/main/week5/damage_curves.xlsx\",sheet_name='flood_curves',engine='openpyxl')\n", + "maxdam = pd.read_excel(\"https://github.com/ElcoK/BigData_AED/raw/main/week5/damage_curves.xlsx\",sheet_name='maxdam')" ] }, { "cell_type": "markdown", - "id": "502b596d-ad9f-4e42-bc2f-e889d216f0e7", - "metadata": {}, - "source": [ - "### Extracting buildings from OpenStreetMap\n", - "---" - ] - }, - { - "cell_type": "markdown", - "id": "4f42ba03-9c17-465d-97eb-77df502dd8ee", + "id": "HT54wRvs-g4K", "metadata": { - "id": "WMOSa6JF47DS" + "id": "HT54wRvs-g4K" }, "source": [ - "There is a lot more data to extract from OpenStreetMap besides land-use information. Let's extract some building data. To do so, we use the *\"building\"* tag." + "And convert our data to `numpy.arrays()` to do our calculation:" ] }, { "cell_type": "code", - "execution_count": null, - "id": "92f86ddc-ce27-4753-9e70-af6df03d7850", - "metadata": {}, + "execution_count": 87, + "id": "qzXKNmg2-g4K", + "metadata": { + "id": "qzXKNmg2-g4K" + }, "outputs": [], "source": [ - "tags = {\"building\": True}\n", - "buildings = ox.features_from_place(place_name, tags)" + "landuse_map = CLC_region['band_data'].to_numpy()\n", + "flood_map = flood_map_area['band_data'].to_numpy()" ] }, { "cell_type": "markdown", - "id": "772d95a6-c2ce-48a6-a7d0-bc1f4b77c323", - "metadata": {}, + "id": "ttGra99k-g4K", + "metadata": { + "id": "ttGra99k-g4K" + }, "source": [ - "In case the above does not work, you can continue the assignment by using the code below (make sure you remove the hashtags to run it). If you decide to use the data as specified below, also change the map at the start to 'Kampen'." + "And now let's run the DamageScanner to obtain the damage results" ] }, { "cell_type": "code", - "execution_count": 64, - "id": "9a7e508a-b20a-416c-8f2a-c8dd36a60389", - "metadata": {}, + "execution_count": 88, + "id": "2qL8UATu-g4K", + "metadata": { + "id": "2qL8UATu-g4K" + }, "outputs": [], "source": [ - "# remote_url = https://github.com/VU-IVM/UNIGIS_ProgrammingGIS/tree/main/TAA2'\n", - "# file = 'kampen_buildings.gpkg'\n", - "# \n", - "# #request.urlretrieve(remote_url, file)\n", - "# buildings = gpd.GeoDataFrame.from_file('kampen_buildings.gpkg')" + "flood_damage_CLC = DamageScanner(landuse_map,flood_map,flood_curves,maxdam)[1]" ] }, { - "cell_type": "markdown", - "id": "53e45bdb-36a3-4efd-b0e4-111f5fd858fe", + "cell_type": "code", + "execution_count": null, + "id": "WnWu6AMUeFA2", "metadata": { - "id": "bon7osXA47DT" + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "WnWu6AMUeFA2", + "outputId": "9b3076fd-a548-462d-a9e1-f532d644ed04" }, + "outputs": [], "source": [ - "Now let's see what information is actually extracted:" + "flood_damage_CLC" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "26a60806-c927-4d0c-ab04-dc07c9c7a688", + "cell_type": "markdown", + "id": "61359310", "metadata": {}, - "outputs": [], "source": [ - "buildings.head()" + "
\n",
+ "Question 10: Create a plot yourself to compare the results of both the windstorm and flood damage results. This could be a simple barplot, for example.\n",
+ "
"
]
},
{
"cell_type": "markdown",
- "id": "2f7f7ff1-ed38-4dca-b2ad-f09472566879",
+ "id": "tpS3A5DA5NiA",
"metadata": {
- "id": "p8dhZx1n47DT"
+ "id": "tpS3A5DA5NiA"
},
"source": [
- "As you notice in the output of the cell above, there are many columns which just contain \"NaN\". And there even seem to be to many columns to even visualize properly in one view.\n",
+ "Now let's try to do this again with the OpenStreetMap data. It would be most convenient to use the same damage curves, so we want to couple our OSM land-use information to the flood curves.\n",
"\n",
- "Let's check what information is collected for the different buildings:"
+ "Let's first have a look at the column values:"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "b78b1253-16b0-4513-8903-966b1816715f",
+ "id": "V9W8ngNm5dI_",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
- "executionInfo": {
- "elapsed": 35,
- "status": "ok",
- "timestamp": 1675087861529,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "W3ZpsIkO47DT",
- "outputId": "385616f8-25fe-4675-f5f0-16d7fa9b0df7"
+ "id": "V9W8ngNm5dI_",
+ "outputId": "71a6bdf8-69d1-4f12-f0c6-02e6d185ab9c"
},
"outputs": [],
"source": [
- "buildings.columns"
+ "flood_curves.columns"
]
},
{
"cell_type": "markdown",
- "id": "772be8a9-7ab6-4996-9311-e165fcc6c8ed",
+ "id": "Q_FkDCtN5se3",
"metadata": {
- "id": "sH7NTKENA4WO"
+ "id": "Q_FkDCtN5se3"
},
"source": [
- "\n",
- "Question 12: Let's have a look at the extracted building information. Please describe in your own words the information it contains. Is there specific information that suprises you to see, and do you think anything is missing that you expected? \n",
- "
"
+ "Ok this does not say anything to us yet. So let's have a look at the list of classes they are refering to:\n",
+ "\n",
+ "\n",
+ "\n",
+ ""
]
},
{
"cell_type": "markdown",
- "id": "d7077034-4cfe-436a-b5b2-64fc77f24ae5",
+ "id": "C7zUFEoZ6Yzn",
"metadata": {
- "id": "Z37JRRc747DT"
+ "id": "C7zUFEoZ6Yzn"
},
"source": [
- "One interesting column is called `start_date`. This shows the building year per building. \n",
- "\n",
- "Let's explore this year of building a bit more.\n",
- "\n",
- "First, it would be interesting to get an idea how many buildings are build in each year through using the `value_counts()` function. Normally, that functions ranks the values in descending order (high to low). We are more interested in how this has developed over time. So we use the `sort_index()` function to sort the values by year. Add these two functions in the cell below."
+ "So the first column of our damage curves relates to \"111: Continuous urban fabric\", and so on. Now let's have a look at the land uses we have within our OSM data. Can you find all the unique values in our 'landuse' column of the **landuse** dataframe?"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "e27c9a71-68b4-4adc-a1b4-52ff94275a6f",
+ "id": "wPX9jZWd6sa-",
"metadata": {
- "executionInfo": {
- "elapsed": 924,
- "status": "ok",
- "timestamp": 1675087884735,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
+ "colab": {
+ "base_uri": "https://localhost:8080/"
},
- "id": "lNSe826i47DT"
+ "id": "wPX9jZWd6sa-",
+ "outputId": "ba5fea0e-2791-4c81-cdc0-001a6703251f"
},
"outputs": [],
- "source": [
- "building_year = buildings.start_date. XXXX"
- ]
+ "source": []
},
{
"cell_type": "markdown",
- "id": "e9e9c436-fd01-46f7-8f73-44eccafe7816",
+ "id": "qUJ7GwF_6vGf",
"metadata": {
- "id": "X6b_T5xs47DU"
+ "id": "qUJ7GwF_6vGf"
},
"source": [
- "There is not better way to further explore this years than through plotting it. Don't forget to add things such as a x label, y label and title. Have a look at some of the matplotlib [tutorials](https://matplotlib.org/stable/tutorials/introductory/quick_start.html). Note that you need to look at the code that also uses subplots and where they use the `ax` option."
+ "Our next step would be to attach a value to each of the land-use classes that corresponds to the value of each column (to make sure we have a damage curve linked to each land-use class)"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "id": "35a6ee4b-f664-4955-9781-86d56920b3e4",
+ "execution_count": 115,
+ "id": "5pGFDGjP6ujm",
"metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 1000
- },
- "executionInfo": {
- "elapsed": 1636,
- "status": "ok",
- "timestamp": 1675087889714,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "fmk6m78I47DU",
- "outputId": "78f3ea6a-284d-495f-b596-12db32305128"
+ "id": "5pGFDGjP6ujm"
},
"outputs": [],
"source": [
- "fig,ax = plt.subplots(1,1,figsize=(5,18))\n",
- "\n",
- "building_year.plot(kind='barh',ax=ax)\n",
- "\n",
- "ax.tick_params(axis='y', which='major', labelsize=7)"
+ "landuse_value_clc = [0, ## complete the list, where each value corresponds to a landuse class in the flood_curves dataframe]"
]
},
{
- "cell_type": "markdown",
- "id": "7e5ce5b6-f7b6-47a4-b4b7-2cac7cc933bf",
+ "cell_type": "code",
+ "execution_count": 116,
+ "id": "IpudoRfT7pru",
"metadata": {
- "id": "AwRyhvNeBn0G"
+ "id": "IpudoRfT7pru"
},
+ "outputs": [],
"source": [
- "\n",
- "Question 13: Please upload a figure that shows the development of building stock over the years in your region of interest. Make sure it contains all the necessary elements (labels on the axis, title, etc.)\n",
- "
"
+ "value_dict_clc = dict(zip(landuse.landuse.unique(),landuse_value_clc))\n",
+ "value_dict_clc['nodata'] = ## add a nodata value\n",
+ "landuse['landuse_value_clc'] = landuse.landuse.apply(lambda x: value_dict_clc[x])"
]
},
{
"cell_type": "markdown",
- "id": "089f6011-87fd-464a-ad8b-52ad810d8b75",
- "metadata": {
- "id": "wrcM1p-m47DU"
- },
+ "id": "24d86032",
+ "metadata": {},
"source": [
- "What we also noticed is that quite some buildings are identified as 'yes'. This is not very useful as it does not really say much about the use of the building. \n",
- "\n",
- "Let's see for how many buildings this is the case: "
+ "and now rasterize the OSM landuse data again to make it matching!"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "id": "ca163a97-54bd-4050-bea6-2721d024a1eb",
+ "execution_count": 132,
+ "id": "38d46699-d759-48e3-a5ef-09d45ec8e41a",
"metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "executionInfo": {
- "elapsed": 205,
- "status": "ok",
- "timestamp": 1675087945407,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "_4eKR2Bo47DU",
- "outputId": "54e29c44-9717-4eee-984f-9555659317be"
+ "id": "38d46699-d759-48e3-a5ef-09d45ec8e41a"
},
"outputs": [],
"source": [
- "buildings.building.value_counts()"
+ "landuse_clc = make_geocube(\n",
+ " vector_data= # add landuse,\n",
+ " output_crs= # add CRS,\n",
+ " resolution= # add resolution (this should be somewhat similar as the resolution of the flood map),\n",
+ " categorical_enums={'landuse_value': landuse.landuse_value.drop_duplicates().values.tolist()\n",
+ "}\n",
+ ")"
]
},
{
"cell_type": "markdown",
- "id": "502fcb4a-cc93-46fa-bc1a-5dad7f4438bc",
+ "id": "126d4106",
"metadata": {},
"source": [
- "As you have seen from the `value_counts` function, there are quite a few buildings with only very few tags. You could either consider to not include them in your plot at all (for example by using the `isin` function or the `query` function, see also [here](https://stackoverflow.com/questions/12096252/use-a-list-of-values-to-select-rows-from-a-pandas-dataframe)), or rename them, similar to how you named the natural land cover classes for the land-use map. "
+ "And run the **match_rasters** function now with the new landuse map"
]
},
{
- "cell_type": "markdown",
- "id": "22598d8d-a73c-41b7-befd-f8abafc86162",
+ "cell_type": "code",
+ "execution_count": 134,
+ "id": "PL1evdBM4rex",
"metadata": {
- "id": "vF6WJ9_k47DU"
+ "id": "PL1evdBM4rex"
},
+ "outputs": [],
"source": [
- "Now let's visualize the buildings again. We need to create a similar color dictionary as we did for the land-use categories. Now its up to you to make it!"
+ "landuse_osm_clc, flood_map_osm = match_rasters(# add landuse_clc,\n",
+ " ## add landuse_clc,\n",
+ " haz_crs=3035,\n",
+ " lu_crs=# add landuse CRS,\n",
+ " resolution= #add desired resolution (I would pick the resolution of the flood map),\n",
+ " hazard_col=['band_data'])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b70655e2",
+ "metadata": {},
+ "source": [
+ "And now turn them into numpy arrays again"
]
},
{
"cell_type": "code",
- "execution_count": null,
- "id": "739a2389-8302-4e4f-8624-2beb0225ecfd",
+ "execution_count": 139,
+ "id": "2VNtw7HW9RXG",
"metadata": {
- "executionInfo": {
- "elapsed": 226,
- "status": "ok",
- "timestamp": 1675087956546,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "RYRxKXxd47DU"
+ "id": "2VNtw7HW9RXG"
},
"outputs": [],
"source": [
- "color_dict = { 'yes' : \"#f1134b\", \n",
- " 'house':'#f13013', \n",
- " 'industrial':'#0f045c',\n",
- " 'farm':'#fcfcb9', \n",
- " 'bungalow':'#f13013',\n",
- " 'service':'#CB8DDB' }"
+ "osm_landuse_map = landuse_osm_clc['band_data'] #complete function\n",
+ "osm_flood_map = flood_map_osm['band_data'] #complete function"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8398430d",
+ "metadata": {},
+ "source": [
+ "And run the DamageScanner"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "54672469-3961-4b67-860e-17f9320d1a62",
+ "id": "91662ad6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "flood_damage_CLC = DamageScanner(# fill the dammagescanner function)[1]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5626418e",
+ "metadata": {},
+ "source": [
+ "\n",
+ "Question 11: How do the results compare to the flood damages estimated with Corine Land Cover? Are you able to visualise the differences?\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "28e483d0-06b1-432f-86e6-d7a68e9811a4",
"metadata": {
- "executionInfo": {
- "elapsed": 232,
- "status": "ok",
- "timestamp": 1675087958726,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "YjRwC-K-47DU"
+ "id": "28e483d0-06b1-432f-86e6-d7a68e9811a4"
},
- "outputs": [],
"source": [
- "# Remove multiple keys from dictionary\n",
- "color_dict = {key: color_dict[key]\n",
- " for key in color_dict if key not in list(set(color_dict)-set(buildings.building.unique()))}\n",
- "\n",
- "map_dict = dict(zip(color_dict.keys(),[x for x in range(len(color_dict))]))\n",
- "buildings['col_landuse'] =buildings.building.apply(lambda x: color_dict[x])"
+ "### 6. Extracting high-resolution data from OpenStreetMap\n",
+ "" ] }, { "cell_type": "markdown", - "id": "0bbaef6d-6b28-45f4-ace9-6a5c9dec0f2f", + "id": "502b596d-ad9f-4e42-bc2f-e889d216f0e7", "metadata": { - "id": "wGh7rbnB47DU" + "id": "502b596d-ad9f-4e42-bc2f-e889d216f0e7" }, "source": [ - "And plot the figure in the same manner!" + "#### Extracting buildings from OpenStreetMap\n", + "---" + ] + }, + { + "cell_type": "markdown", + "id": "4f42ba03-9c17-465d-97eb-77df502dd8ee", + "metadata": { + "id": "4f42ba03-9c17-465d-97eb-77df502dd8ee" + }, + "source": [ + "There is a lot more data to extract from OpenStreetMap besides land-use information. Let's extract some building data. To do so, we use the *\"building\"* tag." ] }, { "cell_type": "code", - "execution_count": null, - "id": "7dad1b9f-855a-468e-9090-25d7ace2990d", + "execution_count": 60, + "id": "92f86ddc-ce27-4753-9e70-af6df03d7850", "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 616 - }, - "executionInfo": { - "elapsed": 3651, - "status": "ok", - "timestamp": 1675087966347, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "oDfamvIV47DV", - "outputId": "dbcee552-e955-4d2b-ddda-909ab4265105" + "id": "92f86ddc-ce27-4753-9e70-af6df03d7850" }, "outputs": [], "source": [ - "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n", - "\n", - "# add color scheme\n", - "color_scheme_map = list(color_dict.values())\n", - "cmap = LinearSegmentedColormap.from_list(name='landuse',\n", - " colors=color_scheme_map) \n", - "\n", - "# and plot the land-use map.\n", - "buildings.plot(color=buildings['col_landuse'],ax=ax,linewidth=0)\n", - "\n", - "# remove the ax labels\n", - "ax.set_xticks([])\n", - "ax.set_yticks([])\n", - "ax.set_axis_off()\n", - "\n", - "# add a legend:\n", - "legend_elements = []\n", - "for iter_,item in enumerate(color_dict):\n", - " legend_elements.append(Patch(facecolor=color_scheme_map[iter_],label=item)) \n", - "\n", - "ax.legend(handles=legend_elements,edgecolor='black',facecolor='#fefdfd',prop={'size':12},loc=(1.02,0.2)) \n", - "\n", - "# add a title\n", - "ax.set_title(place_name,fontweight='bold')" + "tags = {\"building\": True}\n", + "buildings = ox.features_from_place(place_name, tags)" ] }, { "cell_type": "markdown", - "id": "7fc1437d-8deb-4569-a3ee-9270676f0e41", + "id": "772d95a6-c2ce-48a6-a7d0-bc1f4b77c323", "metadata": { - "id": "IJalDDJvB6Yd" + "id": "772d95a6-c2ce-48a6-a7d0-bc1f4b77c323" }, "source": [ - "
\n",
- "Question 14: Please upload a figure of your building stock map of your region of interest. Make sure that the interpretation is clear. If necessary, merge multiple categories into one (i.e., when some categories only contain 1 or 2 buildings).\n",
- "
"
+ "In case the above does not work, you can continue the assignment by using the code below (make sure you remove the hashtags to run it). If you decide to use the data as specified below, also change the map at the start to 'Kampen'."
]
},
{
- "cell_type": "markdown",
- "id": "c59fa9dc-68f1-4d39-8adc-f917a0af3a04",
- "metadata": {},
+ "cell_type": "code",
+ "execution_count": 61,
+ "id": "9a7e508a-b20a-416c-8f2a-c8dd36a60389",
+ "metadata": {
+ "id": "9a7e508a-b20a-416c-8f2a-c8dd36a60389"
+ },
+ "outputs": [],
"source": [
- "### Extracting roads from OpenStreetMap\n",
- "---"
+ "# remote_url = https://github.com/VU-IVM/UNIGIS_ProgrammingGIS/tree/main/TAA2'\n",
+ "# file = 'kampen_buildings.gpkg'\n",
+ "#\n",
+ "# #request.urlretrieve(remote_url, file)\n",
+ "# buildings = gpd.GeoDataFrame.from_file('kampen_buildings.gpkg')"
]
},
{
"cell_type": "markdown",
- "id": "83eb8f23-6b4e-47ce-a1df-12ca10e90897",
+ "id": "53e45bdb-36a3-4efd-b0e4-111f5fd858fe",
"metadata": {
- "id": "HpWnKfIk47DV"
+ "id": "53e45bdb-36a3-4efd-b0e4-111f5fd858fe"
},
"source": [
- "Let's continue (and end) this tutorial with the core data in OpenStreetMap (it is even in the name): roads!\n",
- "\n",
- "Now, instead of using tags, we want to identify what type of roads we would like to extract. Let's first only extract roads that can be used to drive.\n",
- "\n",
- "The `graph_from_place()` function returns a `NetworkX` Graph element. You can read more about these graph elements in the introduction page of [NetworkX](https://networkx.org/documentation/stable/reference/introduction.html)."
+ "Now let's see what information is actually extracted:"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "73036bbd-08b1-4d2d-bc14-9efd48d4ec76",
- "metadata": {
- "executionInfo": {
- "elapsed": 13403,
- "status": "ok",
- "timestamp": 1675088179196,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
+ "id": "26a60806-c927-4d0c-ab04-dc07c9c7a688",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 460
},
- "id": "MpjxUadp47DV"
+ "id": "26a60806-c927-4d0c-ab04-dc07c9c7a688",
+ "outputId": "c4e83f03-b5cf-4421-836b-4262a804f6d1"
},
"outputs": [],
"source": [
- "G = ox.graph_from_place(place_name, network_type=\"drive\")"
+ "buildings.head()"
]
},
{
"cell_type": "markdown",
- "id": "6c1596cb-9a1b-4a67-87cd-cee4f687ad11",
+ "id": "2f7f7ff1-ed38-4dca-b2ad-f09472566879",
"metadata": {
- "id": "NA_HAmHx47DV"
+ "id": "2f7f7ff1-ed38-4dca-b2ad-f09472566879"
},
"source": [
- "Unfortunately, it is bit difficult to easily view all the roads within such a Graph element. To be able to explore the data, we are going to convert it to a `Geopandas GeoDataFrame`, using the `to_pandas_edgelist()` function."
+ "As you notice in the output of the cell above, there are many columns which just contain \"NaN\". And there even seem to be to many columns to even visualize properly in one view.\n",
+ "\n",
+ "Let's check what information is collected for the different buildings:"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "1b52effc-4eaf-44b5-b031-2481d2125197",
+ "id": "b78b1253-16b0-4513-8903-966b1816715f",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
- "executionInfo": {
- "elapsed": 15,
- "status": "ok",
- "timestamp": 1675088179197,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "4dMXZb_v47DV",
- "outputId": "e615d5f8-6b96-4a32-f114-ce38b1481b70"
+ "id": "b78b1253-16b0-4513-8903-966b1816715f",
+ "outputId": "b00fd26f-4ab5-4acb-8e8a-df34bc2b9fc3"
},
"outputs": [],
"source": [
- "roads = gpd.GeoDataFrame(nx.to_pandas_edgelist(G))"
+ "buildings.columns"
]
},
{
"cell_type": "markdown",
- "id": "7a3a191c-0d36-4bb5-8ed7-4f12bb05e1f5",
+ "id": "772be8a9-7ab6-4996-9311-e165fcc6c8ed",
+ "metadata": {
+ "id": "772be8a9-7ab6-4996-9311-e165fcc6c8ed"
+ },
+ "source": [
+ "\n",
+ "Question 12: Let's have a look at the extracted building information. Please describe in your own words the information it contains. Is there specific information that suprises you to see, and do you think anything is missing that you expected?\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d7077034-4cfe-436a-b5b2-64fc77f24ae5",
"metadata": {
- "id": "yds5NBlF47DV"
+ "id": "d7077034-4cfe-436a-b5b2-64fc77f24ae5"
},
"source": [
- "In some cases, roads are classified with more than one category. If that is the case, they are captured within a `list`. To overcome this issue, we specify that we want the entire `highway` column as a `string` dtype."
+ "One interesting column is called `start_date`. This shows the building year per building.\n",
+ "\n",
+ "Let's explore this year of building a bit more.\n",
+ "\n",
+ "First, it would be interesting to get an idea how many buildings are build in each year through using the `value_counts()` function. Normally, that functions ranks the values in descending order (high to low). We are more interested in how this has developed over time. So we use the `sort_index()` function to sort the values by year. Add these two functions in the cell below."
]
},
{
"cell_type": "code",
- "execution_count": null,
- "id": "d5f76461-aac1-4534-8a97-2abc841107b4",
- "metadata": {
- "executionInfo": {
- "elapsed": 294,
- "status": "ok",
- "timestamp": 1675088191403,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "NxTg1jb847DV"
+ "execution_count": 64,
+ "id": "e27c9a71-68b4-4adc-a1b4-52ff94275a6f",
+ "metadata": {
+ "id": "e27c9a71-68b4-4adc-a1b4-52ff94275a6f"
},
"outputs": [],
"source": [
- "roads.highway = roads.highway.astype('str')"
+ "building_year = buildings. #complete function"
]
},
{
"cell_type": "markdown",
- "id": "5080124a-c7f4-45f6-aced-a27f8793ef59",
+ "id": "e9e9c436-fd01-46f7-8f73-44eccafe7816",
"metadata": {
- "id": "rbzl5JPR47DW"
+ "id": "e9e9c436-fd01-46f7-8f73-44eccafe7816"
},
"source": [
- "Now we can create a plot to see how the road network is configured."
+ "There is not better way to further explore this years than through plotting it. Don't forget to add things such as a x label, y label and title. Have a look at some of the matplotlib [tutorials](https://matplotlib.org/stable/tutorials/introductory/quick_start.html). Note that you need to look at the code that also uses subplots and where they use the `ax` option. Perhaps also consider setting a threshold to make the figure more easy to visualise?"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "b0c2f581-4182-471e-823b-0280181494e0",
+ "id": "35a6ee4b-f664-4955-9781-86d56920b3e4",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
- "height": 575
- },
- "executionInfo": {
- "elapsed": 888,
- "status": "ok",
- "timestamp": 1675088196301,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
+ "height": 1000
},
- "id": "uyo3Ggpc47DW",
- "outputId": "e58c915a-b085-498c-e7ea-8d62f237d417"
+ "id": "35a6ee4b-f664-4955-9781-86d56920b3e4",
+ "outputId": "f51f5e22-aeda-4ec6-c0d4-7be7b6dbdd42"
},
"outputs": [],
"source": [
- "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n",
- "\n",
- "\n",
- "roads.plot(column='highway',legend=True,ax=ax,legend_kwds={'loc': 'lower right'});\n",
+ "fig,ax = plt.subplots(1,1,figsize=(5,18))\n",
"\n",
+ "building_year.plot(kind='barh',ax=ax)\n",
"\n",
- "# remove the ax labels\n",
- "ax.set_xticks([])\n",
- "ax.set_yticks([])\n",
- "ax.set_axis_off()"
+ "ax.tick_params(axis='y', which='major', labelsize=7)"
]
},
{
"cell_type": "markdown",
- "id": "82a98a1c-e8d3-4f84-80df-5e8577a3458b",
+ "id": "7e5ce5b6-f7b6-47a4-b4b7-2cac7cc933bf",
"metadata": {
- "id": "M18fuTWM47DW",
- "tags": []
+ "id": "7e5ce5b6-f7b6-47a4-b4b7-2cac7cc933bf"
+ },
+ "source": [
+ "\n",
+ "Question 13: Please upload a figure that shows the development of building stock over the years in your region of interest. Make sure it contains all the necessary elements (labels on the axis, title, etc.)\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "089f6011-87fd-464a-ad8b-52ad810d8b75",
+ "metadata": {
+ "id": "089f6011-87fd-464a-ad8b-52ad810d8b75"
},
"source": [
- "It would also be interesting to explore the network a little but more interactively. **OSMnx** has a function called `plot_graph_folium()`, which allow us to use the [folium](https://python-visualization.github.io/folium/quickstart.html#Getting-Started) package to plot data interactively on a map. "
+ "What we also noticed is that quite some buildings are identified as 'yes'. This is not very useful as it does not really say much about the use of the building.\n",
+ "\n",
+ "Let's see for how many buildings this is the case. Use the .value_counts function:"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "dcc7ab55-f4ce-4706-83a0-5b944c2bfd7d",
+ "id": "ca163a97-54bd-4050-bea6-2721d024a1eb",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
- "height": 866
- },
- "executionInfo": {
- "elapsed": 1720,
- "status": "ok",
- "timestamp": 1675088204394,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
+ "height": 993
},
- "id": "lpWHR0Ez47DW",
- "outputId": "38be3bea-b399-4e8f-a081-210baa559ef7"
+ "id": "ca163a97-54bd-4050-bea6-2721d024a1eb",
+ "outputId": "94a378d4-bd6a-4efb-94e6-076ed1f1afca"
},
"outputs": [],
"source": [
- "m1 = ox.plot_graph_folium(G, popup_attribute=\"highway\", weight=2, color=\"#8b0000\")\n",
- "m1"
+ "buildings.building. #complete function"
]
},
{
- "attachments": {},
"cell_type": "markdown",
- "id": "274d172a-62df-4095-987d-1de387104e8d",
+ "id": "502fcb4a-cc93-46fa-bc1a-5dad7f4438bc",
"metadata": {
- "id": "gm8NIAR947DW"
+ "id": "502fcb4a-cc93-46fa-bc1a-5dad7f4438bc"
},
"source": [
- "One of the exiting things we can do with this data is that we can compute and plot routes between two points on a map.\n",
- "\n",
- "Let's first select two random start and end points from the graph and compute the shortest route between them through using the `shortest_path()` function of the `NetworkX` package.\n",
- "\n",
- "The function `ox.nearest_nodes()` looks for the nearest point in your network based on a `X` and `Y` coordinate. For example, in the code below, the origin node is based on the northwestern corner of your bounding box, whereas the destination node is based on the coordinates of the southeastern corner of your bounding box. \n",
- "\n",
- "So this can also be rewritten as:\n",
- "\n",
- "```\n",
- "origin_node = ox.nearest_nodes(G,4.65465, 56.6778) \n",
- "destination_node = ox.nearest_nodes(G,4.61055, 59.5487) \n",
- "route = nx.shortest_path(G, origin_node, destination_node) \n",
- "```"
+ "As you have seen from the `value_counts` function, there are quite a few buildings with only very few tags. You could either consider to not include them in your plot at all (for example by using the `isin` function or the `query` function, see also [here](https://stackoverflow.com/questions/12096252/use-a-list-of-values-to-select-rows-from-a-pandas-dataframe)), or rename them, similar to how you named the natural land cover classes for the land-use map. Here, we filter the dataframe to include only specific classes of buildings."
]
},
{
- "cell_type": "markdown",
- "id": "6e1d7552-5ec6-42f0-90a7-7e70b5c435ca",
+ "cell_type": "code",
+ "execution_count": 67,
+ "id": "6ffd00f1-e13c-4c60-885e-d7fb6ebe895b",
"metadata": {
- "id": "Zi_xhqXTCi3h"
+ "id": "6ffd00f1-e13c-4c60-885e-d7fb6ebe895b"
},
+ "outputs": [],
"source": [
- "\n",
- "Question 15: The last element of this tutorial is to play around with routing. Please explain in your own words what the .shortest_path() algorithm does. Include the term 'Dijkstra algorithm' in your answer.\n",
- "
"
+ "buildings = buildings[buildings['building'].isin(['yes', 'house', 'industrial', 'apartment'])]"
]
},
{
- "cell_type": "code",
- "execution_count": null,
- "id": "a5e7faf0-43f4-4947-a5b7-0d5f2c1632ea",
- "metadata": {
- "executionInfo": {
- "elapsed": 837,
- "status": "ok",
- "timestamp": 1675088236066,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "Dr7Ftbbh47DW"
+ "cell_type": "markdown",
+ "id": "22598d8d-a73c-41b7-befd-f8abafc86162",
+ "metadata": {
+ "id": "22598d8d-a73c-41b7-befd-f8abafc86162"
},
- "outputs": [],
"source": [
- "origin_node = ox.nearest_nodes(G,area['bbox_west'].values[0], area['bbox_north'].values[0])\n",
- "destination_node = ox.nearest_nodes(G,area['bbox_east'].values[0], area['bbox_south'].values[0])\n",
- "route = nx.shortest_path(G, origin_node, destination_node)"
+ "Now let's visualize the buildings again. We need to create a similar color dictionary as we did for the land-use categories. Now its up to you to make it!"
]
},
{
- "cell_type": "markdown",
- "id": "b3724e1e-0cfa-4e7c-8d94-1d76e10c06d1",
+ "cell_type": "code",
+ "execution_count": 68,
+ "id": "739a2389-8302-4e4f-8624-2beb0225ecfd",
"metadata": {
- "id": "4UuCS8G247DW"
+ "id": "739a2389-8302-4e4f-8624-2beb0225ecfd"
},
+ "outputs": [],
"source": [
- "We can plot the route with folium. Like above, you can pass keyword args along to folium PolyLine to style the lines."
+ "color_dict = { 'yes' : \"#f1134b\",\n",
+ " 'house':'#f13013',\n",
+ " 'industrial':'#0f045c',\n",
+ " 'apartment':'#fcfcb9' }"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "ac7785a7-c4c3-491b-8c65-44353cdddc43",
+ "id": "54672469-3961-4b67-860e-17f9320d1a62",
"metadata": {
"colab": {
- "base_uri": "https://localhost:8080/",
- "height": 866
- },
- "executionInfo": {
- "elapsed": 240,
- "status": "ok",
- "timestamp": 1675088397348,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
+ "base_uri": "https://localhost:8080/"
},
- "id": "7QrZ6Gpi47DX",
- "outputId": "33fecead-a6d7-4090-99da-e09d0c34081a"
+ "id": "54672469-3961-4b67-860e-17f9320d1a62",
+ "outputId": "c172f78b-7643-4c4a-f0c6-655dcc2da355"
},
"outputs": [],
"source": [
- "m2 = ox.plot_route_folium(G, route, weight=10)\n",
- "m2"
+ "map_dict = dict(zip(color_dict.keys(),[x for x in range(len(color_dict))]))\n",
+ "buildings['col_landuse'] =buildings.building.apply(lambda x: color_dict[x])"
]
},
{
"cell_type": "markdown",
- "id": "9df6de86-02e3-4a06-b99f-ec39c71385e8",
+ "id": "0bbaef6d-6b28-45f4-ace9-6a5c9dec0f2f",
"metadata": {
- "id": "eedHqPli47DX"
+ "id": "0bbaef6d-6b28-45f4-ace9-6a5c9dec0f2f"
},
"source": [
- "Plot the route with folium on top of the previously created map\n"
+ "And plot the figure in the same manner!"
]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "d0813cf0-0fb3-4ba7-9012-dc5c3f362743",
+ "id": "7dad1b9f-855a-468e-9090-25d7ace2990d",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
- "height": 866
+ "height": 847
},
- "executionInfo": {
- "elapsed": 2518,
- "status": "ok",
- "timestamp": 1675088413924,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "aHzxZAbZ47DX",
- "outputId": "01ee7d97-4792-41fb-925c-c44fb9b15a3f"
+ "id": "7dad1b9f-855a-468e-9090-25d7ace2990d",
+ "outputId": "68968e35-8949-4abd-c593-2ef1254d81d8"
},
"outputs": [],
"source": [
- "m3 = ox.plot_route_folium(G, route, route_map=m1, popup_attribute=\"name\", weight=7)\n",
- "m3"
+ "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n",
+ "\n",
+ "# add color scheme\n",
+ "color_scheme_map = list(color_dict.values())\n",
+ "cmap = LinearSegmentedColormap.from_list(name='landuse',\n",
+ " colors=color_scheme_map)\n",
+ "\n",
+ "# and plot the land-use map.\n",
+ "buildings.plot(color=buildings['col_landuse'],ax=ax,linewidth=0)\n",
+ "\n",
+ "# remove the ax labels\n",
+ "ax.set_xticks([])\n",
+ "ax.set_yticks([])\n",
+ "ax.set_axis_off()\n",
+ "\n",
+ "# add a legend:\n",
+ "legend_elements = []\n",
+ "for iter_,item in enumerate(color_dict):\n",
+ " legend_elements.append(Patch(facecolor=color_scheme_map[iter_],label=item))\n",
+ "\n",
+ "ax.legend(handles=legend_elements,edgecolor='black',facecolor='#fefdfd',prop={'size':12},loc=(1.02,0.2))\n",
+ "\n",
+ "# add a title\n",
+ "ax.set_title(place_name,fontweight='bold')"
]
},
{
- "attachments": {},
"cell_type": "markdown",
- "id": "d0eda49e-6b12-42b9-b88d-ba0938f2f507",
- "metadata": {},
+ "id": "7fc1437d-8deb-4569-a3ee-9270676f0e41",
+ "metadata": {
+ "id": "7fc1437d-8deb-4569-a3ee-9270676f0e41"
+ },
"source": [
"\n",
- "Question 16: Please add one more routes on a map and upload the resulting figure here.\n",
+ "Question 14: Please upload a figure of your building stock map of your region of interest. Make sure that the interpretation is clear. If necessary, merge multiple categories into one (i.e., when some categories only contain 1 or 2 buildings).\n",
"
"
]
},
{
"cell_type": "markdown",
"id": "508022f5-46d7-4c69-91d2-46711d6a514c",
- "metadata": {},
+ "metadata": {
+ "id": "508022f5-46d7-4c69-91d2-46711d6a514c"
+ },
"source": [
- "## 7. Perform a damage assessment of the road network using OpenStreetMap\n",
+ "### 7. Perform a damage assessment of the road network using OpenStreetMap\n",
"" ] }, @@ -2569,7 +2768,7 @@ "cell_type": "markdown", "id": "b8d7623d-40ee-4262-9781-0c2ed3ea1889", "metadata": { - "id": "bKFmTKpj-g4K" + "id": "b8d7623d-40ee-4262-9781-0c2ed3ea1889" }, "source": [ "Generally, wind damage does not cause much damage to roads. There will be clean-up cost of the trees that will fall on the roads, but structural damage is rare. As such, we will only do a flood damage assessment for the road network of our region.\n", @@ -2579,10 +2778,10 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 71, "id": "abacf8e1-b27c-4156-8521-87f38004ac2e", "metadata": { - "id": "CUqFG7AD-g4K" + "id": "abacf8e1-b27c-4156-8521-87f38004ac2e" }, "outputs": [], "source": [ @@ -2594,7 +2793,7 @@ "cell_type": "markdown", "id": "87de9a96-e3fc-4ab5-b090-be6412a0d0a1", "metadata": { - "id": "modIJTEz-g4K" + "id": "87de9a96-e3fc-4ab5-b090-be6412a0d0a1" }, "source": [ "Now we convert the road network to a `geodataframe`." @@ -2602,9 +2801,11 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 72, "id": "90acac15-3b1e-4c0a-b0a4-6033d243f9ea", - "metadata": {}, + "metadata": { + "id": "90acac15-3b1e-4c0a-b0a4-6033d243f9ea" + }, "outputs": [], "source": [ "roads = gpd.GeoDataFrame(nx.to_pandas_edgelist(G))\n", @@ -2614,22 +2815,26 @@ { "cell_type": "markdown", "id": "56f95cc0-2611-472f-92fa-c256f368c33c", - "metadata": {}, + "metadata": { + "id": "56f95cc0-2611-472f-92fa-c256f368c33c" + }, "source": [ - "In case the above does not work, you can continue the assignment by using the code below (make sure you remove the hashtags to run it). " + "In case the above does not work, you can continue the assignment by using the code below (make sure you remove the hashtags to run it)." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 73, "id": "55752d12-81de-4eb5-b849-90473f88478a", - "metadata": {}, + "metadata": { + "id": "55752d12-81de-4eb5-b849-90473f88478a" + }, "outputs": [], "source": [ "#from urllib import request\n", "# remote_url = 'https://github.com/VU-IVM/UNIGIS_ProgrammingGIS/tree/main/TAA2'\n", "# file = 'kampen_roads.gpkg'\n", - "# \n", + "#\n", "# #request.urlretrieve(remote_url, file)\n", "# roads = gpd.GeoDataFrame.from_file('kampen_roads.gpkg')" ] @@ -2638,7 +2843,7 @@ "cell_type": "markdown", "id": "73f4a201-87b1-4cd8-a206-6b965c9b4107", "metadata": { - "id": "VIaMGLxA-g4K" + "id": "73f4a201-87b1-4cd8-a206-6b965c9b4107" }, "source": [ "And lets have a look at the data:" @@ -2651,10 +2856,10 @@ "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 495 + "height": 807 }, - "id": "dx_299FS-g4L", - "outputId": "ffdab479-6a25-4794-da3a-cbacf2accaa4" + "id": "91a864f8-6af7-454f-869c-197e5e47151b", + "outputId": "01bfca83-8de6-4d76-90df-4702886e66fb" }, "outputs": [], "source": [ @@ -2674,18 +2879,18 @@ "cell_type": "markdown", "id": "4789ae59-e6c9-4a1c-a1ad-44e94c9172ca", "metadata": { - "id": "PSGo7dC3-g4L" + "id": "4789ae59-e6c9-4a1c-a1ad-44e94c9172ca" }, "source": [ - "It is actually quite inconvenient to have all these lists in the data for when we want to do the damage assessment. Let's clean this up a bit. To do so, we first make sure that all the lists are represented as actual lists, and not lists wrapped within a string." + "Dependening on the region you have selected, you may have lists in your data. It is actually quite inconvenient to have all these lists in the data for when we want to do the damage assessment. Let's clean this up a bit. To do so, we first make sure that all the lists are represented as actual lists, and not lists wrapped within a string." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 75, "id": "51c4746a-6efd-4b59-8b6b-5fe9a8c42372", "metadata": { - "id": "S_LZSRI6-g4L" + "id": "51c4746a-6efd-4b59-8b6b-5fe9a8c42372" }, "outputs": [], "source": [ @@ -2696,7 +2901,7 @@ "cell_type": "markdown", "id": "eb311b10-0a1c-4e0a-ad1f-cef37aa40ad7", "metadata": { - "id": "cwQKiRDd-g4L" + "id": "eb311b10-0a1c-4e0a-ad1f-cef37aa40ad7" }, "source": [ "Now we just need to grab the first element of each of the lists." @@ -2704,10 +2909,10 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 76, "id": "9e449847-c279-479e-bf3b-382b8c50e018", "metadata": { - "id": "qe86tcET-g4L" + "id": "9e449847-c279-479e-bf3b-382b8c50e018" }, "outputs": [], "source": [ @@ -2719,7 +2924,7 @@ "cell_type": "markdown", "id": "5b3f95ad-9a1f-43e1-b098-45255c9d5dd2", "metadata": { - "id": "TkyDDDIP-g4L" + "id": "5b3f95ad-9a1f-43e1-b098-45255c9d5dd2" }, "source": [ "And let's have a look whether this worked:" @@ -2732,10 +2937,10 @@ "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 495 + "height": 807 }, - "id": "f5qPSBmq-g4L", - "outputId": "6ac152f2-14f1-4e6c-c53a-9d68db347ce6" + "id": "21dc3a62-c266-4d0e-bd03-ad1d1b4b22e8", + "outputId": "0361fc7e-ad9c-40f4-a767-aac6ac1d84cc" }, "outputs": [], "source": [ @@ -2749,23 +2954,11 @@ "ax.set_axis_off()" ] }, - { - "cell_type": "markdown", - "id": "a3049d42-da1f-4dc2-a56f-1ce6b3080fd0", - "metadata": { - "id": "u0crazq8iQjQ" - }, - "source": [ - "
\n",
- "Question 17: Upload a figure of the cleaned road network (e.g. in which you do not see any of the listed road types anymore)\n",
- "
"
- ]
- },
{
"cell_type": "markdown",
"id": "d784f6dc-7091-4e60-b7df-58682efb66ef",
"metadata": {
- "id": "J63sExRp-g4L"
+ "id": "d784f6dc-7091-4e60-b7df-58682efb66ef"
},
"source": [
"Nice! now let's start with the damage calculation. As you already have may have noticed, our data is now not stored in raster format, but in vector format. One way to deal with this issue is to convert our vector data to raster data, but we will lose a lot of information and detail. As such, we will perform the damage assessment on the road elements, using the xarray flood map.\n",
@@ -2775,10 +2968,10 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 78,
"id": "af68dc43-fd65-416c-bc7a-a1b1231187d3",
"metadata": {
- "id": "uHmaZFXV-g4L"
+ "id": "af68dc43-fd65-416c-bc7a-a1b1231187d3"
},
"outputs": [],
"source": [
@@ -2800,7 +2993,7 @@
"cell_type": "markdown",
"id": "9894f1ba-e637-4ede-b46c-5aa4747eaea0",
"metadata": {
- "id": "sKb-ig4Q-g4M"
+ "id": "9894f1ba-e637-4ede-b46c-5aa4747eaea0"
},
"source": [
"And let's plot the results:"
@@ -2813,10 +3006,10 @@
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
- "height": 306
+ "height": 481
},
- "id": "LL3YU6r1-g4M",
- "outputId": "e0a9e61f-e376-436c-a11a-fa58651bf15a"
+ "id": "5f6ab83a-62af-4142-bb26-157e077d868b",
+ "outputId": "adbb0e5e-7487-4890-f767-3bbd11ded3ae"
},
"outputs": [],
"source": [
@@ -2827,80 +3020,145 @@
"cell_type": "markdown",
"id": "1cdbc0d8-beba-4b37-be49-f5226953f19b",
"metadata": {
- "id": "XBsxnhjN-g4M"
+ "id": "1cdbc0d8-beba-4b37-be49-f5226953f19b"
},
"source": [
- "We will need a bunch of functions to make sure we can do our calculations. They are specified below. "
+ "We will need a bunch of functions to make sure we can do our calculations. They are specified below."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 80,
"id": "d470c747-42aa-4f20-bbf3-90773c3d4cc8",
"metadata": {
- "id": "T-XfgGLB-g4M"
+ "id": "d470c747-42aa-4f20-bbf3-90773c3d4cc8"
},
"outputs": [],
"source": [
- "def reproject(df_ds,current_crs=\"epsg:4326\",approximate_crs = \"epsg:3035\"):\n",
+ "def reproject(df_ds, current_crs=\"epsg:4326\", approximate_crs=\"epsg:3035\"):\n",
+ " \"\"\"\n",
+ " Reproject geometries in a DataFrame from one coordinate reference system (CRS) to another.\n",
+ "\n",
+ " Parameters:\n",
+ " df_ds : pandas.DataFrame\n",
+ " A DataFrame containing a 'geometry' column with geometries to reproject.\n",
+ " current_crs : str or pyproj.CRS, optional\n",
+ " The current coordinate reference system of the geometries. Default is \"epsg:4326\".\n",
+ " approximate_crs : str or pyproj.CRS, optional\n",
+ " The target coordinate reference system to reproject the geometries into. Default is \"epsg:3035\".\n",
+ "\n",
+ " Returns:\n",
+ " shapely.GeometryArray\n",
+ " A Shapely GeometryArray containing the reprojected geometries.\n",
+ " \"\"\"\n",
" geometries = df_ds['geometry']\n",
" coords = shapely.get_coordinates(geometries)\n",
- " transformer=pyproj.Transformer.from_crs(current_crs, approximate_crs,always_xy=True)\n",
+ " transformer = pyproj.Transformer.from_crs(current_crs, approximate_crs, always_xy=True)\n",
" new_coords = transformer.transform(coords[:, 0], coords[:, 1])\n",
- " \n",
- " return shapely.set_coordinates(geometries.copy(), np.array(new_coords).T) \n",
"\n",
- "def buffer_assets(assets,buffer_size=100):\n",
- " assets['buffered'] = shapely.buffer(assets.geometry.values,buffer_size)\n",
+ " return shapely.set_coordinates(geometries.copy(), np.array(new_coords).T)\n",
+ "\n",
+ "\n",
+ "def buffer_assets(assets, buffer_size=100):\n",
+ " \"\"\"\n",
+ " Create a buffer around each geometry in the assets DataFrame.\n",
+ "\n",
+ " Parameters:\n",
+ " assets : pandas.DataFrame\n",
+ " A DataFrame containing a 'geometry' column with geometries to buffer.\n",
+ " buffer_size : float, optional\n",
+ " The distance to buffer around each geometry. Default is 100 units.\n",
+ "\n",
+ " Returns:\n",
+ " pandas.DataFrame\n",
+ " The input DataFrame with an additional 'buffered' column containing the buffered geometries.\n",
+ " \"\"\"\n",
+ " assets['buffered'] = shapely.buffer(assets.geometry.values, buffer_size)\n",
" return assets\n",
"\n",
- "def overlay_hazard_assets(df_ds,assets):\n",
"\n",
- " #overlay \n",
+ "def overlay_hazard_assets(df_ds, assets):\n",
+ " \"\"\"\n",
+ " Find the indices of hazards that overlay or intersect with assets.\n",
+ "\n",
+ " Parameters:\n",
+ " df_ds : pandas.DataFrame\n",
+ " A DataFrame containing hazard geometries in a 'geometry' column.\n",
+ " assets : pandas.DataFrame\n",
+ " A DataFrame containing asset geometries in a 'geometry' column.\n",
+ "\n",
+ " Returns:\n",
+ " numpy.ndarray\n",
+ " An array of indices of hazards that intersect with the assets.\n",
+ " \"\"\"\n",
+ " # Build a spatial index for the hazard geometries\n",
" hazard_tree = shapely.STRtree(df_ds.geometry.values)\n",
- " if (shapely.get_type_id(assets.iloc[0].geometry) == 3) | (shapely.get_type_id(assets.iloc[0].geometry) == 6):\n",
- " return hazard_tree.query(assets.geometry,predicate='intersects') \n",
+ " # Determine geometry type of the first asset\n",
+ " asset_geom_type = shapely.get_type_id(assets.iloc[0].geometry)\n",
+ " # If the asset geometry is a polygon or multipolygon\n",
+ " if (asset_geom_type == 3) or (asset_geom_type == 6):\n",
+ " return hazard_tree.query(assets.geometry, predicate='intersects')\n",
" else:\n",
- " return hazard_tree.query(assets.buffered,predicate='intersects')\n",
- " \n",
- "def get_damage_per_asset(asset,df_ds,assets):\n",
- " # find the exact hazard overlays:\n",
- " get_hazard_points = df_ds.iloc[asset[1]['hazard_point'].values].reset_index()\n",
- " get_hazard_points = get_hazard_points.loc[shapely.intersects(get_hazard_points.geometry.values,assets.iloc[asset[0]].geometry)]\n",
+ " # If the asset geometry is not polygon/multipolygon, use buffered geometries\n",
+ " return hazard_tree.query(assets.buffered, predicate='intersects')\n",
"\n",
+ "\n",
+ "def get_damage_per_asset(asset, df_ds, assets):\n",
+ " \"\"\"\n",
+ " Calculate the total damage for a single asset based on overlapping hazards.\n",
+ "\n",
+ " Parameters:\n",
+ " asset : tuple\n",
+ " A tuple containing the asset index and a DataFrame with hazard points intersecting the asset.\n",
+ " df_ds : pandas.DataFrame\n",
+ " A DataFrame containing hazard data with a 'geometry' column.\n",
+ " assets : pandas.DataFrame\n",
+ " A DataFrame containing asset data with a 'geometry' column.\n",
+ "\n",
+ " Returns:\n",
+ " tuple\n",
+ " A tuple containing the asset index and the calculated damage.\n",
+ " \"\"\"\n",
+ " # Find the exact hazard overlays\n",
+ " get_hazard_points = df_ds.iloc[asset[1]['hazard_point'].values].reset_index()\n",
+ " # Select hazard points that intersect with the asset geometry\n",
" asset_geom = assets.iloc[asset[0]].geometry\n",
+ " get_hazard_points = get_hazard_points.loc[shapely.intersects(get_hazard_points.geometry.values, asset_geom)]\n",
+ "\n",
+ " # Parameters for damage calculation\n",
+ " maxdam_asset = 100 # Maximum damage per asset\n",
+ " hazard_intensity = np.arange(0, 10, 0.1) # Hazard intensity levels\n",
+ " fragility_values = np.arange(0, 1, 0.01) # Fragility values corresponding to hazard intensity\n",
"\n",
- " maxdam_asset = 100\n",
- " hazard_intensity = np.arange(0,10,0.1) \n",
- " fragility_values = np.arange(0,1,0.01) \n",
- " \n",
" if len(get_hazard_points) == 0:\n",
- " return asset[0],0\n",
+ " return asset[0], 0\n",
" else:\n",
- " get_hazard_points['overlay_meters'] = shapely.length(shapely.intersection(get_hazard_points.geometry.values,asset_geom))\n",
- " return asset[0],np.sum((np.interp(get_hazard_points.band_data.values,hazard_intensity,fragility_values))*get_hazard_points.overlay_meters*maxdam_asset)"
+ " # Calculate the length of the intersection between hazard and asset geometries\n",
+ " get_hazard_points['overlay_meters'] = shapely.length(\n",
+ " shapely.intersection(get_hazard_points.geometry.values, asset_geom))\n",
+ " # Interpolate fragility values based on hazard intensity\n",
+ " damage = np.sum(\n",
+ " (np.interp(get_hazard_points.band_data.values, hazard_intensity, fragility_values))\n",
+ " * get_hazard_points.overlay_meters * maxdam_asset)\n",
+ " return asset[0], damage"
]
},
{
"cell_type": "markdown",
"id": "622a243c-dd99-44c1-975c-7cb4eab99359",
"metadata": {
- "id": "og2Bkcv--g4M"
+ "id": "622a243c-dd99-44c1-975c-7cb4eab99359"
},
"source": [
- "Now we need to make sure that the road data is the same coordinate system. "
+ "Now we need to make sure that the road data is the same coordinate system."
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 81,
"id": "484c68a5-90ae-4394-9e4f-4ed5ba0c665d",
"metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "NvtDuspN-g4M",
- "outputId": "8d920938-ab07-42f9-bb88-34483e751c3f"
+ "id": "484c68a5-90ae-4394-9e4f-4ed5ba0c665d"
},
"outputs": [],
"source": [
@@ -2911,7 +3169,7 @@
"cell_type": "markdown",
"id": "efebb735-7955-4a75-bedf-1b213067fe20",
"metadata": {
- "id": "4JT25WTv-g4M"
+ "id": "efebb735-7955-4a75-bedf-1b213067fe20"
},
"source": [
"And we can now overlay the roads with the flood data"
@@ -2919,10 +3177,10 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 82,
"id": "60de298f-9e06-40c3-b3a2-eecf2aa7205d",
"metadata": {
- "id": "8rtBYbX_-g4M"
+ "id": "60de298f-9e06-40c3-b3a2-eecf2aa7205d"
},
"outputs": [],
"source": [
@@ -2933,7 +3191,7 @@
"cell_type": "markdown",
"id": "53aeaec5-4ea3-48a5-93b5-109a1578c77e",
"metadata": {
- "id": "s82DyD_y-g4M"
+ "id": "53aeaec5-4ea3-48a5-93b5-109a1578c77e"
},
"source": [
"And estimate the damages"
@@ -2947,8 +3205,8 @@
"colab": {
"base_uri": "https://localhost:8080/"
},
- "id": "LGqPFklh-g4N",
- "outputId": "207937c9-dcc5-41a5-ebc4-76aa03003022"
+ "id": "daa56503-90e8-4b7f-a5dc-82a4b235e95b",
+ "outputId": "059ff95d-6e35-4a68-ebcf-31ee13910468"
},
"outputs": [],
"source": [
@@ -2956,7 +3214,7 @@
"for asset in tqdm(overlay_roads.groupby('asset'),total=len(overlay_roads.asset.unique()),\n",
" desc='polyline damage calculation for'):\n",
" collect_output.append(get_damage_per_asset(asset,flood_map_vector,roads))\n",
- " \n",
+ "\n",
"damaged_roads = roads.merge(pd.DataFrame(collect_output,columns=['index','damage']),\n",
" left_index=True,right_on='index')[['highway','geometry','damage']]"
]
@@ -2965,11 +3223,11 @@
"cell_type": "markdown",
"id": "580663ca-b83f-4724-a699-b6a4790678ad",
"metadata": {
- "id": "RFGZxWl7i7pQ"
+ "id": "580663ca-b83f-4724-a699-b6a4790678ad"
},
"source": [
"\n",
- "Question 18: Describe the various steps we have taken to perform the damage assessment on the road network. How is this approach different compared to the raster-based approach? Highlight the differences you find most important. Include any line of code you may want to include to make your story clear.\n",
+ "Question 15: Describe the various steps we have taken to perform the damage assessment on the road network. How is this approach different compared to the raster-based approach? Highlight the differences you find most important. Include any line of code you may want to include to make your story clear.\n",
"
"
]
},
@@ -2977,7 +3235,7 @@
"cell_type": "markdown",
"id": "2e42c7f3-c868-400c-858e-7ac0c848fd4c",
"metadata": {
- "id": "B5jpsbyC-g4N"
+ "id": "2e42c7f3-c868-400c-858e-7ac0c848fd4c"
},
"source": [
"And let's plot the results"
@@ -2990,40 +3248,33 @@
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
- "height": 514
+ "height": 864
},
- "id": "n25j-3wG-g4N",
- "outputId": "b926a8e7-7e51-4434-f3b8-d61e6278bc24"
+ "id": "ef910b4e-201e-444b-92b6-826be82a5164",
+ "outputId": "eead7d6c-01ea-4851-d8d0-b756fea0798e"
},
"outputs": [],
"source": [
"fig, ax = plt.subplots(1, 1,figsize=(12,10))\n",
"\n",
- "damaged_roads.plot(column='damage',cmap='Reds',ax=ax);"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "49d5033f-ac25-4ae1-9991-ca0899c1b8d4",
- "metadata": {
- "id": "bTfmvwW2jchB"
- },
- "source": [
- "\n",
- "Question 19: Describe the most severely damaged parts of the road network. Use Google Maps to identify these roads. Are you surprised by the results?\n",
- "
"
+ "damaged_roads.plot(column='damage',cmap='Reds',ax=ax)"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 84,
"id": "fe92fb12-9049-4e15-8992-b3d0cbd459b5",
- "metadata": {},
+ "metadata": {
+ "id": "fe92fb12-9049-4e15-8992-b3d0cbd459b5"
+ },
"outputs": [],
"source": []
}
],
"metadata": {
+ "colab": {
+ "provenance": []
+ },
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
@@ -3039,7 +3290,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.12.3"
+ "version": "3.11.9"
}
},
"nbformat": 4,
diff --git a/TAA2/tutorial1.ipynb b/TAA2/tutorial1.ipynb
deleted file mode 100644
index 071cda0..0000000
--- a/TAA2/tutorial1.ipynb
+++ /dev/null
@@ -1,1913 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "jTmaE4gi47C9"
- },
- "source": [
- "# Tutorial 1: Working with OpenStreetMap\n",
- "\n",
- "Within this tutorial, we will explore the power of OpenStreetMap. We will learn how to extract information from OpenStreetMap, how you can explore and visualize this, and how to use it for some basic analysis.\n",
- "\n",
- "### Important before we start\n",
- "---\n",
- "Make sure that you save this file before you continue, else you will lose everything. To do so, go to **Bestand/File** and click on **Een kopie opslaan in Drive/Save a Copy on Drive**!\n",
- "\n",
- "Now, rename the file into Week5_Tutorial1.ipynb. You can do so by clicking on the name in the top of this screen."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "KqMg54GZ47DB",
- "tags": []
- },
- "source": [
- "## Learning Objectives\n",
- "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "JC0K9vub47DC" - }, - "source": [ - "- To understand the use of **OSMnx** to extract geospatial data from OpenStreetmap.\n", - "- To know how to rasterize vector data through using **Geocube**.\n", - "- To know how to visualise vector and raster data.\n", - "- To understand the basic functioning of **Matplotlib** to create a map.\n", - "- To know how one can generate routes between two points using **NetworkX**.\n", - "- To visualize networks on an interactive map." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "W6efV5lD47DC" - }, - "source": [ - "
Tutorial Outline
\n", - "\n", - "
- \n",
- "
- 1. Introducing the packages \n", - "
- 2. Extracting land-use information from OpenStreetMap \n", - "
- 3. Rasterize land-use information \n", - "
- 4. Extracting buildings from OpenStreetMap \n", - "
- 5. Analyze and visualize building stock \n", - "
- 6. Extracting roads from OpenStreetMap \n", - "
- 7. Plot Routes Using OpenStreetMap and Folium
" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "mr4J-7kq47DD" - }, - "source": [ - "Within this tutorial, we are going to make use of the following packages: \n", - "\n", - "[**GeoPandas**](https://geopandas.org/) is a Python package that extends the datatypes used by pandas to allow spatial operations on geometric types.\n", - "\n", - "[**OSMnx**](https://osmnx.readthedocs.io/) is a Python package that lets you download geospatial data from OpenStreetMap and model, project, visualize, and analyze real-world street networks and any other geospatial geometries. You can download and model walkable, drivable, or bikeable urban networks with a single line of Python code then easily analyze and visualize them. You can just as easily download and work with other infrastructure types, amenities/points of interest, building footprints, elevation data, street bearings/orientations, and speed/travel time.\n", - "\n", - "[**NetworkX**](https://networkx.org/) is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.\n", - "\n", - "[**Matplotlib**](https://matplotlib.org/) is a comprehensive Python package for creating static, animated, and interactive visualizations in Python. Matplotlib makes easy things easy and hard things possible.\n", - "\n", - "[**Geocube**](https://corteva.github.io/geocube) is a Python package to convert geopandas vector data into rasterized data.\n", - "\n", - "[**xarray**](https://docs.xarray.dev/) is a Python package that allows for easy and efficient use of multi-dimensional arrays.\n", - "\n", - "*We will first need to install these packages in the cell below. Uncomment them to make sure we can pip install them*" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "executionInfo": { - "elapsed": 31325, - "status": "ok", - "timestamp": 1675086280074, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "VD7sW5hx47DE", - "outputId": "36a15ed9-7936-4f89-9284-6562c265d75c" - }, - "outputs": [], - "source": [ - "!pip install osmnx\n", - "!pip install geocube\n", - "!pip install contextily" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "eVsADIc847DG" - }, - "source": [ - "Now we will import these packages in the cell below:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 3237, - "status": "ok", - "timestamp": 1675086283294, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "EJeE3bc047DG" - }, - "outputs": [], - "source": [ - "import warnings\n", - "warnings.filterwarnings('ignore') \n", - "\n", - "import osmnx as ox\n", - "import numpy as np\n", - "import networkx as nx\n", - "import contextily as cx\n", - "import matplotlib\n", - "import geopandas as gpd\n", - "\n", - "from matplotlib.colors import LinearSegmentedColormap,ListedColormap\n", - "from matplotlib.patches import Patch\n", - "import matplotlib.pyplot as plt\n", - "from geocube.api.core import make_geocube\n", - "from urllib import request\n", - "\n", - "%matplotlib inline" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "YPgVj6al47DH" - }, - "source": [ - "## 2. Extract and visualize land-use information from OpenStreetMap\n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Uhh94J5Z47DI" - }, - "source": [ - "The first step is to define which area you want to focus on. In the cell below, you will now read \"Zoeterwoude, The Netherlands\". Change this to any area or municipality in the Netherlands that (1) you can think of and (2) will work. \n", - "\n", - "In some cases, the function does not recognize the location. You could either try a different phrasing or try a different location. Many parts of the Netherlands should work." - ] - }, - { - "cell_type": "code", - "execution_count": 34, - "metadata": { - "executionInfo": { - "elapsed": 1398, - "status": "ok", - "timestamp": 1675086381517, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "9cijX5J047DI" - }, - "outputs": [], - "source": [ - "place_name = \"Zoeterwoude, The Netherlands\"\n", - "area = ox.geocode_to_gdf(place_name)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "kX3feV7O47DJ" - }, - "source": [ - "Now let us visualize the bounding box of the area. As you will notice, we also estimate the size of the area. If the area size is above 50km2, or when you have many elements within your area (for example the amsterdam city centre), extracting the data from OpenStreetMap may take a little while. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 608 - }, - "executionInfo": { - "elapsed": 5155, - "status": "ok", - "timestamp": 1675086404602, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "JFBsptnt47DJ", - "outputId": "3a628a04-9ade-43dd-f684-efae51795dc5", - "tags": [] - }, - "outputs": [], - "source": [ - "area_to_check = area.to_crs(epsg=3857)\n", - "ax = area_to_check.plot(figsize=(10, 10), color=\"none\", edgecolor=\"k\", linewidth=4)\n", - "ax.set_xticks([])\n", - "ax.set_yticks([])\n", - "ax.set_axis_off()\n", - "cx.add_basemap(ax, zoom=11)\n", - "\n", - "size = int(area_to_check.area/1e6)\n", - "\n", - "ax.set_title(\"{}. Total area: {} km2\".format(place_name,size),fontweight='bold')" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "pT5odIz-6fki" - }, - "source": [ - "
\n",
- "Question 1: To make sure we understand which area you focus on, please submit the figure that outlines your area.\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "6zbwxAev47DL"
- },
- "source": [
- "Now we are satisfied with the selected area, we are going to extract the land-use information from OpenStreetMap. To find the right information from OpenStreetMap, we use **tags**.\n",
- "\n",
- "As you will see in the cell below, we use the tags *\"landuse\"* and *\"natural\"*. We need to use the *\"natural\"* tag to ensure we also obtain water bodies and other natural elements. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 63,
- "metadata": {
- "executionInfo": {
- "elapsed": 104940,
- "status": "ok",
- "timestamp": 1675086538272,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "pWxGFjgN47DL"
- },
- "outputs": [],
- "source": [
- "tags = {'landuse': True, 'natural': True} \n",
- "landuse = ox.features_from_place(place_name, tags)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In case the above does not work, you can continue the assignment by using the code below (make sure you remove the hashtags to run it). If you decide to use the data as specified below, also change the map at the start to 'Kampen'."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 67,
- "metadata": {},
- "outputs": [],
- "source": [
- "# remote_url = 'https://github.com/ElcoK/BigData_AED/raw/main/week5/kampen_landuse.gpkg'\n",
- "# file = 'kampen_landuse.gpkg'\n",
- "\n",
- "# request.urlretrieve(remote_url, file)\n",
- "#landuse = gpd.GeoDataFrame.from_file('kampen_landuse.gpkg')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "brNqZmi547DM"
- },
- "source": [
- "To ensure we really only get the area that we want, we use geopandas's `clip` function to only keep the area we want. This function does exactly the same as the `clip` function in QGIS."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Anrn7wiL47DM"
- },
- "source": [
- "When we want to visualize or analyse the data, we want all information in a single column. However, at the moment, all information that was tagged as *\"natural\"*, has no information stored in the *\"landuse\"* tags. It is, however, very convenient if we can just use a single column for further exploration of the data. \n",
- "\n",
- "To overcome this issue, we need to add the missing information to the landuse column, as done below. Let's first have a look which categories we have in the **natural** column. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "landuse.natural.unique()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "And now we can add them to the **landuse** column. We made a start, but its up to you to fill in the rest."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 56,
- "metadata": {
- "executionInfo": {
- "elapsed": 429,
- "status": "ok",
- "timestamp": 1675086633493,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "nlyHmEzg47DM"
- },
- "outputs": [],
- "source": [
- "landuse.loc[landuse.natural=='water','landuse'] = 'water'\n",
- "landuse.loc[landuse.natural=='wetland','landuse'] = 'wetlands'\n",
- "\n",
- "\n",
- "landuse = landuse.dropna(subset=['landuse'])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "FOhhcbFs8Nsz"
- },
- "source": [
- "\n",
- "Question 2: Please provide in the answer box in Canvas the code that you used to make sure that all land uses are now registered within the landuse column.\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Ii4rjR3q47DN"
- },
- "source": [
- "Our next step is to prepare the visualisation of a map. What better way to explore land-use information than plotting it on a map? \n",
- "\n",
- "As you will see below, we can create a dictionary with color codes that will color each land-use class based on the color code provided in this dictionary."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 57,
- "metadata": {
- "executionInfo": {
- "elapsed": 864,
- "status": "ok",
- "timestamp": 1675086777083,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "urAI5MAG47DN"
- },
- "outputs": [],
- "source": [
- "color_dict = { \"grass\":'#c3eead', \"railway\": \"#000000\",\n",
- " \"forest\":'#1c7426', \"orchard\":'#fe6729',\n",
- " \"residential\":'#f13013', \"industrial\":'#0f045c',\n",
- " \"retail\":'#b71456', \"education\":'#d61181', \n",
- " \"commercial\":'#981cb8', \"farmland\":'#fcfcb9',\n",
- " \"cemetery\":'#c39797', \"construction\":'#c0c0c0',\n",
- " \"meadow\":'#c3eead', \"farmyard\":'#fcfcb9',\n",
- " \"plant_nursery\":'#eaffe2', \"scrub\":'#98574d',\n",
- " \"allotments\":'#fbffe2', \"reservoir\":'#8af4f2',\n",
- " \"static_caravan\":'#ff3a55', \"wetlands\": \"#c9f5e5\",\n",
- " \"water\": \"#c9e5f5\", \"beach\": \"#ffeead\",\n",
- " \"landfill\" : \"#B08C4D\", \"recreation_ground\" : \"#c3eead\",\n",
- " \"brownfield\" : \"#B08C4D\", \"village_green\" : \"#f13013\" ,\n",
- " \"military\": \"#52514E\", \"garden\" : '#c3eead'\n",
- " } "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Daf-4BMa47DN"
- },
- "source": [
- "Unfortunately, OpenSteetMap very often contains elements that have a unique tag. As such, it may be the case that some of our land-use categories are not in the dictionary yet. \n",
- "\n",
- "Let's first create an overview of the unique land-use categories within our data through using the `.unique()` function within our dataframe:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "executionInfo": {
- "elapsed": 221,
- "status": "ok",
- "timestamp": 1675086893656,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "h8t5a6_Z47DN",
- "outputId": "9e17ba91-d0a4-4dcc-df05-8483f352a228"
- },
- "outputs": [],
- "source": [
- "landuse.landuse.unique()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "h0p7fYBd47DN"
- },
- "source": [
- "Ofcourse we can visually compare the array above with our color_dict, but it is much quicker to use `Sets` to check if there is anything missing:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "executionInfo": {
- "elapsed": 248,
- "status": "ok",
- "timestamp": 1675086896661,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "UZaHsgNq47DO",
- "outputId": "8a7f6f9e-8bde-4c5c-8012-6ed7f8b071ed"
- },
- "outputs": [],
- "source": [
- "set(landuse.landuse.unique())-set(color_dict)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "EtMHGvQk47DO"
- },
- "source": [
- "In case anything is missing, add them to the color_dict dictionairy and re-run that cell. \n",
- "\n",
- "\n",
- "Question 3: Show us in Canvas (i) which land-use categories you had to add, and (ii) how your final color dictionary looks like.\n",
- "
\n",
- "\n",
- "```{tip}\n",
- "You can easily find hexcodes online to find the right colour for each land-use category. Just google hexcodes!\n",
- "```\n",
- "\n",
- "\n",
- "Our next step is to make sure that we can connect our color codes to our dataframe with land-use categories."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "executionInfo": {
- "elapsed": 208,
- "status": "ok",
- "timestamp": 1675086997214,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "Fkkqz3Px47DO"
- },
- "outputs": [],
- "source": [
- "color_dict = {key: color_dict[key]\n",
- " for key in color_dict if key not in list(set(color_dict)-set(landuse.landuse.unique()))}\n",
- "\n",
- "map_dict = dict(zip(color_dict.keys(),[x for x in range(len(color_dict))]))\n",
- "\n",
- "landuse['col_landuse'] = landuse.landuse.apply(lambda x: color_dict[x])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "NxWuztp347DO"
- },
- "source": [
- "Now we can plot the figure!\n",
- "\n",
- "As you will see in the cell below, we first state that we want to create a figure with a specific figure size. You can change the dimensions to your liking."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 608
- },
- "executionInfo": {
- "elapsed": 1825,
- "status": "ok",
- "timestamp": 1675087046285,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "NH6j-qJ147DO",
- "outputId": "23c77638-0509-4bb3-dc19-904f586b2a70"
- },
- "outputs": [],
- "source": [
- "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n",
- "\n",
- "# add color scheme\n",
- "color_scheme_map = list(color_dict.values())\n",
- "cmap = LinearSegmentedColormap.from_list(name='landuse',\n",
- " colors=color_scheme_map) \n",
- "\n",
- "# and plot the land-use map.\n",
- "landuse.plot(color=landuse['col_landuse'],ax=ax,linewidth=0)\n",
- "\n",
- "# remove the ax labels\n",
- "ax.set_xticks([])\n",
- "ax.set_yticks([])\n",
- "ax.set_axis_off()\n",
- "\n",
- "# add a legend:\n",
- "legend_elements = []\n",
- "for iter_,item in enumerate(color_dict):\n",
- " legend_elements.append(Patch(facecolor=color_scheme_map[iter_],label=item)) \n",
- "\n",
- "ax.legend(handles=legend_elements,edgecolor='black',facecolor='#fefdfd',prop={'size':12},loc=(1.02,0.2)) \n",
- "\n",
- "# add a title\n",
- "ax.set_title(place_name,fontweight='bold')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "EGwHPQEL9_hD"
- },
- "source": [
- "\n",
- "Question 4: Please upload a figure of your land-use map, using OpenStreetMap. \n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "EzQP70Om47DO"
- },
- "source": [
- "## 3. Rasterize land-use information\n",
- "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "4w-Py4fw47DP" - }, - "source": [ - "As you have noticed already during the lecture, and as we will again see next week when using the Google Earth Engine, most land-use data is in raster format. \n", - "\n", - "In OpenStreetMap everything is stored in vector format. As such, the land-use information we extracted from OpenStreetMap is also in vector format. While it is not always necessary to have this information in raster format, it is useful to know how to convert your data into a raster format.\n", - "\n", - "To do so, we can make use of the **GeoCube** package, which is a newly developed Python package that can very easily convert vector data into a raster format." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "PXETqUIN47DP" - }, - "source": [ - "The first thing we will need to do is to define all the unique land-use classes and store them in a dictionary:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 290, - "status": "ok", - "timestamp": 1675087253309, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "4st7ApDp47DP" - }, - "outputs": [], - "source": [ - "categorical_enums = {'landuse': landuse.landuse.drop_duplicates().values.tolist()\n", - "}" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "4_qzExh447DP" - }, - "source": [ - "And now we simply use the `make_geocube()` function to convert our vector data into raster data. \n", - "\n", - "In the `make_geocube()` function, we have to specify several arguments:\n", - "\n", - "- Through the `vector_data` argument we have to state which dataframe we want to rasterize.\n", - "- Through the `output_crs` argument we have to state the coordinate reference system (CRS). We use the OpenStreetMap default EPSG:4326.\n", - "- Through the `resolution` argument we have to state the resolution. In our case, we will have to set this in degrees. 0.01 degrees is equivalent to roughly 10km around the equator. \n", - "- Through the `categorical_enums` argument we specify the different land-use categories.\n", - "\n", - "Play around with the different resolutions to find the level of detail. The higher the resolution (i.e., the more zeros behind the comma), the longer it will take to rasterize." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 551, - "status": "ok", - "timestamp": 1675087257707, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "wjGoNrwg47DP", - "tags": [] - }, - "outputs": [], - "source": [ - "landuse_grid = make_geocube(\n", - " vector_data=XXXX,\n", - " output_crs=\"epsg:4326\",\n", - " resolution=(-XXXX, XXXX),\n", - " categorical_enums=categorical_enums\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "wx4FAAsg47DP" - }, - "source": [ - "Let's explore what this function has given us:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 389 - }, - "executionInfo": { - "elapsed": 429, - "status": "ok", - "timestamp": 1675087480922, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "JqdlWZUe47DQ", - "outputId": "8880c112-9200-4613-ef45-01e8b88c0bde" - }, - "outputs": [], - "source": [ - "landuse_grid[\"landuse\"]" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "UB4gbheF47DQ" - }, - "source": [ - "The output above is a typical output of the **xarray** package. \n", - "\n", - "- The `array` shows the numpy array with the actual values. As you can see, the rasterization process has used the value `-1` for NoData. \n", - "- The `Coordinates` table shows the x (longitude) and y (latitude) coordinates of the array. It has the exact same size as the `array` with land-use values.\n", - "- The `Attributes` table specifies the NoData value (the `_FillValue` element, which indeed shows `-1`) and the name of the dataset.\n", - "\n", - "Now let's plot the data to see the result!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 608 - }, - "executionInfo": { - "elapsed": 1852, - "status": "ok", - "timestamp": 1675087494788, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "extRLk1W47DQ", - "outputId": "19fe91a4-5785-494e-a370-9408a563a70a" - }, - "outputs": [], - "source": [ - "fig, ax = plt.subplots(1, 1,figsize=(14,10))\n", - "\n", - "landuse_grid[\"landuse\"].plot(ax=ax,vmin=0,vmax=15,levels=15,cmap='tab20')\n", - "\n", - "# remove the ax labels\n", - "ax.set_xticks([])\n", - "ax.set_yticks([])\n", - "ax.set_axis_off()\n", - "\n", - "#add a title\n", - "\n", - "ax.set_title('')" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "94GLYG5K47DQ" - }, - "source": [ - "As we can see in the figure above, the land-use categories have turned into numbers, instead of land-use categories described by a string value. \n", - "\n", - "This is of course a lot harder to interpret. Let's re-do some parts to make sure we can properly link them back to the original data.\n", - "\n", - "To do so, we will first need to make sure that we know which values (numbers) are connected to each land-use category. Instead of trying to match, let's predefine this ourselves!\n", - "\n", - "We will start with creating a dictionary that allows us to couple a number to each category:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 209, - "status": "ok", - "timestamp": 1675087654241, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "iSHZ4nq247DQ" - }, - "outputs": [], - "source": [ - "value_dict = dict(zip(landuse.landuse.unique(),np.arange(0,len(landuse.landuse.unique()),1)))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 2, - "status": "ok", - "timestamp": 1675087655102, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "ZCng7Xyk47DQ" - }, - "outputs": [], - "source": [ - "value_dict['nodata'] = -1" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "zv1-QoKv47DR" - }, - "source": [ - "And we now use this dictionary to add a new column to the dataframe with the values:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 904, - "status": "ok", - "timestamp": 1675087658329, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "9c2byjZn47DR" - }, - "outputs": [], - "source": [ - "landuse['landuse_value'] = landuse.landuse.apply(lambda x: value_dict[x])" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "FP4pzj2A47DR" - }, - "source": [ - "Now let us use the `make_geocube()` function again to rasterize." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 1101, - "status": "ok", - "timestamp": 1675087662054, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "Z-1RmkFC47DR" - }, - "outputs": [], - "source": [ - "landuse_valued = make_geocube(\n", - " vector_data=XXXX,\n", - " output_crs=XXXX,\n", - " resolution=(-XXXX, XXXX),\n", - " categorical_enums={'landuse_value': landuse.landuse_value.drop_duplicates().values.tolist()\n", - "}\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "V0-NKtL147DR" - }, - "source": [ - "And let's use the original `color_dict` dictionary to find the right hex codes for each of the land-use categories" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 227, - "status": "ok", - "timestamp": 1675087671651, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "4FKdnZ4V47DR" - }, - "outputs": [], - "source": [ - "unique_classes = landuse.landuse.drop_duplicates().values.tolist()\n", - "colormap_raster = [color_dict[lu_class] for lu_class in unique_classes] " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "be1TcNAK47DR" - }, - "source": [ - "To plot the new result:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 608 - }, - "executionInfo": { - "elapsed": 1116, - "status": "ok", - "timestamp": 1675087675785, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "qBo9kmWy47DR", - "outputId": "269fe715-c71f-4f82-ca71-021b67a3ccf7" - }, - "outputs": [], - "source": [ - "fig, ax = plt.subplots(1, 1,figsize=(14,10))\n", - "\n", - "landuse_valued[\"landuse_value\"].plot(ax=ax,vmin=0,vmax=19,levels=len(unique_classes),colors=colormap_raster)\n", - "\n", - "# remove the ax labels\n", - "ax.set_xticks([])\n", - "ax.set_yticks([])\n", - "ax.set_axis_off()\n", - "\n", - "# add title\n", - "ax.set_title('')" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "cO2YPNMU-9hQ" - }, - "source": [ - "
\n",
- "Question 5: In the rasterization process, we use the `.make_geocube()` function. Please elaborate on the following: i)why is it important to specify the right coordinate system? What could happen if you choose the wrong coordinate system? ii) which resolution did you choose and why? iii)Why did the first result did not give us the right output with the correct colors? How did you solve this? \n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "dx5ZK8Vm47DS"
- },
- "source": [
- "But to be honest, this legend is still not entirely what we are looking for. So let's do some Python magic to get a legend like we desire when plotting a land-use map"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "executionInfo": {
- "elapsed": 242,
- "status": "ok",
- "timestamp": 1675087680453,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "plSLRJp947DS"
- },
- "outputs": [],
- "source": [
- "unique_classes = landuse.landuse.drop_duplicates().values.tolist()\n",
- "colormap_raster = [color_dict[lu_class] for lu_class in unique_classes] \n",
- "color_dict_raster = dict(zip(np.arange(-1,len(landuse.landuse.unique())+1,1),['#ffffff']+colormap_raster))\n",
- "\n",
- "# We create a colormar from our list of colors\n",
- "cm = ListedColormap([color_dict_raster[x] for x in color_dict_raster.keys()])\n",
- "\n",
- "# Let's also define the description of each category. Order should be respected here!\n",
- "labels = np.array(['nodata'] + unique_classes)\n",
- "len_lab = len(labels)\n",
- "\n",
- "# prepare normalizer\n",
- "## Prepare bins for the normalizer\n",
- "norm_bins = np.sort([*color_dict_raster.keys()]) + 0.5\n",
- "norm_bins = np.insert(norm_bins, 0, np.min(norm_bins) - 1.0)\n",
- "\n",
- "## Make normalizer and formatter\n",
- "norm = matplotlib.colors.BoundaryNorm(norm_bins, len_lab, clip=True)\n",
- "fmt = matplotlib.ticker.FuncFormatter(lambda x, pos: labels[norm(x)])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "MRsijBkf47DS"
- },
- "source": [
- "Let's plot the map again!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 591
- },
- "executionInfo": {
- "elapsed": 1663,
- "status": "ok",
- "timestamp": 1675087684895,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "odFEVMUc47DS",
- "outputId": "31905199-3798-4cf3-b4c6-effb7dacff10"
- },
- "outputs": [],
- "source": [
- "fig, ax = plt.subplots(1, 1,figsize=(14,10))\n",
- "\n",
- "ax = landuse_valued[\"landuse_value\"].plot(levels=len(unique_classes), cmap=cm, norm=norm)\n",
- "\n",
- "# remove the ax labels\n",
- "diff = norm_bins[1:] - norm_bins[:-1]\n",
- "tickz = norm_bins[:-1] + diff / 2\n",
- "cb = fig.colorbar(ax, format=fmt, ticks=tickz)\n",
- "\n",
- "# set title again\n",
- "fig.axes[0].set_title('')\n",
- "\n",
- "fig.axes[0].set_xticks([])\n",
- "fig.axes[0].set_yticks([])\n",
- "fig.axes[0].set_axis_off()\n",
- "\n",
- "# for some weird reason we get two colorbars, so we remove one:\n",
- "fig.delaxes(fig.axes[1])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "u9yzw5hE47DS"
- },
- "source": [
- "## 4. Extracting buildings from OpenStreetMap\n",
- "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "WMOSa6JF47DS" - }, - "source": [ - "There is a lot more data to extract from OpenStreetMap besides land-use information. Let's extract some building data. To do so, we use the *\"building\"* tag." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 119709, - "status": "ok", - "timestamp": 1675087861528, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "B6EiF_dN47DS" - }, - "outputs": [], - "source": [ - "tags = {\"building\": True}\n", - "buildings = ox.features_from_place(place_name, tags)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In case the above does not work, you can continue the assignment by using the code below (make sure you remove the hashtags to run it). If you decide to use the data as specified below, also change the map at the start to 'Kampen'." - ] - }, - { - "cell_type": "code", - "execution_count": 64, - "metadata": {}, - "outputs": [], - "source": [ - "# remote_url = 'https://github.com/ElcoK/BigData_AED/raw/main/week5/kampen_buildings.gpkg'\n", - "# file = 'kampen_buildings.gpkg'\n", - "# \n", - "# #request.urlretrieve(remote_url, file)\n", - "# buildings = gpd.GeoDataFrame.from_file('kampen_buildings.gpkg')" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "bon7osXA47DT" - }, - "source": [ - "Now let's see what information is actually extracted:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 765 - }, - "executionInfo": { - "elapsed": 37, - "status": "ok", - "timestamp": 1675087861529, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "6HdghuMv47DT", - "outputId": "dbec25d2-8c62-4420-f125-b1a4ac80c318" - }, - "outputs": [], - "source": [ - "buildings.head()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "p8dhZx1n47DT" - }, - "source": [ - "As you notice in the output of the cell above, there are many columns which just contain \"NaN\". And there even seem to be to many columns to even visualize properly in one view.\n", - "\n", - "Let's check what information is collected for the different buildings:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "executionInfo": { - "elapsed": 35, - "status": "ok", - "timestamp": 1675087861529, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "W3ZpsIkO47DT", - "outputId": "385616f8-25fe-4675-f5f0-16d7fa9b0df7" - }, - "outputs": [], - "source": [ - "buildings.columns" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "sH7NTKENA4WO" - }, - "source": [ - "
\n",
- "Question 6: Let's have a look at the extracted building information. Please describe in your own words the information it contains. Is there specific information that suprises you to see, and do you think anything is missing that you expected? \n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "8OHDhc-047DT"
- },
- "source": [
- "## 5. Analyze and visualize building stock\n",
- "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Z37JRRc747DT" - }, - "source": [ - "One interesting column is called `start_date`. This shows the building year per building. \n", - "\n", - "Let's explore this year of building a bit more.\n", - "\n", - "First, it would be interesting to get an idea how many buildings are build in each year through using the `value_counts()` function. Normally, that functions ranks the values in descending order (high to low). We are more interested in how this has developed over time. So we use the `sort_index()` function to sort the values by year. Add these two functions in the cell below." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 924, - "status": "ok", - "timestamp": 1675087884735, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "lNSe826i47DT" - }, - "outputs": [], - "source": [ - "building_year = buildings.start_date. XXXX" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "X6b_T5xs47DU" - }, - "source": [ - "There is not better way to further explore this years than through plotting it. Don't forget to add things such as a x label, y label and title. Have a look at some of the matplotlib [tutorials](https://matplotlib.org/stable/tutorials/introductory/quick_start.html). Note that you need to look at the code that also uses subplots and where they use the `ax` option." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "executionInfo": { - "elapsed": 1636, - "status": "ok", - "timestamp": 1675087889714, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "fmk6m78I47DU", - "outputId": "78f3ea6a-284d-495f-b596-12db32305128" - }, - "outputs": [], - "source": [ - "fig,ax = plt.subplots(1,1,figsize=(5,18))\n", - "\n", - "building_year.plot(kind='barh',ax=ax)\n", - "\n", - "ax.tick_params(axis='y', which='major', labelsize=7)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "AwRyhvNeBn0G" - }, - "source": [ - "
\n",
- "Question 7: Please upload a figure that shows the development of building stock over the years in your region of interest. Make sure it contains all the necessary elements (labels on the axis, title, etc.)\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "wrcM1p-m47DU"
- },
- "source": [
- "What we also noticed is that quite some buildings are identified as 'yes'. This is not very useful as it does not really say much about the use of the building. \n",
- "\n",
- "Let's see for how many buildings this is the case: "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "executionInfo": {
- "elapsed": 205,
- "status": "ok",
- "timestamp": 1675087945407,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "_4eKR2Bo47DU",
- "outputId": "54e29c44-9717-4eee-984f-9555659317be"
- },
- "outputs": [],
- "source": [
- "buildings.building.value_counts()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "As you have seen from the `value_counts` function, there are quite a few buildings with only very few tags. You could either consider to not include them in your plot at all (for example by using the `isin` function or the `query` function, see also [here](https://stackoverflow.com/questions/12096252/use-a-list-of-values-to-select-rows-from-a-pandas-dataframe)), or rename them, similar to how you named the natural land cover classes for the land-use map. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "vF6WJ9_k47DU"
- },
- "source": [
- "Now let's visualize the buildings again. We need to create a similar color dictionary as we did for the land-use categories. Now its up to you to make it!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "executionInfo": {
- "elapsed": 226,
- "status": "ok",
- "timestamp": 1675087956546,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "RYRxKXxd47DU"
- },
- "outputs": [],
- "source": [
- "color_dict = { 'yes' : \"#f1134b\", \n",
- " 'house':'#f13013', \n",
- " 'industrial':'#0f045c',\n",
- " 'farm':'#fcfcb9', \n",
- " 'bungalow':'#f13013',\n",
- " 'service':'#CB8DDB' }"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "executionInfo": {
- "elapsed": 232,
- "status": "ok",
- "timestamp": 1675087958726,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "YjRwC-K-47DU"
- },
- "outputs": [],
- "source": [
- "# Remove multiple keys from dictionary\n",
- "color_dict = {key: color_dict[key]\n",
- " for key in color_dict if key not in list(set(color_dict)-set(buildings.building.unique()))}\n",
- "\n",
- "map_dict = dict(zip(color_dict.keys(),[x for x in range(len(color_dict))]))\n",
- "buildings['col_landuse'] =buildings.building.apply(lambda x: color_dict[x])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "wGh7rbnB47DU"
- },
- "source": [
- "And plot the figure in the same manner!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 616
- },
- "executionInfo": {
- "elapsed": 3651,
- "status": "ok",
- "timestamp": 1675087966347,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "oDfamvIV47DV",
- "outputId": "dbcee552-e955-4d2b-ddda-909ab4265105"
- },
- "outputs": [],
- "source": [
- "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n",
- "\n",
- "# add color scheme\n",
- "color_scheme_map = list(color_dict.values())\n",
- "cmap = LinearSegmentedColormap.from_list(name='landuse',\n",
- " colors=color_scheme_map) \n",
- "\n",
- "# and plot the land-use map.\n",
- "buildings.plot(color=buildings['col_landuse'],ax=ax,linewidth=0)\n",
- "\n",
- "# remove the ax labels\n",
- "ax.set_xticks([])\n",
- "ax.set_yticks([])\n",
- "ax.set_axis_off()\n",
- "\n",
- "# add a legend:\n",
- "legend_elements = []\n",
- "for iter_,item in enumerate(color_dict):\n",
- " legend_elements.append(Patch(facecolor=color_scheme_map[iter_],label=item)) \n",
- "\n",
- "ax.legend(handles=legend_elements,edgecolor='black',facecolor='#fefdfd',prop={'size':12},loc=(1.02,0.2)) \n",
- "\n",
- "# add a title\n",
- "ax.set_title(place_name,fontweight='bold')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "IJalDDJvB6Yd"
- },
- "source": [
- "\n",
- "Question 8: Please upload a figure of your building stock map of your region of interest. Make sure that the interpretation is clear. If necessary, merge multiple categories into one (i.e., when some categories only contain 1 or 2 buildings).\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "3NeIaiLO47DV"
- },
- "source": [
- "## 6. Extracting roads from OpenStreetMap\n",
- "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "HpWnKfIk47DV" - }, - "source": [ - "Let's continue (and end) this tutorial with the core data in OpenStreetMap (it is even in the name): roads!\n", - "\n", - "Now, instead of using tags, we want to identify what type of roads we would like to extract. Let's first only extract roads that can be used to drive.\n", - "\n", - "The `graph_from_place()` function returns a `NetworkX` Graph element. You can read more about these graph elements in the introduction page of [NetworkX](https://networkx.org/documentation/stable/reference/introduction.html)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 13403, - "status": "ok", - "timestamp": 1675088179196, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "MpjxUadp47DV" - }, - "outputs": [], - "source": [ - "G = ox.graph_from_place(place_name, network_type=\"drive\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "NA_HAmHx47DV" - }, - "source": [ - "Unfortunately, it is bit difficult to easily view all the roads within such a Graph element. To be able to explore the data, we are going to convert it to a `Geopandas GeoDataFrame`, using the `to_pandas_edgelist()` function." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "executionInfo": { - "elapsed": 15, - "status": "ok", - "timestamp": 1675088179197, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "4dMXZb_v47DV", - "outputId": "e615d5f8-6b96-4a32-f114-ce38b1481b70" - }, - "outputs": [], - "source": [ - "roads = gpd.GeoDataFrame(nx.to_pandas_edgelist(G))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "yds5NBlF47DV" - }, - "source": [ - "In some cases, roads are classified with more than one category. If that is the case, they are captured within a `list`. To overcome this issue, we specify that we want the entire `highway` column as a `string` dtype." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "executionInfo": { - "elapsed": 294, - "status": "ok", - "timestamp": 1675088191403, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "NxTg1jb847DV" - }, - "outputs": [], - "source": [ - "roads.highway = roads.highway.astype('str')" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "rbzl5JPR47DW" - }, - "source": [ - "Now we can create a plot to see how the road network is configured." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 575 - }, - "executionInfo": { - "elapsed": 888, - "status": "ok", - "timestamp": 1675088196301, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "uyo3Ggpc47DW", - "outputId": "e58c915a-b085-498c-e7ea-8d62f237d417" - }, - "outputs": [], - "source": [ - "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n", - "\n", - "\n", - "roads.plot(column='highway',legend=True,ax=ax,legend_kwds={'loc': 'lower right'});\n", - "\n", - "\n", - "# remove the ax labels\n", - "ax.set_xticks([])\n", - "ax.set_yticks([])\n", - "ax.set_axis_off()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "M18fuTWM47DW", - "tags": [] - }, - "source": [ - "It would also be interesting to explore the network a little but more interactively. **OSMnx** has a function called `plot_graph_folium()`, which allow us to use the [folium](https://python-visualization.github.io/folium/quickstart.html#Getting-Started) package to plot data interactively on a map. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 866 - }, - "executionInfo": { - "elapsed": 1720, - "status": "ok", - "timestamp": 1675088204394, - "user": { - "displayName": "RA Odongo", - "userId": "17326618845752559881" - }, - "user_tz": -60 - }, - "id": "lpWHR0Ez47DW", - "outputId": "38be3bea-b399-4e8f-a081-210baa559ef7" - }, - "outputs": [], - "source": [ - "m1 = ox.plot_graph_folium(G, popup_attribute=\"highway\", weight=2, color=\"#8b0000\")\n", - "m1" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "M6-fqL9L47DW", - "tags": [] - }, - "source": [ - "## 7. Plot Routes Using OpenStreetMap and Folium\n", - "
" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": { - "id": "gm8NIAR947DW" - }, - "source": [ - "One of the exiting things we can do with this data is that we can compute and plot routes between two points on a map.\n", - "\n", - "Let's first select two random start and end points from the graph and compute the shortest route between them through using the `shortest_path()` function of the `NetworkX` package.\n", - "\n", - "The function `ox.nearest_nodes()` looks for the nearest point in your network based on a `X` and `Y` coordinate. For example, in the code below, the origin node is based on the northwestern corner of your bounding box, whereas the destination node is based on the coordinates of the southeastern corner of your bounding box. \n", - "\n", - "So this can also be rewritten as:\n", - "\n", - "```\n", - "origin_node = ox.nearest_nodes(G,4.65465, 56.6778) \n", - "destination_node = ox.nearest_nodes(G,4.61055, 59.5487) \n", - "route = nx.shortest_path(G, origin_node, destination_node) \n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Zi_xhqXTCi3h" - }, - "source": [ - "
\n",
- "Question 9: The last element of this tutorial is to play around with routing. Please explain in your own words what the .shortest_path() algorithm does. Include the term 'Dijkstra algorithm' in your answer.\n",
- "
"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "executionInfo": {
- "elapsed": 837,
- "status": "ok",
- "timestamp": 1675088236066,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "Dr7Ftbbh47DW"
- },
- "outputs": [],
- "source": [
- "origin_node = ox.nearest_nodes(G,area['bbox_west'].values[0], area['bbox_north'].values[0])\n",
- "destination_node = ox.nearest_nodes(G,area['bbox_east'].values[0], area['bbox_south'].values[0])\n",
- "route = nx.shortest_path(G, origin_node, destination_node)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4UuCS8G247DW"
- },
- "source": [
- "We can plot the route with folium. Like above, you can pass keyword args along to folium PolyLine to style the lines."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 866
- },
- "executionInfo": {
- "elapsed": 240,
- "status": "ok",
- "timestamp": 1675088397348,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "7QrZ6Gpi47DX",
- "outputId": "33fecead-a6d7-4090-99da-e09d0c34081a"
- },
- "outputs": [],
- "source": [
- "m2 = ox.plot_route_folium(G, route, weight=10)\n",
- "m2"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "eedHqPli47DX"
- },
- "source": [
- "Plot the route with folium on top of the previously created map\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 866
- },
- "executionInfo": {
- "elapsed": 2518,
- "status": "ok",
- "timestamp": 1675088413924,
- "user": {
- "displayName": "RA Odongo",
- "userId": "17326618845752559881"
- },
- "user_tz": -60
- },
- "id": "aHzxZAbZ47DX",
- "outputId": "01ee7d97-4792-41fb-925c-c44fb9b15a3f"
- },
- "outputs": [],
- "source": [
- "m3 = ox.plot_route_folium(G, route, route_map=m1, popup_attribute=\"name\", weight=7)\n",
- "m3"
- ]
- },
- {
- "attachments": {},
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "\n",
- "Question 10: Please add one more routes on a map and upload the resulting figure here.\n",
- "
"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "colab": {
- "provenance": [
- {
- "file_id": "https://github.com/ElcoK/BigData_AED/blob/main/week4/tutorial1.ipynb",
- "timestamp": 1675085725524
- }
- ]
- },
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.11"
- },
- "vscode": {
- "interpreter": {
- "hash": "f323064ae63d54ed8d769390a968e914fbf7abacffc63e116cd2e04a08ed2d24"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/TAA2/tutorial2.ipynb b/TAA2/tutorial2.ipynb
deleted file mode 100644
index 35384c5..0000000
--- a/TAA2/tutorial2.ipynb
+++ /dev/null
@@ -1,1846 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "s1nFKY6h-g35",
- "tags": []
- },
- "source": [
- "# Tutorial 2: Natural Hazard Risk Assessment"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "F5jQ4EU--g38"
- },
- "source": [
- "In the second tutorial of this week, we are going to use publicly available hazard data and exposure data to do a risk assessment for an area of choice within Europe. More specifically we will look at damage due to wind storms and flooding. \n",
- "\n",
- "We will use both Copernicus Land Cover data and OpenStreetMap to estimate the potential damage of natural hazards to the built environment. We will use Copernicus Land Cover data to estimate the damage to specific land-uses, whereas we will use OpenStreetMap to assess the potential damage to the road system."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "K66fASIyP--9"
- },
- "source": [
- "### Important before we start\n",
- "---\n",
- "Make sure that you save this file before you continue, else you will lose everything. To do so, go to **Bestand/File** and click on **Een kopie opslaan in Drive/Save a Copy on Drive**!\n",
- "\n",
- "Now, rename the file into Week5_Tutorial2.ipynb. You can do so by clicking on the name in the top of this screen."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "hkXRFEqH-g38"
- },
- "source": [
- "## Learning Objectives\n",
- "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "WDQJZqvL-g39" - }, - "source": [ - "- To know how to download data from the Copernicus Climate Data Store using the `cdsapi` and access it through Python.\n", - "- To be able to open and visualize this hazard data.\n", - "- To know how to access and open information from the Copernicus Land Monitoring System. Specifically the Corine Land Cover data.\n", - "- To understand the basic approach of a natural hazard risk assessment.\n", - "- To be able to use the `DamageScanner` to do a damage assessment.\n", - "- To interpret and compare the damage estimates." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "4KseyvEh-g39" - }, - "source": [ - "
Tutorial Outline
\n", - "\n", - "
- \n",
- "
- 1. Introducing the packages \n", - "
- 2. Downloading and accessing natural hazard data \n", - "
- 3. Explore the natural hazard data \n", - "
- 4. Download and access Corine Land Cover \n", - "
- 5. Perform a damage assessment using Coring Land Cover \n", - "
- 6. Perform a damage assessment using OpenStreetMap \n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "h3aMPYWo-g3-" - }, - "source": [ - "Within this tutorial, we are going to make use of the following packages: \n", - "\n", - "[**GeoPandas**](https://geopandas.org/) is a Python packagee that extends the datatypes used by pandas to allow spatial operations on geometric types.\n", - "\n", - "[**OSMnx**](https://osmnx.readthedocs.io/) is a Python package that lets you download geospatial data from OpenStreetMap and model, project, visualize, and analyze real-world street networks and any other geospatial geometries. You can download and model walkable, drivable, or bikeable urban networks with a single line of Python code then easily analyze and visualize them. You can just as easily download and work with other infrastructure types, amenities/points of interest, building footprints, elevation data, street bearings/orientations, and speed/travel time.\n", - "\n", - "[**xarray**](https://docs.xarray.dev/) is a Python package that allows for easy and efficient use of multi-dimensional arrays.\n", - "\n", - "[**Matplotlib**](https://matplotlib.org/) is a comprehensive Python package for creating static, animated, and interactive visualizations in Python. Matplotlib makes easy things easy and hard things possible.\n", - "\n", - "*We will first need to install the missing packages in the cell below. Uncomment them to make sure we can pip install them*" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "xAk93LFz-g3-", - "outputId": "b0dca5b3-3894-457e-8f2e-e0188b7b8706" - }, - "outputs": [], - "source": [ - "!pip install osmnx\n", - "!pip install rioxarray\n", - "!pip install cdsapi" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "T28OUcKK-g3_" - }, - "source": [ - "Now we will import these packages in the cell below:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "HNZcbQQ_-g4A", - "outputId": "2e8b1214-69e9-40f5-bc84-d02672bf3a5d" - }, - "outputs": [], - "source": [ - "import os\n", - "import cdsapi\n", - "import shapely \n", - "import matplotlib\n", - "import urllib3\n", - "import pyproj\n", - "\n", - "import osmnx as ox\n", - "import numpy as np\n", - "import xarray as xr\n", - "import geopandas as gpd\n", - "import pandas as pd\n", - "import matplotlib.pyplot as plt\n", - "import networkx as nx\n", - "\n", - "from matplotlib.colors import ListedColormap\n", - "from zipfile import ZipFile\n", - "from io import BytesIO\n", - "from urllib.request import urlopen\n", - "from zipfile import ZipFile\n", - "from tqdm import tqdm\n", - "\n", - "urllib3.disable_warnings()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "TJr_yUdjFMKz" - }, - "source": [ - "### Connect to google drive\n", - "---\n", - "To be able to read the data from Google Drive, we need to *mount* our Drive to this notebook.\n", - "\n", - "As you can see in the cell below, make sure that in your **My Drive** folder, you have created a **BigData** folder and within that folder, you have created a **Week5_Data** folder in which you can store the files that are required to run this analysis.\n", - "\n", - "Please go the URL when its prompted in the box underneath the following cell, and copy the authorization code in that box." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "-Xqbo0nMFTkd", - "outputId": "8ce3d2fd-fb29-44f4-b874-a77a9199a41f" - }, - "outputs": [], - "source": [ - "from google.colab import drive\n", - "drive.mount('/content/gdrive/')\n", - "\n", - "import sys\n", - "sys.path.append(\"/content/gdrive/My Drive/BigData/\")\n", - "\n", - "data_path = os.path.join('/content/gdrive/My Drive/BigData/','Week5_Data')" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "fqDSHMah-g4A", - "tags": [] - }, - "source": [ - "## 2. Downloading and accessing natural hazard data\n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "1YWXcMRV-g4A" - }, - "source": [ - "We are going to perform a damage assessment using both windstorm data and flood data for Europe.\n", - "\n", - "### Windstorm Data\n", - "\n", - "The windstorm data will be downloaded from the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/). As we have seen during the lecture, and as you can also see by browsing on this website, there is an awful lot of climate data available through this Data Store. As such, it is very valuable to understand how to access and download this information to use within an analysis. To keep things simple, we only download one dataset today: [A winter windstorm](https://cds.climate.copernicus.eu/cdsapp#!/dataset/sis-european-wind-storm-indicators?tab=overview). \n", - "\n", - "We will do so using an **API**, which is the acronym for application programming interface. It is a software intermediary that allows two applications to talk to each other. APIs are an accessible way to extract and share data within and across organizations. APIs are all around us. Every time you use a rideshare app, send a mobile payment, or change the thermostat temperature from your phone, you’re using an API.\n", - "\n", - "However, before we can access this **API**, we need to take a few steps. Most importantly, we need to register ourselves on the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/) portal. To do so, we need to register, as explained in the video clip below:\n", - "\n", - "
\n",
- "\n", - "\n", - "Now, the next step is to access the API. You can now login on the website of the [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/). After you login, you can click on your name in the top right corner of the webpage (next to the login button). On the personal page that has just opened, you will find your user ID (**uid**) and your personal **API**. You need to add those in the cell below to be able to download the windstorm.\n", - "\n", - "As you can see in the cell below, we download a specific windstorm that has occured on the 28th of October in 2013. This is storm [Carmen (also called St Jude)](https://en.wikipedia.org/wiki/St._Jude_storm). " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "42ejGQJF-g4B", - "outputId": "2936be1c-6e11-4d67-91cc-e339af3e661b" - }, - "outputs": [], - "source": [ - "uid = XXX\n", - "apikey = 'XXX'\n", - "\n", - "c = cdsapi.Client(key=f\"{uid}:{apikey}\", url=\"https://cds.climate.copernicus.eu/api/v2\")\n", - "\n", - "c.retrieve(\n", - " 'sis-european-wind-storm-indicators',\n", - " {\n", - " 'variable': 'all',\n", - " 'format': 'zip',\n", - " 'product': 'windstorm_footprints',\n", - " 'year': '2013',\n", - " 'month': '10',\n", - " 'day': '28',\n", - " },\n", - " 'Carmen.zip')" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0RKepWbV-g4B", - "tags": [] - }, - "source": [ - "### Flood Data\n", - "\n", - "The flood data we will extract from a repository maintained by the European Commission Joint Research Centre. We will download river flood hazard maps from their [Flood Data Collection](https://data.jrc.ec.europa.eu/dataset/1d128b6c-a4ee-4858-9e34-6210707f3c81). \n", - "\n", - "Here we do not need to use an API and we also do not need to register ourselves, so we can download any of the files directly. To do so, we use the `urllib` package." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 363 - }, - "id": "gdS3cvIv-g4B", - "outputId": "233e924d-c368-471b-9590-5b13c9e4e806" - }, - "outputs": [], - "source": [ - "## this is the link to the 1/100 flood map for Europe\n", - "zipurl = 'https://jeodpp.jrc.ec.europa.eu/ftp/jrc-opendata/FLOODS/EuropeanMaps/floodMap_RP100.zip'\n", - "\n", - "# and now we open and extract the data\n", - "with urlopen(zipurl) as zipresp:\n", - " with ZipFile(BytesIO(zipresp.read())) as zfile:\n", - " zfile.extractall(data_path)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "IJrBotZevbZ6" - }, - "source": [ - "The download and zip in the cell above sometimes does not work. If that is indeed the case (e.g., when it seems to remain stuck), download the files manually through the link and upload them in the data folder for this week (as explained at the start of this tutorial.)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "GCBoCWv2-g4B" - }, - "source": [ - "### Set location to explore\n", - "---\n", - "Before we continue, we need to specify our location of interest. This should be a province that will have some flooding and relative high wind speeds occuring (else we will find zero damage).\n", - "\n", - "Specify the region in the cell below by using the `geocode_to_gdf()` function again. I have chosen Gelderland, but feel free to choose a different region. It would be good to double check later on whether there is actually some high wind speeds and flooding in your chosen area." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "bQfZLu6T-g4B" - }, - "outputs": [], - "source": [ - "place_name = \"Gelderland, The Netherlands\"\n", - "area = ox.geocode_to_gdf(place_name)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "qES1iP8N-g4C" - }, - "source": [ - "## 3. Explore the natural hazard data\n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "v89JIHRz-g4C" - }, - "source": [ - "As you can see in the section above, we have downloaded the storm footprint in a zipfile. Let's open the zipfile and load the dataset using the `xarray` package through the `open_dataset()` function." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "I8FlFSFw-g4C" - }, - "source": [ - "### Windstorm Data\n", - "---" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "UFcP1DTs-g4C" - }, - "outputs": [], - "source": [ - "with ZipFile('Carmen.zip') as zf:\n", - " \n", - " # Let's get the filename first\n", - " file = zf.namelist()[0]\n", - " \n", - " # And now we can open and select the file within Python\n", - " with zf.open(file) as f:\n", - " windstorm_europe = xr.open_dataset(f)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "KFZEcL_C-g4C" - }, - "source": [ - "Let's have a look at the storm we have downloaded!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "windstorm_europe['FX'].plot()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Xz30IPy7QAiY" - }, - "source": [ - "
\n",
- "Question 1: Describe windstorm Carmen. When did this event happen, which areas were most affected? Can you say something about the maximum wind speeds in different areas, based on the plot? And what does FX mean?\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "zLb6mFA9-g4D"
- },
- "source": [
- "Unfortunately, our data does not have a proper coordinate system defined yet. As such, we will need to use the `rio.write_crs()` function to set the coordinate system to **EPSG:4326** (the standard global coordinate reference system). \n",
- "\n",
- "We also need to make sure that the functions will know what the exact parameters are that we have to use for our spatial dimenions (e.g. longitude and latitude). It prefers to be named `x` and `y`. So we use the `rename()` function before we use the `set_spatial_dims()` function."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 291
- },
- "id": "WqAmuTVo-g4D",
- "outputId": "0337b286-bc80-4d33-f182-f35d0458686f"
- },
- "outputs": [],
- "source": [
- "windstorm_europe.rio.write_crs(4326, inplace=True)\n",
- "windstorm_europe = windstorm_europe.rename({'Latitude': 'y','Longitude': 'x'})\n",
- "windstorm_europe.rio.set_spatial_dims(x_dim=\"x\",y_dim=\"y\", inplace=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "sHxrXrOkRNqA"
- },
- "source": [
- "\n",
- "Question 2: Climate data is often stored as a netCDF file. Please describe what a netCDF file is. Which information is stored in the netCDF file we have downloaded for the windstorm? What type of metadata does it contain?\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "E2cHd5hV-g4D"
- },
- "source": [
- "Following, we also make sure it will be in the European coordinate system **EPSG:3035** to ensure we can easily use it together with the other data. To do so, we use the `reproject()` function. You can simple add the number of the coordinate system."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "AqZvdyGk-g4D"
- },
- "outputs": [],
- "source": [
- "windstorm_europe = windstorm_europe.rio.reproject(XXXX)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "6o_aa1sp-g4D"
- },
- "source": [
- "Now we have all the information to clip the windstorm data to our area of interest:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "NRQV7Io3-g4E"
- },
- "outputs": [],
- "source": [
- "windstorm_map = windstorm_europe.rio.clip(area.envelope.values, area.crs)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "WlWwtOKH-g4E"
- },
- "source": [
- "And let's have a look as well by using the `plot()` function. Please note that the legend is in meters per second."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 324
- },
- "id": "AzolOfUC-g4E",
- "outputId": "3de9b3d2-8907-48ee-b828-4b03a097a80d"
- },
- "outputs": [],
- "source": [
- "windstorm_map['FX']. XXXX"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "5jQ2_51Z-g4E"
- },
- "source": [
- "### Flood data\n",
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "btJy-1aR-g4E"
- },
- "source": [
- "And similarly, we want to open the flood map. But now we do not have to unzip the file anymore and we can directly open it through using `xarray`:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "oRvMEvkm-g4E"
- },
- "outputs": [],
- "source": [
- "flood_map_path = os.path.join(data_path,'floodmap_EFAS_RP100_C.tif')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 328
- },
- "id": "eEA1EKlt-g4E",
- "outputId": "1968dec7-b036-4df4-bd9f-b16c41102b33"
- },
- "outputs": [],
- "source": [
- "flood_map = xr.open_dataset(flood_map_path, engine=\"rasterio\")\n",
- "flood_map"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "O4EIxBK_-g4F"
- },
- "source": [
- "And let's make sure we set all the variables and the CRS correctly again to be able to open the data properly. Note that we now use **EPSG:3035**. This is the standard coordinate system for Europe, in meters (instead of degrees)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 291
- },
- "id": "LxTaPlzW-g4F",
- "outputId": "8c9d5bcb-3ceb-4628-88fb-0aabe161365f",
- "tags": []
- },
- "outputs": [],
- "source": [
- "flood_map.rio.write_crs(3035, inplace=True)\n",
- "flood_map.rio.set_spatial_dims(x_dim=\"x\",y_dim=\"y\", inplace=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "utiDVMpJ-g4F"
- },
- "source": [
- "Now it is pretty difficult to explore the data for our area of interest, so let's clip the flood data. \n",
- "\n",
- "We want to clip our flood data to our chosen area. The code, however, is very inefficient and will run into memories issues on Google Colab. As such, we first need to clip it by using a bounding box, followed by the actual clip.\n",
- "\n",
- "\n",
- "Question 3: Please provide the lines of code below in which you show how you have clipped the flood map to your area.\n",
- "
\n",
- "\n",
- "*A few hints*:\n",
- "\n",
- "* carefully read the documentation of the `.clip_box()` function of rioxarray. Which information do you need? \n",
- "* is the GeoDataFrame of your region (the area GeoDataframe) in the same coordinate system? Perhaps you need to convert it using the `.to_crs()` function. \n",
- "* how do you get the bounds from your area GeoDataFrame? \n",
- "* The final step of the clip would be to use the `.rio.clip()` function, using the actual area file and the flood map clipped to the bounding box. Please note that you should **not** use the envelope here, like we did in the previous clip. Here we really want to use the exact geometry values.\n",
- "\n",
- "As you will see, we first clip it very efficiently using the bounding box. After that, we do an exact clip."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "PuJ290bGISlL"
- },
- "outputs": [],
- "source": [
- "min_lon = area.to_crs(epsg=3035).bounds.minx.values[0]\n",
- "min_lat = area.to_crs(epsg=3035).bounds.miny\n",
- "max_lon = area.to_crs(epsg=3035).bounds\n",
- "max_lat = area.to_crs(epsg=3035).\n",
- "\n",
- "flood_map_area = flood_map.rio.clip_box(minx=.... )\n",
- "flood_map_area = flood_map_area.rio.clip(area.XXXX.values, area.crs)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "xNjj8RT--g4F"
- },
- "source": [
- "And let's have a look as well. Please note that the legend is in meters."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 324
- },
- "id": "v_wldK5x-g4F",
- "outputId": "15c3b526-d094-40e1-dae1-9dbfe57339ea"
- },
- "outputs": [],
- "source": [
- "flood_map_area['band_data'].plot(cmap='Blues',vmax=10)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "iiOgZMi0-g4F",
- "tags": []
- },
- "source": [
- "## 4. Download and access Copernicus Land Cover data\n",
- "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "JTp2SMuK-g4F" - }, - "source": [ - "Unfortunately, there is no API option to download the [Corine Land Cover](https://land.copernicus.eu/pan-european/corine-land-cover) data. We will have to download the data from the website first.\n", - "\n", - "To do so, we will first have to register ourselves again on the website. Please find in the video clip below how to register yourself on the website of the [Copernicus Land Monitoring Service](https://land.copernicus.eu/):\n", - "\n", - "
\n",
- "\n",
- "Now click on the Login button in the top right corner to login on the website. There are many interesting datasets on this website, but we just want to download the Corine Land Cover data, and specifically the latest version: [Corine Land Cover 2018](https://land.copernicus.eu/pan-european/corine-land-cover/clc2018?tab=download). To do so, please select the **Corine Land Cover - 100 meter**. Now click on the large green Download button. Your download should start any minute.\n",
- "\n",
- "Slightly annoying, the file you have downloaded is double zipped. Its slightly inconvenient to open this through Python and within Google Drive. So let's unzip it twice outside of Python (on your local machine) and then direct yourself to the `DATA` directory within the unzipped file. Here you can find a file called `U2018_CLC2018_V2020_20u1.tif`. Drop this file into this week's data directory, as specified at the start of this tutorial when we mounted our Google Drive."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "e3OV0J1N-g4G"
- },
- "outputs": [],
- "source": [
- "CLC_location = os.path.join(data_path,'U2018_CLC2018_V2020_20u1.tif')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "stGRY3U5-g4G",
- "outputId": "afaec16f-d144-46ba-af4c-6cc03df68763"
- },
- "outputs": [],
- "source": [
- "CLC = xr.open_dataset(CLC_location, engine=\"rasterio\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "EF_na6jnKAvZ"
- },
- "source": [
- "Similarly to the flood map data, we need to do a two-stage clip again (like we did before in this tutorial to ensure we get only our area of interest without exceeding our RAM."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "LHBLJONMJ_Zf"
- },
- "outputs": [],
- "source": [
- "CLC_region = CLC.rio.clip_box(minx=.....,)\n",
- "CLC_region = CLC_region.rio.clip(area.geometry.values,area.crs)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 291
- },
- "id": "CE5AhRYt-g4G",
- "outputId": "861397ac-9256-4e1d-a876-c17f0d18cfca"
- },
- "outputs": [],
- "source": [
- "CLC_region = CLC_region.rename({'x': 'lat','y': 'lon'})\n",
- "CLC_region.rio.set_spatial_dims(x_dim=\"lat\",y_dim=\"lon\", inplace=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "dGDKag4cKTnO"
- },
- "source": [
- "And now we create a *color_dict* again, similarly as we did for the raster data in the previous tutorial "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "g7I4fbKs-g4G"
- },
- "outputs": [],
- "source": [
- "CLC_values = [111, 112, 121, 122, 123, 124, 131, 132, 133, 141, 142, 211, 212, 213, 221, 222, 223, 231, 241, 242,\n",
- " 243, 244, 311, 312, 313, 321, 322, 323, 324, 331, 332, 333, 334, 335, 411, 412, 421, 422, 423, 511, 512, 521, 522, 523]\n",
- "\n",
- "CLC_colors = ['#E6004D', '#FF0000', '#CC4DF2', '#CC0000', '#E6CCCC', '#E6CCE6', '#A600CC', '#A64DCC', '#FF4DFF', '#FFA6FF', '#FFE6FF', '#FFFFA8', '#FFFF00', '#E6E600',\n",
- " '#E68000', '#F2A64D', '#E6A600', '#E6E64D', '#FFE6A6', '#FFE64D', '#E6CC4D', '#F2CCA6', '#80FF00', '#00A600',\n",
- " '#4DFF00', '#CCF24D', '#A6FF80', '#A6E64D', '#A6F200', '#E6E6E6', '#CCCCCC', '#CCFFCC', '#000000', '#A6E6CC',\n",
- " '#A6A6FF', '#4D4DFF', '#CCCCFF', '#E6E6FF', '#A6A6E6', '#00CCF2', '#80F2E6', '#00FFA6', '#A6FFE6', '#E6F2FF']"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "Y9Ye19av-g4G"
- },
- "outputs": [],
- "source": [
- "color_dict_raster = dict(zip(CLC_values,CLC_colors))\n",
- "\n",
- "# We create a colormar from our list of colors\n",
- "cm = ListedColormap(CLC_colors)\n",
- "\n",
- "# Let's also define the description of each category : 1 (blue) is Sea; 2 (red) is burnt, etc... Order should be respected here ! Or using another dict maybe could help.\n",
- "labels = np.array(CLC_values)\n",
- "len_lab = len(labels)\n",
- "\n",
- "# prepare normalizer\n",
- "## Prepare bins for the normalizer\n",
- "norm_bins = np.sort([*color_dict_raster.keys()]) + 0.5\n",
- "norm_bins = np.insert(norm_bins, 0, np.min(norm_bins) - 1.0)\n",
- "\n",
- "## Make normalizer and formatter\n",
- "norm = matplotlib.colors.BoundaryNorm(norm_bins, len_lab, clip=True)\n",
- "fmt = matplotlib.ticker.FuncFormatter(lambda x, pos: labels[norm(x)])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "5CQoFmdqKcMe"
- },
- "source": [
- "And let's plot the Corine Land Cover data for our area of interest"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 650
- },
- "id": "JIPpIZRh-g4G",
- "outputId": "59ecd822-c679-42a5-eb5f-ead83aee5102"
- },
- "outputs": [],
- "source": [
- "fig, ax = plt.subplots(1, 1,figsize=(14,10))\n",
- "\n",
- "CLC_region[\"band_data\"].plot(ax=ax,levels=len(CLC_colors),colors=CLC_colors)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "AdSaPHzIfNbi"
- },
- "source": [
- "\n",
- "Question 4: Describe the different land-use classes within your region that you see on the Corine Land Cover map.\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "9zbsc7d_-g4G",
- "tags": []
- },
- "source": [
- "## 5. Perform a damage assessment using Coring Land Cover\n",
- "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Agxq2HqY-g4H" - }, - "source": [ - "To calculate the potential damage to both windstorms and floods, we use stage-damage curves, which relate the intensity of the hazard to the fraction of maximum damage that can be sustained by a certain land use. As you can see on the Corine Land Cover map that we just plotted, there are a lot of land use classes (44), though not all will suffer damage from either the windstorm or the flood event. For each of the land-use classes a curve and a maximum damage number are assigned.\n", - "\n", - "To Assess the damage for both the flood and windstorm event, we are going to make use of the [DamageScanner](https://damagescanner.readthedocs.io/en/latest/), which is a tool to calculate potential flood damages based on inundation depth and land use using depth-damage curves in the Netherlands. The DamageScanner was originally developed for the 'Netherlands Later' project [(Klijn et al., 2007)](https://www.rivm.nl/bibliotheek/digitaaldepot/WL_rapport_Overstromingsrisicos_Nederland.pdf). The original land-use classes were based on the Land-Use Scanner in order to evaluate the effect of future land-use change on flood damages. We have tailored the input of the DamageScanner to make sure it can estimate the damages using Corine Land Cover.\n", - "\n", - "Because the simplicity of the model, we can use this for any raster-based hazard map with some level of intensity. Hence, we can use it for both hazards." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "5m_RAcp_fraF" - }, - "source": [ - "
\n",
- "Question 5: Describe in your own words what the `DamageScanner()` function does. Please walk us through the different steps. Which inputs do you need to be able to run this damage assessment?\n",
- "
"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "jDrTp44Q-g4H"
- },
- "outputs": [],
- "source": [
- "def DamageScanner(landuse_map,inun_map,curve_path,maxdam_path,cellsize=100):\n",
- " \n",
- " # load land-use map\n",
- " landuse = landuse_map.copy()\n",
- " \n",
- " # Load inundation map\n",
- " inundation = inun_map.copy()\n",
- " \n",
- " inundation = np.nan_to_num(inundation) \n",
- "\n",
- " # Load curves\n",
- " if isinstance(curve_path, pd.DataFrame):\n",
- " curves = curve_path.values \n",
- " elif isinstance(curve_path, np.ndarray):\n",
- " curves = curve_path\n",
- "\n",
- " #Load maximum damages\n",
- " if isinstance(maxdam_path, pd.DataFrame):\n",
- " maxdam = maxdam_path.values \n",
- " elif isinstance(maxdam_path, np.ndarray):\n",
- " maxdam = maxdam_path\n",
- " \n",
- " # Speed up calculation by only considering feasible points\n",
- " inun = inundation * (inundation>=0) + 0\n",
- " inun[inun>=curves[:,0].max()] = curves[:,0].max()\n",
- " waterdepth = inun[inun>0]\n",
- " landuse = landuse[inun>0]\n",
- "\n",
- " # Calculate damage per land-use class for structures\n",
- " numberofclasses = len(maxdam)\n",
- " alldamage = np.zeros(landuse.shape[0])\n",
- " damagebin = np.zeros((numberofclasses, 4,))\n",
- " for i in range(0,numberofclasses):\n",
- " n = maxdam[i,0]\n",
- " damagebin[i,0] = n\n",
- " wd = waterdepth[landuse==n]\n",
- " alpha = np.interp(wd,((curves[:,0])),curves[:,i+1])\n",
- " damage = alpha*(maxdam[i,1]*cellsize)\n",
- " damagebin[i,1] = sum(damage)\n",
- " damagebin[i,2] = len(wd)\n",
- " if len(wd) == 0:\n",
- " damagebin[i,3] = 0\n",
- " else:\n",
- " damagebin[i,3] = np.mean(wd)\n",
- " alldamage[landuse==n] = damage\n",
- "\n",
- " # create pandas dataframe with output\n",
- " loss_df = pd.DataFrame(damagebin.astype(float),columns=['landuse','losses','area','avg_depth']).groupby('landuse').sum()\n",
- " \n",
- " # return output\n",
- " return loss_df.sum().values[0],loss_df"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Y7PB8oJz-g4H"
- },
- "source": [
- "### Windstorm Damage\n",
- "---\n",
- "To estimate the potential damage of our windstorm, we use the vulnerability curves developed by [Yamin et al. (2014)](https://www.sciencedirect.com/science/article/pii/S2212420914000466). Following [Yamin et al. (2014)](https://www.sciencedirect.com/science/article/pii/S2212420914000466), we will apply a sigmoidal vulnerability function satisfying two constraints: (i) a minimum threshold for the occurrence of damage with an upper bound of 100% direct damage; (ii) a high power-law function for the slope, describing an increase in damage with increasing wind speeds. Due to the limited amount of vulnerability curves available for windstorm damage, we will use the damage curve that represents low-rise *reinforced masonry* buildings for all land-use classes that may contain buildings. Obviously, this is a large oversimplification of the real world, but this should be sufficient for this exercise. When doing a proper stand-alone windstorm risk assessment, one should take more effort in collecting the right vulnerability curves for different building types. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "-RxvAEQh-g4H",
- "tags": []
- },
- "outputs": [],
- "source": [
- "wind_curves = pd.read_excel(\"https://github.com/ElcoK/BigData_AED/raw/main/week5/damage_curves.xlsx\",sheet_name='wind_curves')\n",
- "maxdam = pd.read_excel(\"https://github.com/ElcoK/BigData_AED/raw/main/week5/damage_curves.xlsx\",sheet_name='maxdam')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "uLZ7vl1w-g4H"
- },
- "source": [
- "Unfortunately, we run into a *classic* problem when we want to overlay the windstorm data with the Corine Land Cover data. The windstorm data is not only stored in a different coordinate system (we had to convert it from **EPSG:4326** to **EPSG:3035**), it is in a different resolution (**1km** instead of the **100m** of Corine Land Cover). \n",
- "\n",
- "Let's first have a look how our clipped data look's like. If you have decided to use Gelderland, you will see that we have 102 columns (our Lattitude/lat) and 74 rows (our Longitude/lon). If you scroll above to our Corine Land Cover data, you see that dimensions are different: 1270 columns (Lattitude/lat/x) and 870 rows (Longitude/lon/y). "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 291
- },
- "id": "gG2OXOySj8Ra",
- "outputId": "67135491-52de-4571-f8c9-7b6a74e19b66"
- },
- "outputs": [],
- "source": [
- "windstorm_map"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "igfFBqcK-g4H"
- },
- "source": [
- "The first thing we are going to do is try to make sure our data will be in the correct resolution (moving from **1km** to **100m**). To do so, we will use the `rio.reproject()` function. You will see that specify the resolution as **100**. Because **EPSG:3035** is a coordinate system in meters, we can simply use meters to define the resolution. We use the `rio.clip()` function to make sure we clip it again to our area of interest. The function below (`match_rasters`) will do the hard work for us. Please note all the input variables to understand what's happening."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "Kud2CWEDhz1O"
- },
- "outputs": [],
- "source": [
- "def match_rasters(hazard,landuse,haz_crs=3035,lu_crs=3035,resolution=100,hazard_col=['FX']):\n",
- " \"\"\"\n",
- " Clips, reprojections, and matches the resolutions of two rasters, `hazard` and `landuse`,\n",
- " to prepare them for further analysis.\n",
- "\n",
- " Parameters\n",
- " ----------\n",
- " hazard : xarray.DataArray\n",
- " A 2D or 3D array containing hazard data.\n",
- " landuse : xarray.DataArray\n",
- " A 2D array containing land use data.\n",
- " haz_crs : int, optional\n",
- " The CRS of `hazard`. Default is EPSG:3035.\n",
- " lu_crs : int, optional\n",
- " The CRS of `landuse`. Default is EPSG:3035.\n",
- " resolution : float, optional\n",
- " The desired resolution in meters for both `hazard` and `landuse` after reprojection. Default is 100.\n",
- " hazard_col : list, optional\n",
- " A list of column names or indices for the hazard variable. Default is ['FX'].\n",
- "\n",
- " Returns\n",
- " -------\n",
- " tuple\n",
- " A tuple containing two xarray.DataArray objects:\n",
- " - The land use variable with matching resolution and dimensions to the hazard variable.\n",
- " - The hazard variable clipped to the extent of the land use variable, with matching resolution and dimensions.\n",
- " \"\"\"\n",
- " \n",
- " # Set the crs of the hazard variable to haz_crs\n",
- " hazard.rio.write_crs(haz_crs, inplace=True)\n",
- "\n",
- " # Set the x and y dimensions in the hazard variable to 'x' and 'y' respectively\n",
- " hazard.rio.set_spatial_dims(x_dim=\"x\",y_dim=\"y\", inplace=True)\n",
- "\n",
- " # Reproject the landuse variable from EPSG:4326 to EPSG:3857\n",
- " landuse = CLC_region.rio.reproject(\"EPSG:3857\",resolution=resolution)\n",
- "\n",
- " # Get the minimum longitude and latitude values in the landuse variable\n",
- " min_lon = landuse.x.min().to_dict()['data']\n",
- " min_lat = landuse.y.min().to_dict()['data']\n",
- "\n",
- " # Get the maximum longitude and latitude values in the landuse variable\n",
- " max_lon = landuse.x.max().to_dict()['data']\n",
- " max_lat = landuse.y.max().to_dict()['data']\n",
- "\n",
- " # Create a bounding box using the minimum and maximum latitude and longitude values\n",
- " area = gpd.GeoDataFrame([shapely.box(min_lon,min_lat,max_lon, max_lat)],columns=['geometry'])\n",
- "\n",
- " # Set the crs of the bounding box to EPSG:3857\n",
- " area.crs = 'epsg:3857'\n",
- "\n",
- " # Convert the crs of the bounding box to EPSG:4326\n",
- " area = area.to_crs(f'epsg:{haz_crs}')\n",
- "\n",
- " # Clip the hazard variable to the extent of the bounding box\n",
- " hazard = hazard.rio.clip(area.geometry.values, area.crs)\n",
- "\n",
- " # Reproject the hazard variable to EPSG:3857 with the desired resolution\n",
- " hazard = hazard.rio.reproject(\"EPSG:3857\",resolution=resolution)\n",
- "\n",
- " # Clip the hazard variable again to the extent of the bounding box\n",
- " hazard = hazard.rio.clip(area.geometry.values, area.crs)\n",
- "\n",
- " # If the hazard variable has fewer columns and rows than the landuse variable, reproject the landuse variable to match the hazard variable\n",
- " if (len(hazard.x)\n",
- "Question 6: Describe the various steps you have taken to make sure that the windstorm map is now exactly the same extent as the corine land cover map. Feel free to include lines of code in your answer and also describe the different functions you have used along the way.\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "LW158xPh-g4J"
- },
- "source": [
- "Now its finally time to do our damage assessment! To do so, we need to convert our data to `numpy.arrays()` to do our calculation:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "QZIzWIeP-g4J"
- },
- "outputs": [],
- "source": [
- "landuse_map = CLC_region_wind['band_data'].to_numpy()[0,:,:]\n",
- "wind_map = windstorm['FX'].to_numpy()[0,:,:]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "aBqqRqbkmA1Y",
- "outputId": "709d6c91-4ad9-4e27-e6a9-e6201df32dc7"
- },
- "outputs": [],
- "source": [
- "wind_map.shape"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "J9QHyhSU-g4J"
- },
- "source": [
- "And remember that our windstorm data was stored in **m/s**. Hence, we need to convert it to **km/h**:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "GqdUCXD_-g4J"
- },
- "outputs": [],
- "source": [
- "wind_map_kmh = wind_map*XXX"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "ln7NqRB1-g4J"
- },
- "source": [
- "And now let's run the DamageScanner to obtain the damage results"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "y_g0pj1h-g4J",
- "tags": []
- },
- "outputs": [],
- "source": [
- "wind_damage_CLC = DamageScanner(landuse_map,wind_map_kmh,wind_curves,maxdam)[1]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 1000
- },
- "id": "-6DbFD_JeFA1",
- "outputId": "fb251350-8885-4dde-e665-893fa04cde6e"
- },
- "outputs": [],
- "source": [
- "wind_damage_CLC"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "5UNySYvk-g4J",
- "tags": []
- },
- "source": [
- "### Flood Damage\n",
- "---\n",
- "To Assess the flood damage, we are again going to make use of the [DamageScanner](https://damagescanner.readthedocs.io/en/latest/). The Corine Land Cover data is widely used in European flood risk assessments. As such, we can simply make use of pre-developed curves. We are using the damage curves as developed by Huizinga et al. (2007). Again, let's first load the maximum damages and the depth-damage curves:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "ua2xyAGW-g4J"
- },
- "outputs": [],
- "source": [
- "flood_curves = pd.read_excel(\"https://github.com/ElcoK/BigData_AED/raw/main/week5/damage_curves.xlsx\",sheet_name='flood_curves',engine='openpyxl')\n",
- "maxdam = pd.read_excel(\"https://github.com/ElcoK/BigData_AED/raw/main/week5/damage_curves.xlsx\",sheet_name='maxdam')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "HT54wRvs-g4K"
- },
- "source": [
- "And convert our data to `numpy.arrays()` to do our calculation:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "qzXKNmg2-g4K"
- },
- "outputs": [],
- "source": [
- "landuse_map = CLC_region['band_data'].to_numpy()\n",
- "flood_map = flood_map_area['band_data'].to_numpy()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "ttGra99k-g4K"
- },
- "source": [
- "And now let's run the DamageScanner to obtain the damage results"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "2qL8UATu-g4K"
- },
- "outputs": [],
- "source": [
- "flood_damage_CLC = DamageScanner(landuse_map,flood_map,flood_curves,maxdam)[1]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 1000
- },
- "id": "WnWu6AMUeFA2",
- "outputId": "a16faddd-7a66-40e4-f103-9a1c1f723735"
- },
- "outputs": [],
- "source": [
- "flood_damage_CLC"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "hGWhg9fogvM3"
- },
- "source": [
- "\n",
- "Question 7: Describe the results of the flood and wind damage assessments. Do you notice any differences between the outcomes? Do you observe specific land-use classes that are severely damaged?\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "E0ohOKwd-g4K"
- },
- "source": [
- "## 6. Perform a damage assessment of the road network using OpenStreetMap\n",
- "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "bKFmTKpj-g4K" - }, - "source": [ - "Generally, wind damage does not cause much damage to roads. There will be clean-up cost of the trees that will fall on the roads, but structural damage is rare. As such, we will only do a flood damage assessment for the road network of our region.\n", - "\n", - "To do so, we first need to extract the roads again. We will use the `graph_from_place()` function again to do so. However, the area will be to large to extract roads, so we will focus our analysis on the main network." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "CUqFG7AD-g4K" - }, - "outputs": [], - "source": [ - "cf = '[\"highway\"~\"trunk|motorway|primary|secondary\"]'\n", - "G = ox.graph_from_place(place_name, network_type=\"drive\", custom_filter=cf)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "modIJTEz-g4K" - }, - "source": [ - "And convert the road network to a `geodataframe`, as done in the previous tutorial as well." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "sxOBriok-g4K", - "outputId": "6ddc37c2-d8d1-4222-f989-08a2f9d08d43" - }, - "outputs": [], - "source": [ - "roads = gpd.GeoDataFrame(nx.to_pandas_edgelist(G))\n", - "roads.highway = roads.highway.astype('str')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In case the above does not work, you can continue the assignment by using the code below (make sure you remove the hashtags to run it). " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "#from urllib import request\n", - "# remote_url = 'https://github.com/ElcoK/BigData_AED/raw/main/week5/kampen_roads.gpkg'\n", - "# file = 'kampen_roads.gpkg'\n", - "# \n", - "# #request.urlretrieve(remote_url, file)\n", - "# roads = gpd.GeoDataFrame.from_file('kampen_roads.gpkg')" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "VIaMGLxA-g4K" - }, - "source": [ - "And lets have a look at the data:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 495 - }, - "id": "dx_299FS-g4L", - "outputId": "ffdab479-6a25-4794-da3a-cbacf2accaa4" - }, - "outputs": [], - "source": [ - "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n", - "\n", - "\n", - "roads.plot(column='highway',legend=True,ax=ax,legend_kwds={'loc': 'lower right'});\n", - "\n", - "\n", - "# remove the ax labels\n", - "ax.set_xticks([])\n", - "ax.set_yticks([])\n", - "ax.set_axis_off()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "PSGo7dC3-g4L" - }, - "source": [ - "It is actually quite inconvenient to have all these lists in the data for when we want to do the damage assessment. Let's clean this up a bit. To do so, we first make sure that all the lists are represented as actual lists, and not lists wrapped within a string." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "S_LZSRI6-g4L" - }, - "outputs": [], - "source": [ - "roads.highway = roads.highway.apply(lambda x: x.strip('][').split(', '))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "cwQKiRDd-g4L" - }, - "source": [ - "Now we just need to grab the first element of each of the lists." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "qe86tcET-g4L" - }, - "outputs": [], - "source": [ - "roads.highway = roads.highway.apply(lambda x: x[0] if isinstance(x, list) else x)\n", - "roads.highway = roads.highway.str.replace(\"'\",\"\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "TkyDDDIP-g4L" - }, - "source": [ - "And let's have a look whether this worked:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 495 - }, - "id": "f5qPSBmq-g4L", - "outputId": "6ac152f2-14f1-4e6c-c53a-9d68db347ce6" - }, - "outputs": [], - "source": [ - "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n", - "\n", - "roads.plot(column='highway',legend=True,ax=ax,legend_kwds={'loc': 'upper left','ncol':1});\n", - "\n", - "# remove the ax labels\n", - "ax.set_xticks([])\n", - "ax.set_yticks([])\n", - "ax.set_axis_off()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "u0crazq8iQjQ" - }, - "source": [ - "
\n",
- "Question 8: Upload a figure of the cleaned road network (e.g. in which you do not see any of the listed road types anymore)\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "J63sExRp-g4L"
- },
- "source": [
- "Nice! now let's start with the damage calculation. As you already have may have noticed, our data is now not stored in raster format, but in vector format. One way to deal with this issue is to convert our vector data to raster data, but we will lose a lot of information and detail. As such, we will perform the damage assessment on the road elements, using the xarray flood map.\n",
- "\n",
- "Let's start with preparing the flood data into vector format:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "uHmaZFXV-g4L"
- },
- "outputs": [],
- "source": [
- "# get the mean values\n",
- "flood_map_vector = flood_map_area['band_data'].to_dataframe().reset_index()\n",
- "\n",
- "# create geometry values and drop lat lon columns\n",
- "flood_map_vector['geometry'] = [shapely.points(x) for x in list(zip(flood_map_vector['x'],flood_map_vector['y']))]\n",
- "flood_map_vector = flood_map_vector.drop(['x','y','band','spatial_ref'],axis=1)\n",
- "\n",
- "# drop all non values to reduce size\n",
- "flood_map_vector = flood_map_vector.loc[~flood_map_vector['band_data'].isna()].reset_index(drop=True)\n",
- "\n",
- "# and turn them into squares again:\n",
- "flood_map_vector.geometry= shapely.buffer(flood_map_vector.geometry,distance=100/2,cap_style='square').values"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "sKb-ig4Q-g4M"
- },
- "source": [
- "And let's plot the results:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 306
- },
- "id": "LL3YU6r1-g4M",
- "outputId": "e0a9e61f-e376-436c-a11a-fa58651bf15a"
- },
- "outputs": [],
- "source": [
- "gpd.GeoDataFrame(flood_map_vector.copy()).plot(column='band_data',cmap='Blues',vmax=5,linewidth=0)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "XBsxnhjN-g4M"
- },
- "source": [
- "We will need a bunch of functions to make sure we can do our calculations. They are specified below. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "T-XfgGLB-g4M"
- },
- "outputs": [],
- "source": [
- "def reproject(df_ds,current_crs=\"epsg:4326\",approximate_crs = \"epsg:3035\"):\n",
- " geometries = df_ds['geometry']\n",
- " coords = shapely.get_coordinates(geometries)\n",
- " transformer=pyproj.Transformer.from_crs(current_crs, approximate_crs,always_xy=True)\n",
- " new_coords = transformer.transform(coords[:, 0], coords[:, 1])\n",
- " \n",
- " return shapely.set_coordinates(geometries.copy(), np.array(new_coords).T) \n",
- "\n",
- "def buffer_assets(assets,buffer_size=100):\n",
- " assets['buffered'] = shapely.buffer(assets.geometry.values,buffer_size)\n",
- " return assets\n",
- "\n",
- "def overlay_hazard_assets(df_ds,assets):\n",
- "\n",
- " #overlay \n",
- " hazard_tree = shapely.STRtree(df_ds.geometry.values)\n",
- " if (shapely.get_type_id(assets.iloc[0].geometry) == 3) | (shapely.get_type_id(assets.iloc[0].geometry) == 6):\n",
- " return hazard_tree.query(assets.geometry,predicate='intersects') \n",
- " else:\n",
- " return hazard_tree.query(assets.buffered,predicate='intersects')\n",
- " \n",
- "def get_damage_per_asset(asset,df_ds,assets):\n",
- " # find the exact hazard overlays:\n",
- " get_hazard_points = df_ds.iloc[asset[1]['hazard_point'].values].reset_index()\n",
- " get_hazard_points = get_hazard_points.loc[shapely.intersects(get_hazard_points.geometry.values,assets.iloc[asset[0]].geometry)]\n",
- "\n",
- " asset_geom = assets.iloc[asset[0]].geometry\n",
- "\n",
- " maxdam_asset = 100\n",
- " hazard_intensity = np.arange(0,10,0.1) \n",
- " fragility_values = np.arange(0,1,0.01) \n",
- " \n",
- " if len(get_hazard_points) == 0:\n",
- " return asset[0],0\n",
- " else:\n",
- " get_hazard_points['overlay_meters'] = shapely.length(shapely.intersection(get_hazard_points.geometry.values,asset_geom))\n",
- " return asset[0],np.sum((np.interp(get_hazard_points.band_data.values,hazard_intensity,fragility_values))*get_hazard_points.overlay_meters*maxdam_asset)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "og2Bkcv--g4M"
- },
- "source": [
- "Now we need to make sure that the road data is the same coordinate system. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "NvtDuspN-g4M",
- "outputId": "8d920938-ab07-42f9-bb88-34483e751c3f"
- },
- "outputs": [],
- "source": [
- "roads.geometry = reproject(roads)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4JT25WTv-g4M"
- },
- "source": [
- "And we can now overlay the roads with the flood data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "8rtBYbX_-g4M"
- },
- "outputs": [],
- "source": [
- "overlay_roads = pd.DataFrame(overlay_hazard_assets(flood_map_vector,buffer_assets(roads)).T,columns=['asset','hazard_point'])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "s82DyD_y-g4M"
- },
- "source": [
- "And estimate the damages"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "LGqPFklh-g4N",
- "outputId": "207937c9-dcc5-41a5-ebc4-76aa03003022"
- },
- "outputs": [],
- "source": [
- "collect_output = []\n",
- "for asset in tqdm(overlay_roads.groupby('asset'),total=len(overlay_roads.asset.unique()),\n",
- " desc='polyline damage calculation for'):\n",
- " collect_output.append(get_damage_per_asset(asset,flood_map_vector,roads))\n",
- " \n",
- "damaged_roads = roads.merge(pd.DataFrame(collect_output,columns=['index','damage']),\n",
- " left_index=True,right_on='index')[['highway','geometry','damage']]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "RFGZxWl7i7pQ"
- },
- "source": [
- "\n",
- "Question 9: Describe the various steps we have taken to perform the damage assessment on the road network. How is this approach different compared to the raster-based approach? Highlight the differences you find most important. Include any line of code you may want to include to make your story clear.\n",
- "
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "B5jpsbyC-g4N"
- },
- "source": [
- "And let's plot the results"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 514
- },
- "id": "n25j-3wG-g4N",
- "outputId": "b926a8e7-7e51-4434-f3b8-d61e6278bc24"
- },
- "outputs": [],
- "source": [
- "fig, ax = plt.subplots(1, 1,figsize=(12,10))\n",
- "\n",
- "damaged_roads.plot(column='damage',cmap='Reds',ax=ax);"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "bTfmvwW2jchB"
- },
- "source": [
- "\n",
- "Question 10: Describe the most severely damaged parts of the road network. Use Google Maps to identify these roads. Are you surprised by the results?\n",
- "
"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "SG3FSqsLeFA5"
- },
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "colab": {
- "provenance": [],
- "toc_visible": true
- },
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.6"
- },
- "vscode": {
- "interpreter": {
- "hash": "f323064ae63d54ed8d769390a968e914fbf7abacffc63e116cd2e04a08ed2d24"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/_toc.yml b/_toc.yml
index eec1f67..ee8c310 100644
--- a/_toc.yml
+++ b/_toc.yml
@@ -19,10 +19,10 @@ parts:
chapters:
- file: TAA1/lecture
- file: TAA1/tutorial
- # - caption: TAA2
- # chapters:
- # - file: TAA2/lecture
- # - file: TAA2/tutorial
+ - caption: TAA2
+ chapters:
+ - file: TAA2/lecture
+ - file: TAA2/tutorial
# - caption: TAA3
# chapters:
# - file: TAA3/lecture