WIP: masking outliers + conflicting distributions + incidence angles + high uncertainty classification + majority filter #6

MartinSchobben · 2023-12-18T08:05:33Z

There is no mask method in the local instance, so for now I just use multiplication to do the same thing.

review-notebook-app · 2023-12-18T08:05:38Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

review-notebook-app · 2023-12-20T06:48:53Z

View / edit / reply to this conversation on ReviewNB

SwamyDev commented on 2023-12-20T06:48:52Z
----------------------------------------------------------------

I was shortly confused by the "multi-band datacube" term thinking "where is VH or L-Band here?", but then I realised multi-band in the openEO sense. However, I'd still better write "The merged preprocessed datacube..." since these aren't really bands, even if openEO pretends.

review-notebook-app · 2023-12-20T06:48:53Z

View / edit / reply to this conversation on ReviewNB

SwamyDev commented on 2023-12-20T06:48:53Z
----------------------------------------------------------------

You could consider putting this nice function into a python module.

MartinSchobben · 2023-12-20T07:22:09Z

I can't see these comments now, as I blocked this app. Somehow it keeps showing up in this repo. Can you comment on the qmd in GitHub?

SwamyDev · 2023-12-20T08:17:53Z

Lol, works great then this app ^^, yeah I'll do a normal review then

SwamyDev

Well done, works really nicely also the results are very well presented!

notebooks/1_yeoda_dc.qmd

SwamyDev · 2023-12-20T08:17:19Z

notebooks/1_yeoda_dc.qmd

+def view_flood_map(df):
+    # selecting a subsection of the data and reprojecting
+    flood_map = df[0, 13070:14000, 12900:14200]
+    flood_map = flood_map.rio.reproject(f"EPSG:4326", nodata=np.nan)
+    # add open streetmap
+    request = cimgt.OSM()
+    # initialize figure
+    fig = plt.figure(figsize=(13,9))
+    axis = plt.axes(projection=ccrs.PlateCarree(), frameon=True)
+    axis.add_image(request, 15)
+    # add the data
+    flood_map = flood_map.plot(
+        ax=axis,
+        transform=ccrs.PlateCarree(),
+        levels=[0, 1, 2],
+        colors=["#00000000", "#ff0000"],
+        add_colorbar=False
+    )
+    # legend and title
+    cbar = fig.colorbar(flood_map, ax=axis, location="bottom", shrink=0.6)
+    cbar.ax.get_xaxis().set_ticks([])
+    for j, lab in enumerate(['non-flood','flood']):
+        cbar.ax.text((2 * j + 1) / 2.0, 0.5, lab, ha='center', va='center')
+    cbar.ax.get_xaxis().labelpad = 10
+    tk = fig.gca()
+    tk = tk.set_title("Flood map")


You could consider putting this nice function into a python module.

I took the lazy approach for now whereby I do not show this code chunk in the final webpage: https://martinschobben.github.io/openeo-flood-mapper-local/. But I'll consider this.

MartinSchobben · 2023-12-20T08:38:14Z

There is no mask method in the local instance, so for now I just use multiplication to do the same thing.

I will now then also consider implementing mask in the client side processing module of openEO. Although it feels a bit redundant as the ultimate goal would be to use the workflow with the EODC backend. I guess a more complete openeo.local would help development of remote openeo workflows.

MartinSchobben · 2023-12-20T13:06:42Z

Adding mask as a process for this workflow was actually not that hard. You can see the results here: https://martinschobben.github.io/openeo-flood-mapper-local/. Although the output of execute() is now quite verbose.

clausmichele · 2023-12-20T13:15:42Z

@MartinSchobben I did implement mask already, there's an open PR here Open-EO/openeo-processes-dask#165

It would be nice if you could review it! We need anyway someone at EODC to review and approve it, since the repo is still officially maintained by them.

clausmichele · 2023-12-20T13:36:07Z

@MartinSchobben the verboseness of the logs when you call .execute() depends on your logging level. If you set it to INFO, you won't see any logging message at all. (I just run the notebook and didn't get any log message).
Please also make sure to have the latest version of openeo-processes-dask and openeo-pg-parser-networkx.

notebooks/1_yeoda_dc.qmd

MartinSchobben · 2023-12-20T13:47:42Z

@MartinSchobben I did implement mask already, there's an open PR here Open-EO/openeo-processes-dask#165

It would be nice if you could review it! We need anyway someone at EODC to review and approve it, since the repo is still officially maintained by them.

@clausmichele Ah great! Even better. I'll have a look then.

MartinSchobben · 2023-12-20T14:40:31Z

@clausmichele, but to clarify we are not EODC. We are TUWien.

clausmichele · 2023-12-20T14:49:47Z

@clausmichele, but to clarify we are not EODC. We are TUWien.

I know!

MartinSchobben · 2023-12-20T16:21:59Z

I am now getting closer to what the openeo workflow should look like with mask from the PR. I do note, however, that processing time has substantially increased with mask included.

clausmichele · 2023-12-21T08:02:17Z

A bit of overhead is understandable, since the mask process needs to take care about multiple possibilities, depending on the input datacube. However, it would be interesting to understand what's the step taking the most time! I will also try and check.
By the way, nothing prevents to use a normal multiplication when in full control of the input datacubes!

MartinSchobben · 2023-12-21T08:31:58Z

Hmm, interesting. I was using multiplication at first. But, as the idea of this repo was to showcase transforming an existing pipeline (Copernicus GFM) into openEO syntax, I found it useful to stick to the given processes.

I'll also have a look what causes this increased processing time.

clausmichele · 2023-12-21T08:36:32Z

But a simple multiplication is also represented as a basic openEO process, so there's nothing wrong with it. But as I was saying, it could fail if the input datacubes are not matching. Anyway, I'm doing some tests and the time consuming part seems to be calling the .where method. I'm investigating if it makes sense to keep it or replace it with a multiplication, since the whole code before is checking if the inputs are aligned.

clausmichele · 2023-12-21T09:03:29Z

The main issue is handling no data values and the replacement value, that's why we rely on .where.
If you can find a faster approach to reproduce the result we get with where it would be cool 🥇
Here an example of what the openEO mask process is doing:

>>> import xarray as xr
>>> import numpy as np

>>> data = np.arange(25).reshape(5, 5).astype(np.float32)
>>> data[0,0] = np.nan
>>> data[0,1] = 0
>>> input_data = xr.DataArray(data, dims=("x", "y"))
>>> print(input_data)
<xarray.DataArray (x: 5, y: 5)>
array([[nan,  0.,  2.,  3.,  4.],
       [ 5.,  6.,  7.,  8.,  9.],
       [10., 11., 12., 13., 14.],
       [15., 16., 17., 18., 19.],
       [20., 21., 22., 23., 24.]], dtype=float32)
Dimensions without coordinates: x, y
>>> mask = input_data > 4
>>> print(mask)
<xarray.DataArray (x: 5, y: 5)>
array([[False, False, False, False, False],
       [ True,  True,  True,  True,  True],
       [ True,  True,  True,  True,  True],
       [ True,  True,  True,  True,  True],
       [ True,  True,  True,  True,  True]])
Dimensions without coordinates: x, y
>>> replacement = np.nan
>>> masked_data = input_data.where(~mask,replacement)
>>> print(masked_data)
<xarray.DataArray (x: 5, y: 5)>
array([[nan,  0.,  2.,  3.,  4.],
       [nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan],
       [nan, nan, nan, nan, nan]], dtype=float32)
Dimensions without coordinates: x, y

MartinSchobben · 2023-12-21T12:07:26Z

A further question @clausmichele, regarding the above comment. I would actually expect the cube for openeo-processes-dask to be:

import xarray as xr
import numpy as np
import dask.array as da
data = da.arange(size).reshape(x, y).astype(np.float32)
data[0,0] = np.nan
data[0,1] = 0
input_data = xr.DataArray(data, dims=("x", "y"))

~~But it appears to be a normal xarray. Is this correct? And could parallel computing of .where not make a difference as well?~~

I think I answered it myself, openeo local processing loads data chunked as far as I can tell.

clausmichele · 2023-12-21T12:57:42Z

A further question @clausmichele, regarding the above comment. I would actually expect the cube for openeo-processes-dask to be:
import xarray as xr
import numpy as np
import dask.array as da
data = da.arange(size).reshape(x, y).astype(np.float32)
data[0,0] = np.nan
data[0,1] = 0
input_data = xr.DataArray(data, dims=("x", "y"))
But it appears to be a normal xarray. Is this correct? And could parallel computing of .where not make a difference as well?

Well, the .where call would work in both cases, so with an xarray object based on numpy or dask arrays. With Dask it can/will be parallelized, but it can't be faster than a single multiplication.
(The above code was just an example to understand how the mask process should work with some sample numers)

MartinSchobben · 2024-01-04T13:30:35Z

I reverted to masking by multiplication instead of using the process mask to speed up processing. I also added apply_neighborhood to mimic the majority filter for speckle removal. See here, https://github.com/MartinSchobben/openeo-processes-dask/tree/add-apply-neighborhood, the initial implementation of apply_neighborhood. At the moment I do not understand the complete definition of this process by openEO, especially the overlap parameter is unclear to me.

MartinSchobben · 2024-01-09T11:47:17Z

@clausmichele FYI, I put this here: Open-EO/openeo-processes-dask#215. We have trouble understanding the definition of apply_neighborhood. Perhaps you find this interesting.

MartinSchobben requested review from SwamyDev and fl0roth December 18, 2023 08:06

MartinSchobben changed the title ~~WIP: preprocessing outliers + conflicting distributions + incidence angles~~ WIP: preprocessing outliers + conflicting distributions + incidence angles + high uncertainty classification Dec 18, 2023

masking outliers + conflicting distributions + incidence angles

5d175e3

MartinSchobben force-pushed the dev branch from eacbbaf to 5d175e3 Compare December 18, 2023 12:17

MartinSchobben changed the title ~~WIP: preprocessing outliers + conflicting distributions + incidence angles + high uncertainty classification~~ WIP: masking outliers + conflicting distributions + incidence angles + high uncertainty classification Dec 18, 2023

SwamyDev approved these changes Dec 20, 2023

View reviewed changes

MartinSchobben requested a review from clausmichele December 20, 2023 13:12

clausmichele reviewed Dec 20, 2023

View reviewed changes

notebooks/1_yeoda_dc.qmd Outdated Show resolved Hide resolved

add mask method

131b7b9

add apply_neighborhood

5a8a7f5

MartinSchobben changed the title ~~WIP: masking outliers + conflicting distributions + incidence angles + high uncertainty classification~~ WIP: masking outliers + conflicting distributions + incidence angles + high uncertainty classification + majority filter Jan 4, 2024

typo

bb942de

update apply_neigborhood

6376465

MartinSchobben merged commit 165949e into interTwin-eu:main Jan 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: masking outliers + conflicting distributions + incidence angles + high uncertainty classification + majority filter #6

WIP: masking outliers + conflicting distributions + incidence angles + high uncertainty classification + majority filter #6

MartinSchobben commented Dec 18, 2023 •

edited

Loading

review-notebook-app bot commented Dec 18, 2023

review-notebook-app bot commented Dec 20, 2023 •

edited

Loading

review-notebook-app bot commented Dec 20, 2023 •

edited

Loading

MartinSchobben commented Dec 20, 2023 •

edited

Loading

SwamyDev commented Dec 20, 2023

SwamyDev left a comment

SwamyDev Dec 20, 2023

MartinSchobben Dec 20, 2023 •

edited

Loading

MartinSchobben commented Dec 20, 2023

MartinSchobben commented Dec 20, 2023

clausmichele commented Dec 20, 2023 •

edited

Loading

clausmichele commented Dec 20, 2023

MartinSchobben commented Dec 20, 2023

MartinSchobben commented Dec 20, 2023

clausmichele commented Dec 20, 2023

MartinSchobben commented Dec 20, 2023

clausmichele commented Dec 21, 2023

MartinSchobben commented Dec 21, 2023

clausmichele commented Dec 21, 2023

clausmichele commented Dec 21, 2023 •

edited

Loading

MartinSchobben commented Dec 21, 2023 •

edited

Loading

clausmichele commented Dec 21, 2023 •

edited

Loading

MartinSchobben commented Jan 4, 2024 •

edited

Loading

MartinSchobben commented Jan 9, 2024

WIP: masking outliers + conflicting distributions + incidence angles + high uncertainty classification + majority filter #6

WIP: masking outliers + conflicting distributions + incidence angles + high uncertainty classification + majority filter #6

Conversation

MartinSchobben commented Dec 18, 2023 • edited Loading

review-notebook-app bot commented Dec 18, 2023

review-notebook-app bot commented Dec 20, 2023 • edited Loading

review-notebook-app bot commented Dec 20, 2023 • edited Loading

MartinSchobben commented Dec 20, 2023 • edited Loading

SwamyDev commented Dec 20, 2023

SwamyDev left a comment

Choose a reason for hiding this comment

SwamyDev Dec 20, 2023

Choose a reason for hiding this comment

MartinSchobben Dec 20, 2023 • edited Loading

Choose a reason for hiding this comment

MartinSchobben commented Dec 20, 2023

MartinSchobben commented Dec 20, 2023

clausmichele commented Dec 20, 2023 • edited Loading

clausmichele commented Dec 20, 2023

MartinSchobben commented Dec 20, 2023

MartinSchobben commented Dec 20, 2023

clausmichele commented Dec 20, 2023

MartinSchobben commented Dec 20, 2023

clausmichele commented Dec 21, 2023

MartinSchobben commented Dec 21, 2023

clausmichele commented Dec 21, 2023

clausmichele commented Dec 21, 2023 • edited Loading

MartinSchobben commented Dec 21, 2023 • edited Loading

clausmichele commented Dec 21, 2023 • edited Loading

MartinSchobben commented Jan 4, 2024 • edited Loading

MartinSchobben commented Jan 9, 2024

MartinSchobben commented Dec 18, 2023 •

edited

Loading

review-notebook-app bot commented Dec 20, 2023 •

edited

Loading

review-notebook-app bot commented Dec 20, 2023 •

edited

Loading

MartinSchobben commented Dec 20, 2023 •

edited

Loading

MartinSchobben Dec 20, 2023 •

edited

Loading

clausmichele commented Dec 20, 2023 •

edited

Loading

clausmichele commented Dec 21, 2023 •

edited

Loading

MartinSchobben commented Dec 21, 2023 •

edited

Loading

clausmichele commented Dec 21, 2023 •

edited

Loading

MartinSchobben commented Jan 4, 2024 •

edited

Loading