Challenge 24 - CliMetLab web application

[jwagemann]

Challenge 24 - CliMetLab web application

Stream 2 - Web development for weather, climate and atmosphere

Goal

Develop a web app for CliMetLab

Mentors and skills

• Mentors: Florian Pinault, Baudouin Raoult
• Skills required:
  • Front-end web development
  • ReactJS
  • JavaScript/CSS/HTML
  • Some knowledge of Python

---

Challenge description

Why do we need a solution
CliMetLab is a Python package aiming at simplifying access to climate and meteorological datasets, allowing users to focus on science instead of technical issues such as data access and data formats. It is mostly intended to be used in Jupyter notebooks, and be interoperable with all popular data analytic packages, such as NumPy, Pandas, Xarray, SciPy, Matplotlib, etc. and well as machine learning frameworks, such as TensorFlow, Keras or PyTorch.

In this challenge, we would like to see a user-friendly dashboard interface to some of the CliMetLab functionalities using a lightweight GUI web app.

What could be the solution
The relevant functionalities are:

• Cache inspection (filtering, disk usage, preview) and management (batch deletion)
• Easy to maintain
• Settings management

As a second step, additional functionalities may include:

• Plugins management
• Debugging
• Credential management
• Cache synchronisation

Framework and dependencies:

• The backend should be python-based such as web.py (recommended but others could be considered), delivering content through a REST API. The backend server may use some templating (such as jinja2) but it is recommended to put most of the logic in the javascript front-end.
• The front-end Javascript app should use a framework (React recommended, but a JQuery-based such as Angular, ReactJS, etc. could be considered). During development, it is expected to use npm (perhaps bower package manager) and serve the static javascript/css/html. Using typescript is possible.
• The appropriate CSS should be included, bootstrap is recommended (and SCSS).

The whole code of this web app must be packaged as a python package plugin for the climetlab CLI. To reiterate and clarify the javascript dependencies: for the end-user, the javascript part needs to be packaged as static content and shipped inside a unique pip package. There should be no dependency on the javascript stack: the end-user will only install a pip package and the end-user will not install npm/nodejs/etc. Installing the python package allows to run “climetlab web” and have a local server on http://localhost:some_port (similar to a local jupyter server).

Emphasis should be put on clean code, reusability, modularity, ease of maintainability. If you master Reactjs and wand a small applied use case from real life, this is a perfect opportunity for you to demonstrate the capabilities of the framework and yours.

References

A draft is available here.

[theairbend3r]

Hello mentors! 😄

I’m interested in this project. I have a few questions about it.

In this challenge, we would like to see a user-friendly dashboard interface to some of the CliMetLab functionalities using a lightweight GUI web app.

• The aim of this project would be to expose configuration-like settings of the climetlab package via a web app?
• What is the motivation behind building this?

Cache inspection (filtering, disk usage, preview) and management (batch deletion)

• Cache inspection refers to data downloaded and cached by cml.load_source()?

• Filtering: does this mean filtering the cached data according to data types like GRIB, NetCDF, etc.?

• Disk Usage: this is how much space do the GRIB/NetCDF/etc data types occupy? Will/should this scan the entire computer or only directories set by the user?

• Preview: this means visualizing GRIB/NetCDF/etc data as SVG in the web app?

• Management: batch deletion of the cached GRIB/NetCDF/etc data?

it is easy to implement

• I didn’t understand this. Does this mean easy to implement analysis/pre-processing + visualization? For example, a user can perform simple analysis and visualize it via the web-app which, under the hood, is performed by numpy/pandas/matplotlib? This could mean simple filtering+plotting? Or changing plotting + changing themes? etc.

Settings management

• Does this refers to settings, like cache directory etc., mentioned here: https://climetlab.readthedocs.io/en/latest/guide/settings.html#default-values?

As a second step, additional functionalities may include:

• Debugging: What kind of debugging are we talking about here?
• Credential management: I couldn’t find stuff related to this in the climetlab docs. Is this something like Dropbox but for climate data that interfaces with climetlab?

Cache synchronisation:

• Is this related to this: https://climetlab.readthedocs.io/en/latest/guide/howtos.html#how-to-share-my-cache-directory-with-another-user?

[floriankrb]

Happy to see you are interested in this challenge, here are a few answers:

The aim of this project would be to expose configuration-like settings of the climetlab package via a web app?
What is the motivation behind building this?

Yes. Using the shell command can be cumbersome, therefore the seek for a graphical UI.

Cache inspection refers to data downloaded and cached by cml.load_source()?

Yes. (btw, there are also a few additional files that are cached, and sometimes the files are modified when they are cached (zip files are unzipped for instance)).

Filtering: does this mean filtering the cached data according to data types like GRIB, NetCDF, etc.?

Filtering when managing the cache can be done with the command line with --newer --match etc. This filtering can be done more easily in a web form.

Disk Usage: this is how much space do the GRIB/NetCDF/etc data types occupy? Will/should this scan the entire computer or only directories set by the user?

There will be no need to scan directories, the js GUI relies on the python API which relies on a cache db. The update on the db is already implemented. This could show how much space is used by the cache files matching a given filter. (yes, the filter could be on the format of the files).
Additionally, the GUI can also show the information from "climetlab df".

Preview: this means visualizing GRIB/NetCDF/etc data as SVG in the web app?

Yes. Most likely using climetlab.plot_map()

Management: batch deletion of the cached GRIB/NetCDF/etc data?

Yes.

it is easy to implement
I didn’t understand this...

It is meant to say "easy to maintain".

Settings management
Does this refers to settings, like cache directory etc., mentioned here: https://climetlab.readthedocs.io/en/latest/guide/settings.html#default-values?

Yes.

Debugging: What kind of debugging are we talking about here?

Just helping people to see what is in their cache to help them debugging (with other tools that will not be developed in this project).

Credential management: I couldn’t find stuff related to this in the climetlab docs. Is this something like Dropbox but for climate data that interfaces with climetlab?

The credentials used in climetlab often consists in some files in the $HOME directory where authentication keys are store. We may want to add these in the GUI. Or maybe not, for security reason.

Cache synchronisation:
Is this related to this: https://climetlab.readthedocs.io/en/latest/guide/howtos.html#how-to-share-my-cache-directory-with-another-user?

Yes.

[theairbend3r]

Thank you @floriankrb. Your comments are helpful! :)

For the tech-stack, can we choose libraries other than those mentioned above?

Specifically, I'd like to use FastAPI instead of web.py because it's fast, has great documentation, big community, and is well maintained.

I'd also like to use Tailwind CSS + Styled Components instead of Bootstrap in our React app. This gives us more control over the styling and is easy to understand + extend + modify compared to Bootstrap. Tailwind CSS is also very popular and has a huge community behind it.

What do you think?

[floriankrb]

Yes, why not using FastAPI, it seems promising too. Notice that a strong requirement is that the end-users will not have to install many dependencies, especially they must not have to install the JavaScript stack (nodesjs, npm, etc.).
Same for Tailwind and Styled components: yes, feel free to use it, as long as it is easy to understand and to maintain.
If you choose other packages, there may be more work to package everything, including updating the github/workflow tests, but if you are able to do this, go for it!

[theairbend3r]

Thank you @floriankrb! 😄