Demo Notebooks

[jejjohnson]

Add demo notebooks to demonstrate different aspects of how Helio-Tools can be used and its potential interface to machine learning.

---

Preprocessing

SDO Pipeline

• Add demo preprocessing steps
• Add demo config for preprocessing
• Add demo config for preprocessing + dataset + dataloader
• Add demo domain-expert validation for dataset
• Add analysis functions for plots (see ITI paper)
• [ ]

---

ML-Ready Data

Demo showcasing how we can create ML-Ready Data using scripts and parallelization. We will create 3 dataset types for each tutorial 1) single channel datasets, 2) multi-channel datasets, and 3) multi-channel time-series datasets.

• Numpy Dataset
• Image Dataset
• xarray.Dataset

---

Machine Learning DataModules

Demo Training Datasets + DataLoaders

These demos will showcase how we can create datasets and dataloaders for different datastructures. We will focus on PyTorch datasets for three data structures: 1) numpy.ndarray, 2) .png/.jpeg/… image files, and 3) xarray.Datasets. Well will discuss things like global dataset normalization and (random) patching.

• Demo DataLoader with numpy (see example notebook)
• Demo DataLoader with xarray (see example)
• Demo DataLoader with RasterVision (see example)

These demos will showcase how we can create datasets and dataloaders for inference (making predictions for new datasets). . We will focus on PyTorch datasets for three data structures: 1) numpy.ndarray, 2) .png/.jpeg/… image files, and 3) xarray.Datasets. We will discuss additional transformations that are needed for the datasets like training dataset normalizing and sliding window patching.

• Inference on a demo dataset (see example)
• Inference on a real dataset with ITI model weights (see translate.py)