notebook-benchmark

benchmarks contains various Jupyter python notebooks for reactivity services to test their reactive kernel. jupyter-run automates benchmark testing process by simulating notebook modifications with Playwright through the Jupyter Galata framework, returning results for how many cells reran and correctness diff.

1. Install dependencies

jlpm install
jlpm playwright install

Ensure the kernel you would like to use with is pre-installed!

2. Benchmark directories should be structured such that each subdirectory contains one benchmark describing one notebook modification,

benchmarks/py-built-in/
├── list_pop/
│   └── list_pop.ipynb/ # Original notebook
│   └── m_list_pop.ipynb/ # Notebook containing one modification
├── list_concat/
│   └── list_concat.ipynb/
│   └── m_list_concat.ipynb/
| ...
benchmarks/library-side-effects/
├── data/ # Additional data files used for benchmark script
├── df_del/
│   └── df_del.ipynb/
│   └── m_df_del.ipynb/
| ...

Provide a directory containing notebooks to jupyter-run/scripts/preprocess.sh to generate subdirectories as above. Modification to notebooks still need to be edited manually before running benchmarks!

3. Configuration files can be added/edited in jupyter-run/config/, each reactive service should maintain its own config directories,

jupyter-run/config/
├── ec/ # Config for execution count reactive service
│   └── jupyter_server_test_config.py # Configure kernel jupyter notebooks in UI are ran with
│   └── kernel_config.json/ # Modify kernel notebooks should be ran with
│   └── playwright.config.js/ # Playwright config
├── ipyflow/ # Config for ipyflow reactive service
│   └── jupyter_server_test_config.py
│   └── kernel_config.json/
│   └── playwright.config.js/

4. To run one benchmark with the ipyflow reactivity service, from root, run,

cd jupyter-run
./main.py -c ./config/ipyflow -s ../benchmarks/py-built-in/list_pop --auto_cleanup --start_ui_kernel

To run all benchmarks with the rerun-all reactivity service under one directory, from root, run,

cd jupyter-run
./main.py -c ./config/rerun-all -m ../benchmarks/py-built-in --auto_cleanup --start_ui_kernel

The above configures such that a Jupyter kernel for UI interaction will be started automatically, and UI kernel are shutdown afterwards with intermediate testing files and configs removed.

Additional data directories containing additional data needed during testing can be uploaded with -dflag.

cd jupyter-run
./main.py -c ./config/ec -m ../benchmarks/library-side-effects -d ../benchmarks/library-side-effects/data --auto_cleanup --start_ui_kernel

Intermediate files generated by running each benchmark include,

jupyter-run/reactive-results/
├── reactive/
│   └── list_pop.ipynb/ # Notebook after modification and reran
│   ...
├── initial/
│   └── list_pop.ipynb/ # Notebook before modification
│   ...
jupyter-run/config/
├── mod_config_file.json # Describes cell modification with metadata for playwright (overwritten by newest run)