Update dev/main

.github/workflows/docs.yaml

@@ -16,7 +16,17 @@ jobs:
       with:
         fetch-depth: 0 # otherwise, you will failed to push refs to dest repo
     - name: install dacapo
-      run: pip install .[docs]
+      # run: 
+      run: |
+        pip install sphinx-autodoc-typehints sphinx-autoapi sphinx-click sphinx-rtd-theme myst-parser jupytext ipykernel nbsphinx myst_nb
+        python -m ipykernel install --user --name python3
+        pip install .[docs]
+
+    - name: parse notebooks
+      run: |
+        jupytext --to notebook --execute ./examples/starter_tutorial/minimal_tutorial.py --output ./docs/source/notebooks/minimal_tutorial.ipynb
+    - name: remove notebook scripts
+      run: rm ./docs/source/notebooks/*.py
     - name: Build and Commit
       uses: sphinx-notes/pages@v2
       with:

.gitignore

@@ -29,4 +29,7 @@ tmp/
 # daisy logs
 daisy_logs/
 
-*.csv
+*.csv
+*.private
+
+user_experiments/*

CITATION.cff

@@ -0,0 +1,62 @@
+
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+authors:
+- family-names: "Patton"
+  given-names: "William"
+  orcid: "https://orcid.org/0000-0002-9652-3222"
+- family-names: "Rhoades"
+  given-names: "Jeff L."
+  orcid: "https://orcid.org/0000-0001-5077-2533"
+- family-names: "Zouinkhi"
+  given-names: "Marwan"
+  orcid: "https://orcid.org/0000-0002-9441-2908"
+- family-names: "Funke"
+  given-names: "Jan"
+  orcid: "http://orcid.org/0000-0003-4388-7783"
+title: "DaCapo"
+version: 0.3.0
+doi: 10.48550/arXiv.2408.02834
+date-released: 2024-08-05
+url: "https://github.com/janelia-cellmap/dacapo"
+preferred-citation:
+  type: article
+  authors:
+  - family-names: "Patton"
+    given-names: "William"
+    orcid: "https://orcid.org/0000-0002-9652-3222"
+  - family-names: "Rhoades"
+    given-names: "Jeff L."
+    orcid: "https://orcid.org/0000-0001-5077-2533"
+  - family-names: "Zouinkhi"
+    given-names: "Marwan"
+    orcid: "https://orcid.org/0000-0002-9441-2908"
+  - family-names: "Ackerman"
+    given-names: "David G."
+    orcid: "http://orcid.org/0000-0003-0172-6594"
+  - family-names: "Malin-Mayor"
+    given-names: "Caroline"
+    orcid: "https://orcid.org/0000-0002-9627-6030"
+  - family-names: "Adjavon"
+    given-names: "Diane"
+  - family-names: "Heinrich"
+    given-names: "Larissa"
+    orcid: "http://orcid.org/0000-0003-2852-6664"
+  - family-names: "Bennett"
+    given-names: "Davis"
+    orcid: "http://orcid.org/0000-0001-7579-2848"
+  - family-names: "Zubov"
+    given-names: "Yurii"
+    orcid: "https://orcid.org/0000-0003-1988-8081"
+  - family-names: "Project Team"
+    given-names: "CellMap"
+  - family-names: "Weigel"
+    given-names: "Aubrey V."
+    orcid: "http://orcid.org/0000-0003-1694-4420"
+  - family-names: "Funke"
+    given-names: "Jan"
+    orcid: "http://orcid.org/0000-0003-4388-7783"
+  doi: 10.48550/arXiv.2408.02834
+  journal: "arXiv-cs.CV"
+  title: "DaCapo: a modular deep learning framework for scalable 3D image segmentation"
+  year: 2024

CONTRIBUTING.md

@@ -22,15 +22,15 @@ This will also be run automatically when a PR is made to master and a codecov re
 
 
 ## Branching and PRs
-- Users that have been added to the CellMap organization and the DaCapo project should be able to develop directly into the CellMap fork of DaCapo. Other users will need to create a fork.
-- For a completely new feature, make a branch off of the `dev/main` branch of CellMap's fork of DaCapo with a name describing the feature. If you are collaborating on a feature that already has a branch, you can branch off that feature branch.
-- Currently, you should make your PRs into the `dev/main` branch of CellMap's fork, or the feature branch you branched off of. PRs currently require one maintainer's approval before merging. Once the PR is merged, the feature branch should be deleted. 
-- `dev/main` will be regularly merged to `main` when new features are fully implemented and all tests are passing.
+- Users that have been added to the CellMap organization and the DaCapo project should be able to develop directly into the CellMap fork of DaCapo. Other users will need to create a fork or ask to be added as a collaborator.
+- For a completely new feature, make a branch off of the `main` branch of CellMap's fork of DaCapo with a name describing the feature. If you are collaborating on a feature that already has a branch, you can branch off that feature branch.
+- Currently, you should make your PRs into the `main` branch of CellMap's fork, or the feature branch you branched off of. PRs currently require one maintainer's approval before merging. Once the PR is merged, the feature branch will/should be deleted. 
+- `main` will be regularly published to PyPi when new features are fully implemented and all tests are passing.
 
 
 ## Documentation
 Documentation is built using Sphinx. To build the documentation locally, run
 ```bash
 sphinx-build -M html docs/source docs/build
 ```
-This will generate the html files in the `docs/build/html` directory.
+This will generate the html files in the `docs/build/html` directory.

README.md

@@ -17,15 +17,20 @@
 A framework for easy application of established machine learning techniques on large, multi-dimensional images.
 
 `dacapo` allows you to configure machine learning jobs as combinations of
-[DataSplits](http://docs/api.html#datasplits),
-[Architectures](http://docs/api.html#architectures),
-[Tasks](http://docs/api.html#tasks),
-[Trainers](http://docs/api.html#trainers),
+[DataSplits](https://janelia-cellmap.github.io/dacapo/autoapi/dacapo/experiments/datasplits/index.html),
+[Architectures](https://janelia-cellmap.github.io/dacapo/autoapi/dacapo/experiments/architectures/index.html),
+[Tasks](https://janelia-cellmap.github.io/dacapo/autoapi/dacapo/experiments/tasks/index.html),
+[Trainers](https://janelia-cellmap.github.io/dacapo/autoapi/dacapo/experiments/trainers/index.html),
 on arbitrarily large volumes of
 multi-dimensional images. `dacapo` is not tied to a particular learning
 framework, but currently only supports [`torch`](https://pytorch.org/) with
 plans to support [`tensorflow`](https://www.tensorflow.org/).
 
+
+![DaCapo Diagram](https://raw.githubusercontent.com/janelia-cellmap/dacapo/main/docs/source/_static/dacapo_diagram.png)
+
+
+
 ## Installation and Setup
 Currently, python>=3.10 is supported. We recommend creating a new conda environment for dacapo with python 3.10.
 ```
@@ -52,7 +57,12 @@ Tasks we support and approaches for those tasks:
  - Semantic segmentation
     - Signed distances
     - One-hot encoding of different types of objects
-
+
+## Example Tutorial
+A minimal example tutorial can be found in the examples directory and opened in colab here: <a target="_blank" href="https://colab.research.google.com/github/janelia-cellmap/dacapo/blob/main/examples/starter_tutorial/minimal_tutorial.ipynb">
+  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+</a>
+
 ## Helpful Resources & Tools
  - Chunked data, zarr, and n5
     - OME-Zarr: a cloud-optimized bioimaging file format with international community support (doi: [10.1101/2023.02.17.528834](https://pubmed.ncbi.nlm.nih.gov/36865282/))
@@ -72,4 +82,15 @@ Tasks we support and approaches for those tasks:
     - Example of [groundtruth data](https://tinyurl.com/pu8mespz)
  - Visualization
     - [Neuroglancer GitHub Repo](https://github.com/google/neuroglancer)
-
+
+# Citing this repo
+If you use our code, please cite us and spread the news!
+```
+@article{Patton_DaCapo_a_modular_2024,
+author = {Patton, William and Rhoades, Jeff L. and Zouinkhi, Marwan and Ackerman, David G. and Malin-Mayor, Caroline and Adjavon, Diane and Heinrich, Larissa and Bennett, Davis and Zubov, Yurii and Project Team, CellMap and Weigel, Aubrey V. and Funke, Jan},
+doi = {10.48550/arXiv.2408.02834},
+journal = {arXiv-cs.CV},
+title = {{DaCapo: a modular deep learning framework for scalable 3D image segmentation}},
+year = {2024}
+}
+```

dacapo/__init__.py

@@ -1,10 +1,11 @@
-__version__ = "0.3.0"
+__version__ = "0.3.5"
 __version_info__ = tuple(int(i) for i in __version__.split("."))
 
 from .options import Options  # noqa
 from . import experiments, utils  # noqa
 from .apply import apply  # noqa
 from .train import train  # noqa
-from .validate import validate  # noqa
+from .validate import validate, validate_run  # noqa
 from .predict import predict  # noqa
 from .blockwise import run_blockwise, segment_blockwise  # noqa
+from . import predict_local

dacapo/apply.py

@@ -1,8 +1,8 @@
 import logging
 from typing import Optional
 from funlib.geometry import Roi, Coordinate
+from funlib.persistence import open_ds
 import numpy as np
-from dacapo.experiments.datasplits.datasets.arrays.array import Array
 from dacapo.experiments.datasplits.datasets.dataset import Dataset
 from dacapo.experiments.run import Run
 
@@ -12,7 +12,6 @@
 import dacapo.experiments.tasks.post_processors as post_processors
 from dacapo.store.array_store import LocalArrayIdentifier
 from dacapo.predict import predict
-from dacapo.experiments.datasplits.datasets.arrays import ZarrArray
 from dacapo.store.create_store import (
     create_config_store,
     create_weights_store,
@@ -164,7 +163,9 @@ def apply(
 
     # make array identifiers for input, predictions and outputs
     input_array_identifier = LocalArrayIdentifier(Path(input_container), input_dataset)
-    input_array = ZarrArray.open_from_array_identifier(input_array_identifier)
+    input_array = open_ds(
+        f"{input_array_identifier.container}/{input_array_identifier.dataset}"
+    )
     if roi is None:
         _roi = input_array.roi
     else:
@@ -226,7 +227,7 @@ def apply_run(
         output_dtype (np.dtype | str, optional): The output data type. Defaults to np.uint8.
         overwrite (bool, optional): Whether to overwrite existing output. Defaults to True.
     Raises:
-        ValueError: If the input array is not a ZarrArray.
+        ValueError: If the input array is not a zarr array.
     Examples:
         >>> apply_run(
         ...     run=run,

dacapo/blockwise/argmax_worker.py

@@ -1,8 +1,9 @@
 from upath import UPath as Path
 import sys
-from dacapo.experiments.datasplits.datasets.arrays.zarr_array import ZarrArray
+
 from dacapo.store.array_store import LocalArrayIdentifier
 from dacapo.compute_context import create_compute_context
+from dacapo.tmp import open_from_identifier
 
 import daisy
 
@@ -54,6 +55,22 @@ def start_worker(
     output_container: Path | str,
     output_dataset: str,
     return_io_loop: bool = False,
+):
+    return start_worker_fn(
+        input_container=input_container,
+        input_dataset=input_dataset,
+        output_container=output_container,
+        output_dataset=output_dataset,
+        return_io_loop=return_io_loop,
+    )
+
+
+def start_worker_fn(
+    input_container: Path | str,
+    input_dataset: str,
+    output_container: Path | str,
+    output_dataset: str,
+    return_io_loop: bool = False,
 ):
     """
     Start the threshold worker.
@@ -66,12 +83,12 @@ def start_worker(
     """
     # get arrays
     input_array_identifier = LocalArrayIdentifier(Path(input_container), input_dataset)
-    input_array = ZarrArray.open_from_array_identifier(input_array_identifier)
+    input_array = open_from_identifier(input_array_identifier)
 
     output_array_identifier = LocalArrayIdentifier(
         Path(output_container), output_dataset
     )
-    output_array = ZarrArray.open_from_array_identifier(output_array_identifier)
+    output_array = open_from_identifier(output_array_identifier)
 
     def io_loop():
         # wait for blocks to run pipeline
@@ -86,7 +103,7 @@ def io_loop():
                 # write to output array
                 output_array[block.write_roi] = np.argmax(
                     input_array[block.write_roi],
-                    axis=input_array.axes.index("c"),
+                    axis=input_array.axis_names.index("c^"),
                 )
 
     if return_io_loop:
@@ -111,7 +128,7 @@ def spawn_worker(
     """
     compute_context = create_compute_context()
     if not compute_context.distribute_workers:
-        return start_worker(
+        return start_worker_fn(
             input_array_identifier.container,
             input_array_identifier.dataset,
             output_array_identifier.container,

dacapo/blockwise/empanada_function.py

@@ -50,7 +50,7 @@ def segment_function(input_array, block, **parameters):
 
     Args:
         input_array (np.ndarray): The 3D array to segment.
-        block (dask.array.core.Block): The block object.
+        block (daisy.Block): The block object.
         **parameters: Parameters for the empanada-napari segmenter.
     Returns:
         np.ndarray: The segmented 3D array.
@@ -374,7 +374,7 @@ def start_consensus_worker(trackers_dict):
     assert image.ndim in [3, 4], "Only 3D and 4D input images can be handled!"
     if image.ndim == 4:
         # channel dimensions are commonly 1, 3 and 4
-        # check for dimensions on zeroeth and last axes
+        # check for dimensions on zeroeth and last axis_names
         shape = image.shape
         if shape[0] in [1, 3, 4]:
             image = image[0]