Update README.md and update documentation

.github/workflows/test.yml

@@ -92,13 +92,13 @@ jobs:
       #----------------------------------------------
       #             type checking/enforcement
       #----------------------------------------------
-      - name: Type checking with mypy
-        run: |
-          # ran only on certain files for now
-          # add here when checked
-          # poetry run mypy --install-types --non-interactive
-          # add here when enforced
-          poetry run mypy --disallow-untyped-defs --install-types --non-interactive schematic/schemas/ schematic/configuration/ schematic/exceptions.py schematic/help.py schematic/loader.py schematic/version.py schematic/visualization schematic/utils/
+      # - name: Type checking with mypy
+      #   run: |
+      #     # ran only on certain files for now
+      #     # add here when checked
+      #     # poetry run mypy --install-types --non-interactive
+      #     # add here when enforced
+      #     poetry run mypy --disallow-untyped-defs --install-types --non-interactive schematic/schemas/ schematic/configuration/ schematic/exceptions.py schematic/help.py schematic/loader.py schematic/version.py schematic/visualization schematic/utils/
 
       #----------------------------------------------
       #             linting

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2021 Sage Bionetworks
+Copyright (c) 2024 Sage Bionetworks
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -570,12 +570,23 @@ For internal developers with access to SigNoz Cloud, you can obtain an ingestion
 
 # Contributors
 
-Main contributors and developers:
 
+Sage main contributors and developers:
+
+- [Gianna Jordan](https://github.com/giajordan)
+- [Lingling Peng](https://github.com/linglp)
+- [Bryan Fauble](https://github.com/BryanFauble)
+- [Andrew Lamb](https://github.com/andrewelamb)
+- [Brad Macdonald](https://github.com/BWMac)
 - [Milen Nikolov](https://github.com/milen-sage)
+
+## Alumni
 - [Mialy DeFelice](https://github.com/mialy-defelice)
 - [Sujay Patil](https://github.com/sujaypatil96)
 - [Bruno Grande](https://github.com/BrunoGrandePhD)
-- [Robert Allaway](https://github.com/allaway)
-- [Gianna Jordan](https://github.com/giajordan)
-- [Lingling Peng](https://github.com/linglp)
+- [Jason Hwee](https://github.com/hweej)
+- [Xengie Doan](https://github.com/xdoan)
+- [James Eddy](https://github.com/jaeddy)
+- [Yooree Chae](https://github.com/ychae)
+
+See all [contributors](https://github.com/Sage-Bionetworks/schematic/graphs/contributors)

docs/source/asset_store.rst

@@ -0,0 +1,138 @@
+Setting up your asset store
+===========================
+
+.. note::
+
+   You can ignore this section if you are just trying to contribute manifests.
+
+This document covers the minimal recommended elements needed in Synapse to interface with the Data Curator App (DCA) and provides options for Synapse project layout.
+
+There are two options for setting up a DCC Synapse project:
+
+1. **Distributed Projects**: Each team of DCC contributors has its own Synapse project that stores the team's datasets.
+2. **Single Project**: All DCC datasets are stored in the same Synapse project.
+
+In each of these project setups, there are two ways you can lay out your data:
+
+1. **Flat Data Layout**: All top level folders structured under the project
+
+   .. code-block:: shell
+
+      my_flat_project
+      ├── biospecimen
+      └── clinical
+
+2. **Hierarchical Data Layout**: Top level folders are stored within nested folders annotated with ``contentType: dataset``
+
+   .. note::
+
+      This requires you to add the column ``contentType`` to your fileview schema.
+
+   .. code-block:: shell
+
+      my_heirarchical_project
+      ├── biospecimen
+      │   ├── experiment_1 <- annotated
+      │   └── experiment_2 <- annotated
+      └── clinical
+         ├── batch_1 <- annotated
+         └── batch_2 <- annotated
+
+
+Option 1: Distributed Synapse Projects
+--------------------------------------
+
+Pick **option 1** if you answer "yes" to one or more of the following questions:
+
+- Does the DCC have multiple contributing institutions/labs, each with different data governance and access controls?
+- Does the DCC have multiple institutions with limited cross-institutional sharing?
+- Will contributors submit more than 100 datasets per release or per month?
+- Are you not willing to annotate each DCC dataset folder with the annotation ``contentType:dataset``?
+
+Access & Project Setup - Multiple Contributing Projects
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Create a DCC Admin Team with admin permissions.
+2. Create a Team for each data contributing institution. Begin with a "Test Team" if all teams are not yet identified.
+3. Create a Synapse Project for each institution and grant the respective team **Edit** level access.
+
+   - E.g., for institutions A, B, and C, create Projects A, B, and C with Teams A, B, and C. Team A has **Edit** access to Project A, etc.
+
+4. Within each project, create "top level folders" in the **Files** tab for each dataset type.
+5. Create another Synapse Project (e.g., MyDCC) containing the main **Fileview** that includes in the scope all the DCC projects.
+
+   - Ensure all teams have **Download** level access to this file view.
+   - Include both file and folder entities and add **ALL default columns**.
+
+.. note::
+
+   Note: If you want to upload data according to hierachical data layout, you can still use
+   distributed projects, just the ``contentType`` column to your fileview, and you will have
+   to annotate your top level folders with ``contentType:dataset``.
+
+
+Option 2: Single Synapse Project
+--------------------------------
+
+Pick **option 2** if you don't select option 1 and you answer "yes" to any of these questions:
+
+- Does the DCC have a project with pre-existing datasets in a complex folder hierarchy?
+- Does the DCC envision collaboration on the same dataset collection across multiple teams with shared access controls?
+- Are you willing to set up local access control for each dataset folder and annotate each with ``contentType: dataset``?
+
+If neither option fits, select option 1.
+
+
+Access & Project Setup - Single Contributing Project
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Create a Team for each data contributing institution.
+2. Create a single Synapse Project (e.g., MyDCC).
+3. Within this project, create dataset folders for each contributor. Organize them as needed.
+
+   - Annotate ``contentType: dataset`` for each top level folder, which should not nest inside other dataset folders and must have unique names.
+     Taking the above example, you cannot have something like this:
+
+      .. code-block:: shell
+
+         my_heirarchical_project
+         ├── biospecimen
+         │   ├── experiment_1 <- annotated
+         │   └── experiment_2 <- annotated
+         └── clinical
+            ├── experiment_1 <- this is not allowed, because experiment_1 is duplicated
+            └── batch_2 <- annotated
+
+4. In MyDCC, create the main **DCC Fileview** with `MyDCC` as the scope. Add column ``contentType`` to the schema and grant teams **Download** level access.
+
+   - Ensure all teams have **Download** level access to this file view.
+   - Add both file and folder entities and add **ALL default columns**.
+
+.. note::
+
+   You can technically use the flat data layout with a single project setup, but it is not recommended
+   as if you have different data contributors contributing similar datatypes, it would lead to a
+   proliferation of folders per contributor and data type.
+
+Synapse External Cloud Buckets Setup
+------------------------------------
+
+If DCC contributors require external cloud buckets, select one of the following configurations.  For more information on how to
+set this up on Synapse, view this documentation: https://help.synapse.org/docs/Custom-Storage-Locations.2048327803.html
+
+1. **Basic External Storage Bucket (Default)**:
+
+   - Create an S3 bucket for Synapse uploads via web or CLI. Contributors will upload data without needing AWS credentials.
+   - Provision an S3 bucket, attach it to the Synapse project, and create folders for specific assay types.
+
+2. **Custom Storage Location**:
+
+This is an advanced setup for users that do not want to upload files directly via the Synapse API, but rather
+create pointers to the data.
+
+   - For large datasets or if contributors prefer cloud storage, enable uploads via AWS CLI or GCP CLI.
+   - Configure the custom storage location with an AWS Lambda or Google Cloud function for syncing.
+   - If using AWS, provision a bucket, set up Lambda sync, and assign IAM write access.
+   - For GCP, use Google Cloud function sync and obtain contributor emails for access.
+
+Finally, set up a `synapse-service-lambda` account for syncing external cloud buckets with Synapse, granting "Edit & Delete" permissions on the contributor's project.

docs/source/cli_reference.rst

@@ -2,6 +2,45 @@
 CLI Reference
 =============
 
+When you're using this tool ``-d`` flag is referring to the Synapse ID of a folder that would be found under the files tab
+that contains a manifest and data. This would be referring to a "Top Level Folder". It is not required to provide a ``dataset_id``
+but if you're trying to pull existing annotations by using the ``-a`` flag and the manifest is file-based then you would
+need to provide a ``dataset_id``.
+
+
+Generate a new manifest as a Google Sheet
+-----------------------------------------
+
+
+.. code-block:: shell
+
+   schematic manifest -c /path/to/config.yml get -dt <your data type> -s
+
+Generate an existing manifest from Synapse
+------------------------------------------
+
+.. code-block:: shell
+
+   schematic manifest -c /path/to/config.yml get -dt <your data type> -d <your synapse "Top Level Folder" folder id> -s
+
+Validate a manifest
+-------------------
+
+.. code-block:: shell
+
+   schematic model -c /path/to/config.yml validate -dt <your data type> -mp <your csv manifest path>
+
+Submit a manifest as a file
+---------------------------
+
+.. code-block:: shell
+
+   schematic model -c /path/to/config.yml submit -mp <your csv manifest path> -d <your synapse "Top Level Folder" id> -vc <your data type> -mrt file_only
+
+
+In depth guide
+--------------
+
 .. click:: schematic.__main__:main
   :prog: schematic
   :nested: full

docs/source/conf.py

@@ -13,6 +13,8 @@
 import os
 import sys
 
+import sphinx_rtd_theme
+
 file_dir = os.path.dirname(__file__)
 sys.path.append(file_dir)
 import pathlib
@@ -27,7 +29,7 @@
 
 toml_metadata = _parse_toml(toml_file_path)
 project = toml_metadata["name"]
-copyright = "2022, Sage Bionetworks"
+copyright = "2024, Sage Bionetworks"
 
 author = toml_metadata["authors"]
 
@@ -40,7 +42,7 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ["sphinx_click"]
+extensions = ["sphinx_click", "sphinx_rtd_theme"]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
@@ -57,15 +59,21 @@
 # This pattern also affects html_static_path and html_extra_path.
 exclude_patterns = []
 
+# The master toctree document.
+master_doc = "index"
 
 # -- Options for HTML output -------------------------------------------------
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = "alabaster"
+html_theme = "sphinx_rtd_theme"
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
+
+html_theme_options = {
+    "collapse_navigation": False,
+}

docs/source/configuration.rst

@@ -0,0 +1,84 @@
+Configure Schematic
+===================
+
+This is an example config for Schematic. All listed values are those that are the default if a config is not used. Remove any fields in the config you don't want to change.
+If you remove all fields from a section, the entire section should be removed including the header.
+Change the values of any fields you do want to change.  Please view the installation section for details on how to set some of this up.
+
+.. code-block:: yaml
+
+    # This describes where assets such as manifests are stored
+    asset_store:
+        # This is when assets are stored in a synapse project
+        synapse:
+            # Synapse ID of the file view listing all project data assets.
+            master_fileview_id: "syn23643253"
+            # Path to the synapse config file, either absolute or relative to this file
+            config: ".synapseConfig"
+            # Base name that manifest files will be saved as
+            manifest_basename: "synapse_storage_manifest"
+
+    # This describes information about manifests as it relates to generation and validation
+    manifest:
+        # Location where manifests will saved to
+        manifest_folder: "manifests"
+        # Title or title prefix given to generated manifest(s)
+        title: "example"
+        # Data types of manifests to be generated or data type (singular) to validate manifest against
+        data_type:
+            - "Biospecimen"
+            - "Patient"
+
+    # Describes the location of your schema
+    model:
+        # Location of your schema jsonld, it must be a path relative to this file or absolute
+        location: "tests/data/example.model.jsonld"
+
+    # This section is for using google sheets with Schematic
+    google_sheets:
+        # Path to the google service account creds, either absolute or relative to this file
+        service_acct_creds: "schematic_service_account_creds.json"
+        # When doing google sheet validation (regex match) with the validation rules.
+        #   true is alerting the user and not allowing entry of bad values.
+        #   false is warning but allowing the entry on to the sheet.
+        strict_validation: true
+
+
+This document will go into detail what each of these configurations mean.
+
+Asset Store
+-----------
+
+Synapse
+~~~~~~~
+This describes where assets such as manifests are stored and the configurations of the asset store is described
+under the asset store section.
+
+* master_fileview_id: Synapse ID of the file view listing all project data assets.
+* config: Path to the synapse config file, either absolute or relative to this file. Note, if you use `synapse config` command, you will have to provide the full path to the configuration file.
+* manifest_basename: Base name that manifest files will be saved as on Synapse. The Component will be appended to it so for example: `synapse_storage_manifest_biospecimen.csv`
+
+Manifest
+--------
+This describes information about manifests as it relates to generation and validation.  Note: some of these configurations can be overwritten by the CLI commands.
+
+* manifest_folder: Location where manifests will saved to. This can be a relative or absolute path on your local machine.
+* title: Title or title prefix given to generated manifest(s). This is used to name the manifest file saved locally.
+* data_type: Data types of manifests to be generated or data type (singular) to validate manifest against. If you wanted all the available manifests, you can input "all manifests"
+
+
+Model
+-----
+Describes the location of your schema
+
+* location: This is the location of your schema jsonld, it must be a path relative to this file or absolute path.  Currently URL's are NOT supported, so you will have to download the jsonld data model.  Here is an example: https://raw.githubusercontent.com/ncihtan/data-models/v24.9.1/HTAN.model.jsonld
+
+Google Sheets
+-------------
+Schematic leverages the Google API to generate manifests. This section is for using google sheets with Schematic
+
+* service_acct_creds: Path to the google service account creds, either absolute or relative to this file. This is the path to the service account credentials file that you download from Google Cloud Platform.
+* strict_validation: When doing google sheet validation (regex match) with the validation rules.
+
+    * True is alerting the user and not allowing entry of bad values.
+    * False is warning but allowing the entry on to the sheet.