CI for v3.0.1 publication

[bact]

Summary

Updated CI for v3.0.1 publication and new spec-parser directory structure

• Setup redirection
  • Website: Decision - 3.0 will redirect to the latest. Spec website: How do we like the URL aliases to work? #1018 (comment)
  • Schema/RDF: 3.0 redirection/alias Advertise version: MAJOR.MINOR.PATCH or MAJOR.MINOR ? #996 (comment)
  • Setup case-specific redirections (for name differences between 3.0.0 and 3.0.1) Redirection requirement of v3.0 URL -> latest URL #1069
• Update generation dir names Update CI for new spec-parser output #1066
  • Once new spec-parser is merged, update RDF_BASE_DIR and MKDOCS_BASE_DIR values in .github/workflows/publish_v3.yml to "rdf" and "mkdocs" (see comments at Lines 24-25)
• Update checkout branch/tag names Changes needed for new branch structure #1092
  • Once new branch/tag release names available, update REF_SPEC, REF_MODEL, and REF_PARSER values in .github/workflows/publish_v3.yml (see Lines 14-16)

Will resolve #996, resolve #1018, resolve #1069

Removal of old workflows

This PR also remove .github/workflows/publish.yml and .github/workflows/publish_common.yml as they are no longer in use for v3 publication.

All of them are still available in v2 branches.
For example, https://github.com/spdx/spdx-spec/tree/development/v2.3/.github/workflows

Main workflow steps

Below are steps in .github/workflows/publish_v3.yml.
The big ones are the last two where they do extra work to keep old URLs working.
The YAML file has more comments.

1. Checkout spdx-spec
2. Checkout spdx-3-model
3. Checkout spec-parser
4. Install pre-requisites for spdx-spec
5. Install pre-requisites for spec-parser
6. Build model files
   • Using spec-parser
7. Create directories for model (MkDocs) and RDF files
8. Copy JSON annotations
9. Copy JSON-LD context and RDFs
10. Generate JSON schema
11. Copy model files and file list for MkDocs
12. Set Git identity
13. Sync GitHub Pages
14. Build complete MkDocs configuration
    • In this step, it combines model file list (mkdocs-files.yml, generated by spec-parser) with mkdocs.yml (it replaces "MODEL_PLACEHOLDER" string with the content of mkdocs-files.yml)
15. Deploy and set aliases
16. Copy JSON annotations, JSON schema, JSON-LD context, and RDFs to alias directories
    • This step creates copies of annotations/schema/RDFs to all alias directories,
      so it can be accessible from all old URLs for backward compatibility.
    • For example,
      • https://spdx.github.io/spdx-spec/v3.0/model/schema.json
      • https://spdx.github.io/spdx-spec/v3.0.1/rdf/schema.json
        will all accessible and have the same content.
17. Make redirections (for renamed model elements and moved annexes)
    • This step make old names like imports to redirect to its new name at import.
    • It reads a list of redirect mapping from etc/redirect-map.csv;
      First value is 'from' (source), second value is 'to' (target).
    • 'from' and 'to' values will be inserted into the redirection HTML template etc/redirect-template.html
    • to create HTML files with 'from' name that will refresh the browser to the 'to'.

Demo

The demo of this workflow is at https://bact.github.io/spdx-spec/

To see what the workflow will look like when running, see a log at
https://github.com/bact/spdx-spec/actions/runs/10716224938
(this is deployed on my own personal repo, not the spdx one).

Redirection tests

To illustrates the redirection, here are some example.

• Versions
  • / (root) --> v3.0.1/
  • latest --> v3.0.1/
  • v3.0 --> v3.0.1/
  • v3.0/model/Software/Classes/File/ --> v3.0.1/model/Software/Classes/File/
• 3.0.0 vs 3.0.1 name differences in model
  • v3.0/model/Software/Properties/contentType/ --> v3.0.1/model/Core/Properties/contentType/
  • v3.0/model/Build/Properties/parameters/ --> v3.0.1/model/Build/Properties/parameter/
  • v3.0.1/model/Core/Properties/imports/ --> v3.0.1/model/Core/Properties/import/
• Moved docs
  • v3.0/introduction/ --> v3.0.1/front/introduction/
  • v3.0/normative-references/ --> v3.0.1/references/
  • v3.0/annexes/RDF-object-model-and-identifier-syntax/ -> v3.0.1/annexes/rdf-model/
  • v3.0/annexes/getting-started/ -> https://github.com/spdx/using/blob/main/docs/getting-started.md/ (External)

[bact]

@licquia here's the new CI for 3.0.1. Please kindly review. Thank you.

• Most of the configurations are at Lines 13-30 of .github/workflows/publish_v3.yml
  This is designed to work with new spec-parser output directory structure (change Lines 23-24 for new dir structure).

• The last step is to handle few name differences between 3.0 and 3.0.1,
  so links like https://spdx.github.io/spdx-spec/v3.0/model/Core/Properties/imports/ will not break.
  The mapping table is in etc/redirect-map.csv

• The step before last is to handle URL from old spec-parser output dir structure,
  so links like https://spdx.github.io/spdx-spec/v3.0/model/schema.json will still working,
  in parallel with the new link https://spdx.github.io/spdx-spec/v3.0/rdf/schema.json

[bact]

Ping @zvr - I have to add index.md back at the docs root level here.

Because without it, URLs like https://spdx.github.io/spdx-spec/v3.0/ and https://spdx.github.io/spdx-spec/v3.0.1/ will return 404 File Not Found as there is no index.html being generated. MkDocs also use docs/index.md as an entry point.

Not sure if this will affect the ISO/OMG submissions or not, we will figure out, but in any case the index.md really has be at that exact location :(

UPDATE: it is now fixed by #1101

[goneall]

I didn't see any issue - but it is a very large set of changes so some additional reviews would be helpful.

The redirects-map probably needs another set of eyes as well as the CI logic

[bact]

Thank you. It is really a huge changeset.

I just added comments explaining few steps more in the publish_v3.yml to facilitate the review.

[zvr]

I think this whole thing got too complicated and not easy to review.

I'll approve it and merge it and then we can incrementally fix issues (if there are any).