A more resilient build chain for document collections

[ronaldtse]

This issue is to decide/develop a more resilient build child for people who work on document collections. I think this ties into Metanorma collection work @abunashir is working on (#180) and we could fix things from there.

From @Kritner below. This flow is also one that I know some others (and I) work with, so it is indicative of how people build collections.

Perhaps my current workflow context is important.

• Work on document x locally from a clone of a fork of our main repository
• Finish work on document x
• Build document x locally
• Ensure document meets expectations
• Commit changes to document x and push up to fork of main repository
• PR fork to main
• Wait for successful build to merge PR (at a minimum, we may wait on the timing of other moving parts to merge as well)
• After successful build, push all artifacts to github pages
  • this step is where "the problem lies", in that the "silent failure" documents were never produced, and GH pages now does not have these documents available to it. Though this may be an issue of force pushing the gh-pages branch, not super familiar with the GH actions.
• If build fails, take no action on github pages

Given the above scenario, I only ever worked on document x, so I'm being careful about monitoring its progress. However, in this case documents 1...n failed to build in both our PR build as well as our "merge PR" build, without any sort of indication of said failure, and without having worked on any document in the range of 1...n.

It would be one thing if document x "silently failed" to build, as we know to be wary of that document as we're actively working on it, but that's not the case here, documents that weren't touched are now "silently failing" due to a change in a dependency down stream.

Perhaps a "strict" mode or similar would work, though we have plenty of warnings we'll need to work through.

My personal preference would be if any one document in the batch fails, even if 99% pass, the overall build is deemed a failure. This could be on our end as well in regards to how we're doing our make file, but I haven't really looked closely into how all the pieces fit together as it relates to the make file and the github action.

Originally posted by @Kritner in #189 (comment)

[ronaldtse]

We have just merged #180 which will aid this issue and gives us the possibility to propagate the error codes from individual document compilation.

As @Kritner mentioned, the problem is at this step:

• After successful build, push all artifacts to github pages
  • this step is where "the problem lies", in that the "silent failure" documents were never produced, and GH pages now does not have these documents available to it. Though this may be an issue of force pushing the gh-pages branch, not super familiar with the GH actions.

My personal preference would be if any one document in the batch fails, even if 99% pass, the overall build is deemed a failure. This could be on our end as well in regards to how we're doing our make file, but I haven't really looked closely into how all the pieces fit together as it relates to the make file and the github action.

Now with the metanorma site generation command in place, it is possible to define different behavior:

• --strict: once a failure occurs, bail out
• --continue-on-error: don't bail out on error, at the end display errors but return failure code.

Output necessary:

• A summary at the end should tell the user what documents have succeeded and what documents have failed.
• Warnings should be made available in some manner at the end, a summary with per-document details prepended with the document's path.

And we can provide different error codes per situation:

• 0: All documents compiled successful. A success message should be shown.
• 1: Some failure occurred. The errored-out document should be warned out at the end.

This is not unlike the spec/test outputs, where a summary is provided (test successes) and the individual test errors are singled out for developer attention.

@abunashir can you help design this feature and proceed with it? Thanks!

[ronaldtse]

We are facing this issue again:

• Build failure due to unable to upload artifact packed-mn#135
• (URGENT) PDF: Crash in mn-samples-ogc 14-065r2 metanorma-ogc#332

When PDF generation fails, we MUST fail the whole build process at the end.