add facility for displaying status checks to front page

[trel]

a new facility for running defined 'checks'

• a check should have a name, description, and 'code'
• code would be a single query, or multiple queries, and that and some math or comparisons...
• should be able to return a status... possible enum of ['okay','unexpected','warning','bad']
• should display those statuses on the front page
• should run these periodically ( a config? per item? ) but also allow 'check now' on demand
• should be able to be turned off or marked 'inactive' (but 'active' by default)

this could possibly 'eat' the 'API endpoint status' functionality currently provided in the footer

[connectbo]

I had some thoughts and questions while doing some experiments on this today.

One possible solution could be we add a 'check' context provider, which will provide each check name, info, check logic, status. To achieve that, we need to perform checks on all targets (i.e. all resources ...) That logic seems straightforward, but we need to have each resource in place, which is relatively time-consuming and not good for performance perspective (we only load information we need for now using offset and limit). A page refresh could cause ZMT do all checks again.

I was thinking if irods could perform checks at the time a new resource is created (or modified ?) and update the result in its status property? so it's always in place, and we can have a /status endpoint that gives us an overview?

That's a lot and I could be wrong, open to discussion :)

[trel]

The checks I'm thinking of can be per-resource or per-zone or per-user or ... all kinds of things. So let's think of specific tests/checks we'd like to run and see if any patterns emerge.

Performance is not really an issue here - the ZMT can do these in the background and load them into a global context and just hold them... and then maybe check again after 10min or so... Definitely not in the hot path of making the user wait for anything to happen in realtime.

Let's plan to talk and get a couple examples captured here.

First thinking...

• do all compound resources have exactly one cache and one archive as children
• is there any unused metadata in the catalog that can be removed (iadmin rum)
• are any unixfilesystem resources about to run out of space
• are any replication, random, passthru resources currently leaf resources in existing hierarchies
• do any storage resources (unixfilesystem, univmss, s3) have children
• resources with names with spaces, or other potentially problematic characters
• resources with names that are too long
• users with no password

[trel]

Thinking about configuration for this...

I like the idea of a directory of checkfiles

• one that is in the codebase somewhere (we ship it) (the defaults)
• one for additional/overrides - perhaps named custom_checkfiles (optional) (can be empty / not existent)

I'm also open to a single-file format (.json?) for a checkfile that defines multiple checks... but... maybe that's harder to reason about as an administrator? Harder to hold correctly? More brittle since breaking one breaks them all?

So on deployment, the ZMT would find and load the checks:

• shipped/built-in checkfiles (the defaults)
• admin-defined checkfiles
  • overrides of shipped/built-in checks
  • additional custom checks

The interface could distinguish/show which are default, override, additional (with an icon/symbol and/or color)

So... once we know the parts we need... we'll formalize that in a checkfile.json schema?

• https://json-schema.org/specification.html

[trel]

• Every check should also define minimum_server_version and maximum_server_version for when the check is valid / should be run.

• Should have a check to find out if ZMT itself is running the latest version:

  $ curl -s https://github.com/irods/irods_client_zone_management_tool/releases/latest | cut -d '"' -f 2 | cut -d '/' -f 8
  0.1.0
  

• Every check should declare any dependencies by name (aka, the REST_API endpoint has to be alive or the zone_report has to already be downloaded/available). This also suggests that if a named dependency isn't defined, it should itself be listed in the dashboard with an 'undefined/missing/required' icon/label. Also suggests that checks can be represented as a tree? Not sure on this... the case where a check requires more than one dependency... wouldn't be a tree... Have to think on this some more. Definitely not a requirement to ship.

[trel]

I've broken the following into new issues:

• supporting/loading custom checkfiles #104
• define and validate the checkfiles #105
• discovering/loading checkfiles without specifying/naming in index.js #106
• new check: compound, one cache, one archive #107
• new check: any unused metadata #108
• new check: are any unixfilesystem resources about to run out of space #109
• new check: using coordinating as leaf #110
• new check: storage resources with children #111
• new check: resources with problematic characters #112
• new check: resources with names that are too long #113
• new check: users with no password #114

---

I think the remaining work for this is...

• active/inactive - in the checkfiles, and then enforced by the UI itself, persistent in localStorage (for now)
• enforce minimum/maximum server version