Feature request: semantic model validator

[djinnome]

Model sanity checks:

Total population tests

• Is total population information inherent to the initial conditions, or is there a separate parameter?
• Can a model be made scale free? i.e. scaled to a population value of 1
• Check at different time points to ensure conservation of total population

“Aggregate_parameter” interpretation

• Check that a Mapping between model parameters described in literature and the corresponding aggregate parameters used for model representation/simulation
  --> This should help interpret the parameters that are calibrated, as well as develop the mappings below in 5.

Scale/units test for parameters

• In cases of “aggregated parameters” visible at the syntactic level, do the units agree when combining the original model parameters?
• Can be checked for individual parameters: agreement with each other
• Can be checked for aggregated parameters: agreement with state variables & corresponding equation
• Check if parameters are explicitly being divided by population
• if yes, then Derivative calculations will incorrect (since those already divide by total population). Related to issue Should aggregate_parameters include total population in the denominator? gyorilab/mira#172

Parameter value tests

• Is a distribution provided (i.e. uniform distribution between a range of two values, or a gaussian distribution with a mean and variance provided)
• Is provenance or reasonable assumptions for this distribution choice provided?
• we hope this provides the necessary information for a posteriori adjustments to parameter choices of simulated forecasts are not satisfactory
• Argument for evidence ontology?
  * Author statement
  * Calibration result
  * Transformation of existing evidence (aggregate parameters)
  * WAG
• Prior predictive check against data to see if data is in support of parameters

Missing initial conditions:

• Ensure information is provided to map from observed variables in data to initial conditions for model state variables
• Check for the existence of a function that maps observed variables for a start time map to the initial conditions of model state variables
  -Reasoning provided for this function choice
• Observed variables should be conserved if fractions are used e.g. {infected} in observed data vs. {infected, sick, affected} model states
• Check that all model state variables are provided an initial condition
• Check that the initial state values are reasonable
• Check that the solution mapping applied to the start state is the same as the observed variables for that date.

Observation models

• Ensure information is provided to map from simulated state variable outputs from the model to observed variables

  • Check for the existence of a function that maps model state variables to values that agree with variables in observed data
  • Reasoning provided for this function choice
  • Check that values for every observed variables is calculated

• Ensure mapping from IM_3b are not redundant combinations of variables

• Intended idea: Flow from state to state is allowable; one state = scalar multiple of another shouldn’t be allowed

• Example: “Hospitalization = 10% of Infected individuals” should not be allowed

Stiffness test

• Check if stiffness arises from simple “patient zero” initial conditions (i.e. minimum infectious population and maximum susceptible)
  -(b) Check that differences in parameter scales are reasonable to avoid introducing stiffness into the system (may not be a concern…)

Permanence of Death

• Make sure dead individuals stay dead; i.e. no reaction that allow dead individuals to return to a non-dead state

Ensemble Model tests (EM tests)

• Test scale of each model separately to test agreement
• Ensure total populations are in agreement across different models (either scale-free or not)
• Test that initial conditions after mapping are in agreement across different models
• Test post-mapping variables are reasonable across different models (How do we do this?)

[djinnome]

To avoid many of these problems, I would recommend that all models in the ASKEM model representation be normalized to a total population of 1.

This addresses the following issues:

• Whether total population should be a parameter or part of the rate law?
• How to rescale models that are part of an ensemble
• How to specify the units of the state variables and parameters of the models.