Introduce basic sanity checks on input data before model creation

[ojdo]

As the input data specification has grown very big over the years, a set of simple plausibility checks should be added, so that creating a new input file becomes a less frustrating experience.

[ojdo]

Here are some ideas, structured by spreadsheet

Commodity

• Stock commodities must have a numeric attributes for 'price', 'max' and 'maxperstep'. Attribute 'price' must be finite, while the other two may be infinite.
• Environmental commodities must have numeric (may be inf) values in attributes 'max' and 'maxperstep'.
• Emit a (suppressable) warning for all numeric values that are set for Demand and SupIm commodities. They are ignored.

Further thought: Should these basically empty rows point to a bad data model? Essentially, this table could only declare existing (Commodity, Type) tuples and an additional Site-Commodity table could bundle all attributes (including DSM) which are defined over those combinations. Continue this train of thought in another issue, once there is time.

Process

• All columns must be numeric.
• Interval checks: 'cap-lo' <= 'cap-up' and 'inst-cap' <= 'cap-up'. Note that 'inst-cap' < 'cap-lo' is allowed (to enforce minimum capacity expansion goals).

Process-Commodity

• All columns must be numeric.
• Much more advanced: Calculate an indicator from the (hard to spot, if one is not aware of the startup & partial feature) relationship between 'ratio-min' (if set) and the process properties 'min-fraction' and 'startup-cost'. This could trigger an informational message to STDOUT, listing proportional (process, commodity) combinations with an "easy to understand" (whatever that means) indicator of the process characteristic (e.g. efficiency ranges that are possible).

Transmission

• All columns must be numeric.
• Same as for process: checks for 'inst-cap', 'cap-lo', 'cap-up'.
• Efficiency <= 1. At least emit a STRONG warning if not.

Storage

• All columns must be numeric.
• Same as for process: checks for 'inst-cap-c', 'cap-lo-c', 'cap-up-c' and 'inst-cap-p', 'cap-lo-p', 'cap-up-p'.
• As to efficiency: there might be reasons to have 'eff-in' and 'eff-out' assume "interesting" values. Their product, however, should probably be <= 1. Otherwise, emit a warning.

Demand

• All columns must be numeric, finite and non-negative.

SupIm

• All columns must be numeric, non-negative and <= 1.

Buy-Sell-Price

• All columns must be numeric and bounded (but may be negative).

Hacks

• 'Global CO2 limit' must be numeric, though may be inf to disable the constraint.

DSM

• All columns must be numeric.

Please add your thoughts and experiences with funny errors caused by logical glitches in your input files, which you hoped that urbs would have told you about right away!

[lodersky]

validation.py already exists. Will be further improved.