Defining custom math within the model definition

[brynpickering]

Custom math functionality introduced in #414 allows a user to update the base math file with data in other files. However, this custom math exists in a mutually exclusive set of files to the rest of the model definition (these YAMLs can only define model components - variables, constraints, etc. - as top-level keys). So, if you want to define a constraint for a specific technology, you have to construct a suitable where string ([my_tech] in techs) or slice components inside the math expression (flow_out[techs=my_tech, node=my_node]). This could potentially be circumvented by defining math expressions inside the model definition, and then constructing suitable custom math from that.

I could only really see it working if quite verbose syntax is allowed, e.g.:

Model def

techs:
  my_tech:
    ...
    carrier_in: [biofuel, coal]
    carrier_out: electricity
    biofuel_eff: 0.5
    coal_eff: 0.6
    ...
    custom_math:
      constraints:
        my_custom_constraint:
          equations:
            - expression: >
                flow_out[carrier=electricity] == 
                flow_in[carrier=biofuel] * biofuel_eff 
                + flow_in[carrier=coal] * coal_eff
        balance_conversion_plus_primary:  # this constraint exists
          skip: true  # or something similar to tell the constraint to ignore this tech

which would translate to an equivalent custom math syntax of:

Math def

constraints:
  my_custom_constraint:
    foreach: [timesteps, nodes, techs]
    where: "[my_tech] in techs"
    equations:
      - expression: >
          flow_out[carrier=electricity] == 
          flow_in[carrier=biofuel] * biofuel_eff 
          + flow_in[carrier=coal] * coal_eff
  
  balance_conversion_plus_primary:  # using `skip` to update the `where string`
    where:  "inheritance(conversion_plus) AND carrier_ratios>0 AND NOT [my_tech] in techs"

Things we'd need to allow

1. foreach still needs to be there as you need to be able to add dimensions over which to iterate (in the above case, timesteps is a key one that is otherwise missing).

2. skipping or deactivating base math needs to be there as new expressions can easily clash with base constraints. On that note, should it also be possible to "update" an existing math expression for the tech in question? E.g.:

Model def

techs:
  my_tech:
    custom_math:
      constraints:
        balance_conversion_plus_primary:  # this constraint exists
          equations:
            - expression: >
                flow_out[carrier=electricity] == 
                flow_in[carrier=biofuel] * biofuel_eff 
                + flow_in[carrier=coal] * coal_eff

Can be successfully parsed as:

Math def

constraints:
  balance_conversion_plus_primary:  # this constraint exists
    equations:
      - where: "[my_tech] in techs"
        expression: >
          flow_out[carrier=electricity] == 
          flow_in[carrier=biofuel] * biofuel_eff 
          + flow_in[carrier=coal] * coal_eff
      - where: "NOT [my_tech] in techs"
        expression: >
          reduce_carrier_dim(flow_out / carrier_ratios[carrier_tiers=out], carrier_tier=out) == 
          reduce_carrier_dim(flow_in * carrier_ratios[carrier_tiers=in], carrier_tier=in) * flow_eff

3. there should be scope to introduce new decision variables, hence why the math component key level would need to exist (constraints, variables, etc.), so that this can also work:

Model def

techs:
  my_tech:
    custom_math:
      variables:
        my_custom_variable:
          foreach: [techs]  # we'd probably not to always enforce that techs is in foreach
          bounds: 
            min: 0
            max: .inf
      constraints:
        my_custom_constraint1:
          foreach: [timesteps, nodes, techs]
          equations:
            - expression: >
                my_custom_variable >= 
                flow_in[carrier=biofuel] * biofuel_eff 
                + flow_in[carrier=coal] * coal_eff
        my_custom_constraint2:
          foreach: [timesteps, nodes, techs]
          equations:
            - expression: >
                flow_out[carrier=electricity] <= my_custom_variable

Sticking points that come to mind

1. How do we handle applying custom math at the tech level that is then updated at e.g. only one node?

Model def

nodes:
  my_node:
    techs:
      my_tech:
        custom_math:
          constraints:
            my_custom_constraint:
              foreach: [timesteps, nodes, techs]
              equations:
                - expression: >
                    flow_out[carrier=electricity] <= 
                    flow_in[carrier=biofuel] * biofuel_eff 
                    + flow_in[carrier=coal] * coal_eff

This should lead to something like:

Math def

constraints:
  my_custom_constraint:
    foreach: [timesteps, nodes, techs]
    where: "[my_tech] in techs"
    equations:
      - where: "NOT [my_node] in nodes"
        expression: >
          flow_out[carrier=electricity] == 
          flow_in[carrier=biofuel] * biofuel_eff 
          + flow_in[carrier=coal] * coal_eff
      - where: "[my_node] in nodes"
        expression: >
          flow_out[carrier=electricity] <= 
          flow_in[carrier=biofuel] * biofuel_eff 
          + flow_in[carrier=coal] * coal_eff

But I don't really know how that would be implemented internally without a lot of potential for edge cases coming in and destroying everything.

2. how do we handle multiple techs/nodes referring to the same constraint name? If tech1 and tech2 reference the same constraint in their custom math, we can create expression list elements for each with where: [tech1] in techs for the first, and where: [tech2] in techs for the second. That probably works OK, but we would need to make sure they have matching foreach (and maybe some other things need to be coherent between them that I haven't thought of yet).

3. We would be doing string concatenation of the where string, which would require special attention to avoid creating strings that no longer make sense not [tech1, tech2] in techs and [tech1] in techs.

4. The level of nesting this leads to hurts my eyes, so maybe it's just a bad idea to try and shoehorn it in...

Summary

It could work, but need to come up with a neater way of defining it to a. mitigate edge cases and b. make it visually cleaner.