Promtool-rules change configuration and add rule unit testing

[janboll]

ARO-15912

What

Adding call to promtool

Adding support for service owned alerts, where alerts folder and files are stored in service directory:

• cluster-service/alerts
• backend/alerts
  ....
  By putting alerts into service folders we can easily grant code ownership to these files.

These folders will be referenced by a configuration file in observability/alerts:

rulesFolders:
 - cluster-service/alerts
 - backend/alerts
untestedRules:
additionalRules:
 - kubernetesControlPlane-prometheusRule.yaml
outputFile:
 - dev-infrastructure/obsevability/generated-prometheus-rules.bicep

All files in rulesFolders must have tests
Files referenced in additionalRulesuntestedRules are exemptions and don't require a test

Why

Want to have strict testing for prometheus alerts
Need to support more than one alert file

[stevekuznetsov]

Why did this move out of complete? The point of this pattern is to have all necessary data read in and created here so testing past this step is easy.

[janboll]

Oh, my point was, why create the output if the test could fail?

[stevekuznetsov]

Fair, but losing out on having options as useful input to execution is a shame.

[tony-schndr]

Overall this is great! Some improvements/nits for a follow up if you wish:

When testing I don't see any output on what tests were ran, it would be nice to see a format looking something like:
🟢 cluster-service/alert/InstancesDownV1 passed
❌ cluster-service/alert/InstancesDownV1 failed

I saw output only when it failed.

I also tried to generate the alerts in bicep, I updated the name InstancesDownV1 to InstancesDownV2 and I didn't see the update in the generated bicep module. Not sure if I was doing something wrong.

Maybe you're already working on it, but could we also get some documentation on how this is used?

[github-actions]

Please rebase pull request.

[janboll]

Great idea with the output. Let me take another look at this.

I planned on documenting it once it's set. But I can add some docs in this PR

[janboll]

@tony-schndr that is a bit complicated, since promtool does not provide this kind of output. I think it's not worth the effort, since the we are simply calling promtool.

Give this test file my plan would be:

rule_files:
- testing-prometheusRule.yaml
evaluation_interval: 1m
tests:
- interval: 1m
  input_series:
  - series: 'up{job="app", instance="app-1:2223"}'
    # 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    values: "0x14"
  - series: 'up{job="app", instance="app-2:2223"}'
    # 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1
    values: "1x4 0x9 1x4"
  alert_rule_test:
  - eval_time: 4m
    alertname: InstancesDownV1
  - eval_time: 5m
    alertname: InstancesDownV1
    exp_alerts:
    - exp_labels:
        severity: critical
      exp_annotations:
        summary: "All instances of the App are down"
        description: "All instances of the App are down"
  - eval_time: 15m
    alertname: InstancesDownV1

for each alert_rule_test create a testfile and run promtool test on it.

[janboll]

@tony-schndr regarding the missing update, did you run make alerts in the observability folder?