feat: add observability tooling for Flashbox images

[MoeMahhouk]

This pull request introduces a comprehensive observability stack using Prometheus for system and container monitoring, along with dynamic configuration and firewall rules to support secure metrics collection and export. The changes include package additions, new service definitions, Prometheus configuration and recording rules, as well as dynamic and environment-aware firewall and config scripts for both L1 and L2 environments.

Observability and Monitoring Integration

• Added Prometheus, Prometheus Node Exporter, Prometheus Process Exporter, and gomplate to the build (mkosi.conf), enabling metrics collection and templated configuration generation.
• Introduced systemd service units for prometheus, prometheus-node-exporter, and process-exporter to manage the lifecycle of these monitoring components.
• Added Prometheus configuration templates (prometheus.yml.tmpl) supporting scrape configs, remote write endpoints, and dynamic relabeling, as well as a process exporter config and recording rules for aggregated metrics.

Dynamic Configuration and Firewall Support

• Implemented dynamic configuration scripts for both L1 and L2 environments to generate environment-specific settings, supporting both local (qemu) and production (vault) modes.
• Updated firewall configuration scripts for L1 and L2 to allow outbound connections to dynamically configured observability metrics endpoints, ensuring secure and flexible metrics export.

Other System Integration

• Adjusted the searcher-firewall.service to require and wait for fetch-config.service, ensuring configuration is loaded before applying firewall rules.

---

Most Important Changes

Observability/Monitoring Stack

• Added Prometheus and exporters as build dependencies and provided systemd service units for automated startup and management (mkosi.conf, prometheus.service, node-exporter.service, process-exporter.service).
• Introduced Prometheus configuration templates, process exporter config, and recording rules for key system and container metrics aggregation and remote write support (prometheus.yml.tmpl, process-exporter.yml, recording_rules.yml).

Dynamic Configuration

• Added dynamic config scripts for L1 and L2 to generate environment-specific configuration, supporting both local and production deployments (dynamic-config.sh).

Firewall Integration

• Updated firewall scripts to allow outbound connections to dynamically configured metrics endpoints, ensuring secure metrics export from the host only (firewall-config).

Systemd/Service Coordination

• Modified searcher-firewall.service to depend on fetch-config.service, ensuring correct configuration order at boot.