Configurable extraTags when shipping kernel logs externally

[jtcressy]

Feature Request

Addition of a new kernel arg talos.logging.kernel.extra-tag to apply extra tags to the logging destination configuration, similar to service logs.

We could format this new kernel arg similar to talos.environment so that multiple tags can be defined.

Also, if possible, include the support of templating into the value portion similar to talos.config with the options ${uuid} ${serial} ${mac} ${hostname} - this would enable us to include these identifiers as tags without hard-coding them.

Description

Metadata is distinctly lacking when shipping kernel logs to centralized logging systems. Ideally, I would want logs to include the source node's hostname at a minimum. As it stands, when one machine is having issues, it is almost impossible to identify it without opening the console of every machine and inspecting the logs manually.

If we had a way to include metadata via kernel arguments just like we can with service logs, this would solve the problem.

Example

Kernel cmdline for external logging with multiple templated tags
talos.logging.kernel=udp://ingest.local:1234 talos.logging.extra-tag=hostname=${hostname} talos.logging.extra-tag=machine_serial=${serial} talos.logging.extra-tag=machine_uuid=${uuid} talos.logging.extra-tag=machine_mac_addr=${mac}