Extracting metrics from Conductor logs to use for autoscaling in AWS

[ghost]

Hi,

We have logs events in the form:

2023-08-15T13:10:37.498576700Z 2023-08-15 13:10:37.496  INFO 10 --- [rter-1-thread-1] ConductorMetrics                         : type=GAUGE, name=task_queue_depth.class-WorkflowMonitor.ownerApp-unknown.taskType-store_audit_log_events, value=0.0

and we would like to use the values in these logs for autoscaling conductor workers in AWS.

So far we have been able to get the value=0.0 section of the log message using a filter pattern to match space delimited log events. However, we're unable to separate the actual value= part from the value itself.

We've explored trying to change the format of the message using a log4j properties file but we realised that we cannot change the format of the message section of the log as it is being formatted in code. It also looks like the code is part of Dropwizard Metrics and not part of Conductor itself. We would prefer not to have to build our own versions of Dropwizard and Conductor for something so trivial.

Ideally, we would like to output the logs without the key names, e.g. type, name, value etc. So my question is, is there a way to achieve this through configuration?

Thanks in advance,

Ammaar

[boney9]

I think CPU and Memory would be the best metrics you could leverage for this, another option is to scan the queue depth using the queues API.