Code offer: CloudWatch monitoring of job queue size

[ThrawnCA]

If anyone is interested, we've recently developed a small shell script designed to monitor the job queue and report metrics to Amazon CloudWatch.

#!/bin/sh

LOG_FILE=/var/log/ckan/ckan-worker-queue.log
TEMP_FILE=/tmp/ckan-worker-queue.log
echo "CKAN job queue at $(date):" >> $LOG_FILE
JOB_COUNT=$(/usr/lib/ckan/default/bin/paster --plugin=ckan jobs list -c /etc/ckan/default/production.ini 2>/dev/null | tee -a $LOG_FILE | tee $TEMP_FILE | wc -l)
echo "Total: $JOB_COUNT job(s)" >> $LOG_FILE
if [ "$JOB_COUNT" -gt 0 ]; then
    OLDEST_TIME=$(date -u --date $(head -1 $TEMP_FILE | awk '{print $1}') +'%s')
    MAX_AGE=$(expr $(date -u +'%s') - $OLDEST_TIME)
    echo "Oldest job is from $OLDEST_TIME ($(expr $MAX_AGE / 60) minute(s) ago)" >> $LOG_FILE
else
    MAX_AGE=0
fi
aws cloudwatch put-metric-data --region us-east-1 --namespace CKAN --metric-data "MetricName=job_queue_size,Dimensions=[{Name=Application,Value=MyCKANApplication},{Name=Environment,Value=DEV}],Value=$JOB_COUNT" "MetricName=max_job_age,Dimensions=[{Name=Application,Value=MyCKANApplication},{Name=Environment,Value=DEV}],Value=$MAX_AGE"

Note that LOG_FILE must be in a directory that exists and is writable by the user running the script. Also, the machine where this runs needs to have appropriate AWS credentials set up to allow access to CloudWatch.

The metric dimensions are optional; we use them because we have multiple different CKAN portals, and multiple environments (DEV, STAGING, etc), in the same AWS account.