Add channel instrumentation & metrics to Firehose

[bradgessler]

Metrics on the following have been requested for Firehose:

Connection metrics

• Number of connections
• Number of connections established
• Number of closed connections

Channel metrics

• Number of catch-ups performed
• Number of "channels"
• Channels created
• Channels destroyed
• Channel pubs
• Per channel counters(subscribers,events,sizes,etc)

Redis commands

• Counts on events that cause Redis commands to occur?

Implement these as a module in Firehose. We'll use the same Firehose server to track these metrics.

[bradgessler]

@ProTip can the connection metrics be measured from nginx connections? Are you looking for something more specific like WS vs LongPolling vs Multipi-plexing connections?

Also what time intervals are you looking for connections. Connections per second? Connections per minute? The time resolution will need to be known per metric that's to be measured.

[ProTip]

It would be beneficial to track the different types of connections/disconnections separately from nginx and from the POV of Firehose.

A lot of what we collect is at 10 second resolution. For the purposes of Firehose it should probably be configurable. We probably don't wan to expose the metrics as rates from Firehose; just the raw counts.

Do you have a strategy for efficiently flushing the metrics from all the Firehose workers to Redis?