SPIKE: Define and run benchmarking experiments

[doodlesbykumbi]

Overview

The code changes to accommodate metrics collection will likely be shared among the different connectors, however there's nuance to each connector (at the very least when it comes to generating load). We'll likely want to carry out analysis on a variety of traffic profiles in the experiments to come. For example pg_bench is great for creating realistic traffic for Postgres.

The approach to running experiments is similar across connectors.

1. Determine what is possible in terms of controlling the experiment
2. Define the experiment
3. Generate load
4. Observe generated data

To simplify things this exercise will focus on Postgres to prove out the process of running experiments.

NOTE that at this stage the traffic profile of interest is the light operating profile, defined as:

• 2000 open connections
• 10 active (transmitting) connection
• 50 waiting (non-transmitting) connections

Definition of done

• A document exists capturing the means of generic realistic traffic for the Secretless TCP connectors
• A set of experiments are defined for the Secretless TCP connectors
• The set of experiments are run, and the resuting data is made available for reporting

[izgeri]

The requirement from the feature spec is:

Secretless latency under ‘light’ load and maximum throughput (Mbps) per single Secretless container has been measured. ‘Light’ load or operating profile is defined as above, where Secretless is communicating with a database cluster with variable data size per request.

In #1398 we determine precisely how we can capture data within Secretless on throughput and latency. In this card, if I'm reading it correctly, we're asking:

• How can we set up an experiment to determine maximum throughput?
• How can we set up an experiment to measure latency so that:
  • Secretless is communicating with a database cluster with variable data size per request
  • Secretless is operating under a "light" operating profile (as defined in the description)
  • Secretless is operating at maximum throughput