Exposure Notification Private Analytics Ingestion

This repository contains implementations for Apache Beam batch pipelines to process private data shares stored in Firestore according to the Exposure Notification Private Analytics protocol. It assumes private data shares uploaded as Firestore documents, as done in the Exposure Notification Express template app. These documents contain encrypted packets using the Prio protocol. The pipeline implementation converts them into the format that downstream Prio data processing servers expect, defined in the Avro schema here.

This implementation make use of Firestore as a scalable NoSQL db for subsequent batching and aggregation. Alternative implementations might operate a custom backend endpoint to accumulate the packets, or use a pubsub mechanism. Since the packets are encrypted on device, the channel over which the packets travel need not be trusted.

This repository also contains the Firebase configuration to secure Firestore with security rules as well as Terraform scripts to bring up required infrastructure.

Setup

Multiple Maven modules

The project is structured into multiple maven modules to allow incorporation of outside implementations of attestation. Implementations need only depend on the DataShare model module, and a profile can be added to get it included in the pipeline module build. The pipeline pulls available implementations dynamically.

Since there aren't too many individual classes that make up each module, and since they are only meant to be packaged and executed together, we use a single source tree for all modules.

Follow the Getting started with Google Cloud Dataflow page. You will need the following:

1. Set up a Google Cloud project or use an existing one. Then import the Google Cloud project into Firebase.

2. Enable APIs: Container Registry, Cloud Build, Cloud Datastore and Dataflow.

3. Create an asymmetric signing key

4. Create a storage bucket for your outputs.

5. Create a service account with permissions for Firestore, reading the KMS key, Dataflow, and Cloud Storage.

6. Download the a key for your service account and store as credentials.json. Keep those credentials save!

Clone Submodules

A submodule is need to access the avro definitions of the Prio classes.

git submodule update --init

Useful Environment Variables

Setting the following environment variables can be handy when working in the project. Replace values in [...].

export PROJECT="[my-google-cloud-ingestion-project-id]"
export GOOGLE_APPLICATION_CREDENTIALS="credentials.json"
export TEMPLATES="gs://[my-cloud-storage-bucket]/templates"
export PHA_OUTPUT="gs://[my-cloud-storage-bucket]/output/pha"
export FACILITATOR_OUTPUT="gs://[my-cloud-storage-bucket]/output/faciliator"
export KEY_RESOURCE_NAME="projects/[some-ingestion-project]/locations/global/keyRings/[some-signature-key-ring]/cryptoKeys/[some-signature-key]/cryptoKeyVersions/1"

Testing

Unit Tests

To run unit tests:

./mvnw test

Integration Tests

Integration tests go against an actual test project and so need an environment variable:

./mvnw verify

Running the Pipeline

There are two pipelines. One reads Prio data shares from Firestore and generates the outputs which the PHA and Facilitator data share processors will consume. The other deletes expired or already processed data shares from Firestore.

They both take as options the window of time to cover, in the form of a start time and duration. When not supplied, start time for the ingestion pipeline is calculated based on current time rounding back to previous window of length duration. For the deletion pipeline, it goes back two windows to ensure a safety margin of not deleting unprocessed data shares.

Locally

To run the ingestion pipeline locally:

./mvnw compile exec:java \
    -Djava.util.logging.config.file=logging.properties \
    -Dexec.mainClass=com.google.exposurenotification.privateanalytics.ingestion.pipeline.IngestionPipeline \
    -Dexec.args="--keyResourceName=$KEY_RESOURCE_NAME --phaOutput=$PHA_OUTPUT --facilitatorOutput=$FACILITATOR_OUTPUT"

To run the deletion pipeline:

./mvnw compile exec:java \
    -Djava.util.logging.config.file=logging.properties \
    -Dexec.mainClass=com.google.exposurenotification.privateanalytics.ingestion.pipeline.DeletionPipeline \
    -Dexec.args="--project=$PROJECT"

In Google Cloud Dataflow

From local build

export SERVICE_ACCOUNT_EMAIL=$(egrep -o '[^"]+@[^"]+\.iam\.gserviceaccount\.com' $GOOGLE_APPLICATION_CREDENTIALS)

export BEAM_ARGS=(
    "--keyResourceName=$KEY_RESOURCE_NAME"
    "--phaOutput=$PHA_OUTPUT"
    "--facilitatorOutput=$FACILITATOR_OUTPUT"
    "--runner=DataflowRunner"
    "--region=us-central1"
    "--serviceAccount=$SERVICE_ACCOUNT_EMAIL"
)
./mvnw compile exec:java \
    -Dexec.mainClass=com.google.exposurenotification.privateanalytics.ingestion.pipeline.IngestionPipeline \
    -Dexec.args="$BEAM_ARGS"

From Flex Template

See below on how to generate the flex template.

export SERVICE_ACCOUNT_EMAIL=$(egrep -o '[^"]+@[^"]+\.iam\.gserviceaccount\.com' $GOOGLE_APPLICATION_CREDENTIALS)

gcloud dataflow flex-template run "ingestion-pipeline-$USER-`date +%Y%m%d-%H%M%S`" \
    --template-file-gcs-location "$TEMPLATE_PATH" \
    --parameters project="$PROJECT" \
    --parameters keyResourceName="$KEY_RESOURCE_NAME" \
    --parameters phaOutput="$PHA_OUTPUT" \
    --parameters facilitatorOutput="$FACILITATOR_OUTPUT" \
    --service-account-email "$SERVICE_ACCOUNT_EMAIL" \
    --region "us-central1"

Building

We generate a templated dataflow job that takes all pipeline options as runtime parameters.

Building a Flex Template and Launch Container

To build the launch container we added profiles for the ingestion and deletion pipeline.

To build the ingestion pipeline launch container with setting a git derived version:

./mvnw -Pingestion-container-build -Dcontainer-version=$(git describe --tags --always --dirty=-dirty) \
-Dcontainer_registry_tag_prefix='gcr.io/[YOUR_CLOUD_PROJECT]' package

To build the ingestion pipeline with a custom attestation implementation, include the additional attestation profile, which assumes the package is available in any of your configured maven repositories (in .m2/settings.xml or local mvn-settings.xml):

./mvnw -Pingestion-container-build,attestation -Dcontainer-version=$(git describe --tags --always --dirty=-dirty) \
-Dcontainer_registry_tag_prefix='gcr.io/[YOUR_CLOUD_PROJECT]' package

To build the deletion pipeline launch container with the setting a git derived version:

./mvnw -Pdeletion-container-build -Dcontainer-version=$(git describe --tags --always --dirty=-dirty) \
-Dcontainer_registry_tag_prefix='gcr.io/[YOUR_CLOUD_PROJECT]' package

Built containers get automatically published to the container_registry_tag_prefix you set. E.g. for Google container registry: gcr.io/[YOUR_CLOUD_PROJECT]/ingestion-pipeline:$VERSION and gcr.io/[YOUR_CLOUD_PROJECT]/deletion-pipeline:$VERSION respectively.

To generate the Flex Template Metadata files and upload them to GCS run:

The following commands require nodejs json npm install -g json Use the same container_registry_tag_prefix as in the builds above.

export VERSION=$(git describe --tags --always --dirty=-dirty)

json -f templates/dataflow-flex-template.json \
  -e "this.metadata=`cat templates/dataflow-ingestion-metadata-template.json`" \
  -e "this.image='gcr.io/[YOUR_CLOUD_PROJECT]/ingestion-pipeline:$VERSION'" > ingestion-pipeline-$VERSION.json

json -f templates/dataflow-flex-template.json \
  -e "this.metadata=`cat templates/dataflow-deletion-metadata-template.json`" \
  -e "this.image='gcr.io/[YOUR_CLOUD_PROJECT]/deletion-pipeline:$VERSION'" > deletion-pipeline-$VERSION.json

gsutil cp ingestion-pipeline-$VERSION.json $TEMPLATES
gsutil cp deletion-pipeline-$VERSION.json $TEMPLATES

gsutil -h "Content-Type:application/json" cp templates/scheduler-ingestion-template.tmpl \
  $TEMPLATES/scheduler-ingestion-template-$VERSION.tmpl
gsutil -h "Content-Type:application/json" cp templates/scheduler-deletion-template.tmpl \
  $TEMPLATES/scheduler-deletion-template-$VERSION.tmpl

unset VERSION

Contributing

Contributions to this repository are always welcome and highly encouraged.

See CONTRIBUTING for more information on how to get started.

License

Apache 2.0 - See LICENSE for more information.

This is not an official Google product