Skip to content

Latest commit

 

History

History
168 lines (134 loc) · 5.03 KB

README.md

File metadata and controls

168 lines (134 loc) · 5.03 KB

CloudQuery Image Packages Source Plugin

test Go Report Card

An image packages source plugin for CloudQuery that loads data from S3 and Dynamo DB to any database, data warehouse or data lake supported by CloudQuery, such as PostgreSQL, BigQuery, Athena, and many more.

This plugin is currently used to collect data about image packages from the Amigo bake service.

It was created using the CloudQuery skeleton generator: cq-scaffold source guardian image-packages.

Links

Configuration

The following source configuration file will sync to a PostgreSQL database. See the CloudQuery Quickstart for more information on how to configure the source and destination.

kind: source
spec:
  name: "image-packages"
  path: "guardian/image-packages"
  version: "${VERSION}"
  destinations:
    - "postgresql"
  spec:
    # Name of S3 bucket holding Amigo bake package data
    bucket: ...
    # Name of Dynamo table holding bake data
    bakes_table: ...
    # Name of Dynamo table holding recipe data
    recipes_table: ...
    # Name of Dynamo table holding base image data
    base_images_table: ...

Development

Code structure

The entry point is main.go.

Package client contains the code to interact with the AWS services.

Package store contains the code to interact with the AWS data sources.

Package plugin contains the code to interact with the CloudQuery source plugin.

Package services contains the code to generate target tables.

Run tests

make test

Run linter

To do this, you will need to have golangci-lint installed.

make lint

Generate docs

make gen-docs

Debugging

To debug the plugin so that code can be stepped through with breakpoints, you will need:

  1. Delve debugger installed.
  2. If using Intellij, you will need a run/debug configuration set up. To set this up, add an Intellij run/debug configuration:
    1. Run > Edit Configurations... > + > Go Remote
    2. Fill in Host: localhost and Port: 7777
  3. A local spec file that specifies a local gRPC process as the plugin installed in a directory called local in this project root:
kind: source
spec:
  name: image-packages
  registry: grpc
  path: localhost:7777
  tables: ['amigo_bake_packages']
  destinations:
    - sqlite
  spec:
    base_images_table: <name of base images table>
    recipes_table: <name of recipes table>
    bakes_table: <name of bakes table>
    bucket: <name of packages bucket>
---
kind: destination
spec:
  name: sqlite
  path: cloudquery/sqlite
  registry: cloudquery
  version: v2.9.18
  spec:
    connection_string: local/db.sql

Then:

  1. In a terminal window, start up the plugin in debug mode:
make serve-debug
  1. Click on Debug in the Intellij run/debug configuration, or the green triangle in the Debug plugin in the Run and debug pane of Visual Studio.
  2. Insert breakpoints into the code where required.
  3. In another terminal window, run the CloudQuery sync command:
make run

Release a new version

Before releasing a new version, test the integration locally by building a local SQLite database following these steps:

  1. Build the binary:
make build
  1. Set up a local spec file with the required configuration:
kind: source
spec:
  name: image-packages
  registry: local
  path: ./cq-source-image-packages
  tables: ['amigo_bake_packages']
  destinations:
    - sqlite
  spec:
    base_images_table: <name of base images table>
    recipes_table: <name of recipes table>
    bakes_table: <name of bakes table>
    bucket: <name of packages bucket>
---
kind: destination
spec:
  name: sqlite
  path: cloudquery/sqlite
  registry: cloudquery
  version: v2.9.18
  spec:
    connection_string: local/db.sql
  1. Run the binary with the spec file:
make run
  1. Check the log and the output in the destination SQLite database.

Then, to release a new version:

  1. Run git tag v1.0.0 to create a new tag for the release (replace v1.0.0 with the new version number)
  2. Run git push origin v1.0.0 to push the tag to GitHub

Once the tag is pushed, a new GitHub Actions workflow will be triggered to build the release binaries.