Skip to content

Latest commit

 

History

History
176 lines (150 loc) · 5.97 KB

README.md

File metadata and controls

176 lines (150 loc) · 5.97 KB

Destination Tester

Pre-requisites

How To Run

  1. Pull the latest docker image from public-docker-us/sdktesters/sdk-tester Google Artifact Registry, use the following commands:

    • Authenticate Docker to Google Artifact Registry: Run the following command to allow Docker to use your Google credentials
        gcloud auth configure-docker us-docker.pkg.dev
    • Pull the Image:
        docker pull us-docker.pkg.dev/build-286712/public-docker-us/sdktesters/sdk-tester

NOTE: If using V2 proto versions, use the latest docker image of the public-docker-us/sdktesters-v2/sdk-tester artifact in Google Artifact Registry.

  1. Run a container using the image with the following command. Make sure to map a local directory for the tool by replacing <local-data-folder> placeholders in the command, and replace <version> with the version of the image you pulled.
docker run --mount type=bind,source=<local-data-folder>,target=/data -a STDIN -a STDOUT -a STDERR -it -e WORKING_DIR=<local-data-folder> -e GRPC_HOSTNAME=host.docker.internal --network=host fivetrandocker/fivetran-sdk-tester:<version>  --tester-type destination --port <port>
  1. To rerun the container from step #2, use the following command:
docker start -i <container-id>

Input Files

Destination tester simulates operations from a source by reading input files from the local data folder. Each input file represents a batch of operations, encoded in JSON format. Data types in common.proto file can be used as column data types.

List of Operations

Table Operations

  • describe_table
  • create_table
  • alter_table

Single Record Operations

  • upsert
  • update
  • delete
  • soft_delete

Bulk Record Operations

  • truncate_before
  • soft_truncate_before

Example input file

Here is an example input file named input_1.json:

{
    "create_table" : {
        "transaction": {
            "columns": {
                "id": "INT",
                "amount" : "DOUBLE",
                "desc": "STRING"
            },
            "primary_key": ["id"]
        },
        "campaign": {
            "columns": {
                "name": "STRING",
                "num": {"type": "DECIMAL", "precision": 6, "scale": 3}
            },
            "primary_key": []
        }
    },
    "alter_table" : {
        "transaction": {
            "columns": {
                "id": "INT",
                "amount" : "FLOAT",
                "desc": "STRING"
            },
            "primary_key": ["id"]
        }
    },
    "describe_table" : [
        "transaction"
    ],
    "ops" : [
        {
            "upsert": {
                "transaction": [
                    {"id":1, "amount": 100.45, "desc": null},
                    {"id":2, "amount": 150.33, "desc": "two"}
                ],
                "campaign": [
                    {"_fivetran_id": "abc-123-xyz", "name": "Christmas", "num": 100.23},
                    {"_fivetran_id": "vbn-543-hjk", "name": "New Year", "num": 200.56}
                ]
            }
        },
        {
            "truncate_before": [
                "campaign"
            ]
        },
        {
            "update": {
                "transaction": [
                    {"id":1, "amount": 200}
                ]
            }
        },
        {
            "soft_truncate_before": [
                "transaction"
            ]
        },
        {
            "upsert": {
                "transaction": [
                    {"id":10, "amount": 100, "desc": "thee"},
                    {"id":20, "amount": 50, "desc": "mone"}
                ],
                "campaign": [
                    {"_fivetran_id": "dfg-890-lkj", "name": "Christmas 2", "num": 400.32}
                ]
            }
        },
        {
            "delete": {
                "transaction": [
                    {"id":3}
                ],
                "campaign": [
                    {"_fivetran_id": "abc-123-xyz"}
                ]
            }
        },
        {
            "soft_delete": {
                "transaction": [
                    {"id":4}
                ],
                "campaign": [
                    {"_fivetran_id": "dfg-890-lkj"}
                ]
            }
        }
    ]
}

CLI Arguments

The tester supports the following optional CLI arguments to alter its default behavior. You can append these options to the end of the docker run command provided in step 2 of How To Run section above.

--port

This option defines the port the tester should run on.

--plain-text

This option disables encryption and compression of batch files for debugging purposes.

--input-file

The tester by default reads all input files from local data folder and executes them in the alphabetical order they appear. You can specify a single input file to be read and executed using this option. Providing just the filename is sufficient.

--schema-name

The tester by default creates a schema named tester. This option allows the tester to run with a custom schema name by specifying --schema-name <custom_schema_name> where <custom_schema_name> is your custom schema name.

--disable-operation-delay

The tester by default adds a delay to operation for real-time simulation. Specifying this argument disables the delay.

--batch-file-type

We now support both CSV and PARQUET batch files. If this argument is not provided, testers will generate CSV batch files.