App Project Migrate

Application which orchestrates whole process of Keboola Connection project migration from one stack to another.

Prerequisites:

• Source project which will be migrated
• Destination project - empty project where the source project will be cloned

Application is executed in destination project and requires Storage API token and Keboola Connection URL of source project. Admin token of source project is required for GoodData writers migration. Source project is left without any changes.

Migration steps performed by the application:

• Create snapshot of source project https://github.com/keboola/app-project-backup
• Restore project from snapshot https://github.com/keboola/app-project-restore
• Migrate GoodData writers https://github.com/keboola/app-gooddata-writer-migrate
• Migrate Snowflake writers https://github.com/keboola/app-snowflake-writer-migrate
• Migrate Orchestrators https://github.com/keboola/app-orchestrator-migrate

Usage

It is recommended to run migration validation application in the source project before migration.

Run the migration in destination project wil the following command. This is example of project migration from US to EU, please replace these parameters:

• DEST_PROJECT_SAPI_TOKEN - Storage API token associated to admin of destination EU project (master)
• SOURCE_PROJECT_SAPI_TOKEN - Storage API token associated to admin of source US project (non-master, all permissions are required)
• SOURCE_MANAGE_API_TOKEN - Manage API token with super admin rights. Must be from source stack. Required if parameter migrateSecrets is true.

Queue v2

curl -X POST \
--location 'https://queue.eu-central-1.keboola.com/jobs' \
--header 'X-StorageApi-Token: DEST_PROJECT_SAPI_TOKEN' \
--header 'Content-Type: application/json' \
--data '{
  "component": "keboola.app-project-migrate",
  "mode": "run",
  "configData": {
    "parameters": {
      "sourceKbcUrl": "https://connection.keboola.com",
      "#sourceKbcToken": "SOURCE_PROJECT_SAPI_TOKEN",
      "dryRun": false,
      "directDataMigration": true,
      "migrateSecrets": false,
      "#sourceManageToken": "SOURCE_MANAGE_API_TOKEN"
    }
  }
}'

Queue v1 (deprecated)

curl -X POST \
 https://docker-runner.eu-central-1.keboola.com/docker/keboola.app-project-migrate/run \
 -H 'X-StorageApi-Token: DEST_PROJECT_SAPI_TOKEN' \
 -d '{"configData": {"parameters": {"sourceKbcUrl": "https://connection.keboola.com", "#sourceKbcToken":"SOURCE_PROJECT_SAPI_TOKEN"}}}'

Dry-run mode

If you want to save some time and check that everything is set correctly, you can use the dry-run mode. Just set configData.parameters.dryRun on true in your request payload.

What is not executed during dry-run mode?

Project restore

• add project metadata into destination project
• add restored configurations, its rows, metadata, state and row order into destination project
• create buckets and its metadata in destination project
• create tables, table aliases and metadata in destination project

Migrate configurations (via encryption-api)

• add migrated configurations, its rows, metadata, state and row order into destination project

Migrate Snowflake writers

• create workspace for destination project (Keboola-provisioned writers)
• add migrated configurations and its rows

Migrate tables data

• default (API) mode:
  • file upload into destination project
  • write data into destination tables
• database mode:
  • replicate tables

Development

Clone this repository and init the workspace with following command:

git clone https://github.com/keboola/my-component
cd my-component
docker-compose build
docker-compose run --rm dev composer install --no-scripts

Run the test suite using this command:

docker-compose run --rm dev composer tests

Integration

For information about deployment and integration with KBC, please refer to the deployment section of developers documentation

License

MIT licensed, see LICENSE file.