v0.16.0

Add Spacelift module and example. Bump `terraform-provider-utils` versions @aknysh (#27)

what

• Add Spacelift module and example
• Bump terraform-provider-utils version
• Add process_component_deps variable

why

• Directly convert infrastructure YAML stack configs into Spacelift stack configs

• Speed up YAML stack processing (in many cases, from tens of minutes to tens of seconds)

• Reduce unnecessary stack runs in CI/CD systems like Spacelift

• The provider already calculates these parameters:

  • imports - a list of all imports (all levels) for the stack
  • stacks - a list of all stacks in the infrastructure where the component is defined

• Both imports and stacks are not 100% suitable to correctly determine the YAML config files that a component depends on

  • imports is too broad. The provider returns all direct and indirect imports for the stack, even those that don't define any variables for the component. This will trigger the component's stack in Spacelift even if the unrelated imports are modified resulting unrelated stack runs in Spacelift
  • stacks is too broad and too narrow at the same time. On the one hand, it detects all stacks in the infrastructure where the component is defined, but we don't need to trigger a particular stack in Spacelift if some other top-level YAML stack configs are modified. On the other hand, it misses the cases where a YAML stack config file specifies global variables, which are applied to the component as well

For example, in examples/data-sources/utils_stack_config_yaml/stacks/catalog/eks-defaults.yaml we define eks-iam terraform component. The provider returns the following:

stacks:
  - catalog/eks-defaults
  - globals
  - uw2-dev
  - uw2-globals
  - uw2-prod
  - uw2-staging
  - uw2-uat

imports:
  - catalog/eks-defaults
  - catalog/rds-defaults
  - catalog/s3-defaults
  - globals
  - uw2-globals

imports returns the unrelated catalog/s3-defaults and catalog/rds-defaults which will unnecessary trigger this component stack in Spacelift.

stacks returns all top-level YAML stack configs in the infrastructure all of which will unnecessary trigger this component stack in Spacelift.

We need to return only those YAML config files if any of the following conditions is true:

1. The imported config file has the global vars section and it's not empty
2. The imported config file has the component type section, which has a vars section which is not empty
3. The imported config file has the "components" section, which has the component type section, which has the component section
4. The imported config file has the "components" section, which has the component type section, which has the base component section

For example:

vars:
  stage: dev
terraform:
  vars: {}
components:
  terraform:
    aurora-postgres:
      vars:
        instance_type: db.r4.large
        cluster_size: 1
    aurora-postgres-2:
      component: aurora-postgres
      vars:
        instance_type: db.r4.xlarge
  helmfile: {}

vars and terraform.vars are global vars and component-type vars sections
components is the components section
components.terraform is the component type section (as is components.helmfile) - those are component types and get processed similarly
components.terraform.aurora-postgres is the base component section
components.terraform.aurora-postgres-2 is the component section

The changes in this PR return the correct YAML config dependencies for the components, e.g. for eks-iam:

eks-iam:
  deps:
    - catalog/eks-defaults
    - globals
    - uw2-globals
    - uw2-dev

because globals and uw2-globals have global variables for the component, and catalog/eks-defaults defines the component itself.

This will trigger the Spacelift component stack only if these files are modified (in addition to the component project files and the top-level stack, e.g. uw2-prod, themselves).

references

• cloudposse/terraform-provider-utils#32
• cloudposse/terraform-provider-utils#40