Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google backend #93

Open
wants to merge 33 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
4f2fe67
Use gcp credentials
shirkevich Nov 19, 2024
62b7de8
Fix parsing of gcp-project-id
shirkevich Nov 19, 2024
65a8d29
Add credentials file for plan when running in google
shirkevich Nov 19, 2024
75e616b
Do not run aws tasks when backend is google
shirkevich Nov 19, 2024
0d8bdc0
Fix naming for firestore
shirkevich Nov 19, 2024
e9e07b3
Do not cache terraform-plan repo
shirkevich Nov 19, 2024
4cf94d3
Specify latest version via gitsha
shirkevich Nov 19, 2024
5e573e0
Specify latest version via full gitsha
shirkevich Nov 19, 2024
f6a2781
Bump latest sha
shirkevich Nov 19, 2024
c4ddd3c
Bump sha
shirkevich Nov 19, 2024
59f12cc
Bumping to latest
shirkevich Nov 19, 2024
9022d6d
Pass proper bucket name
shirkevich Nov 19, 2024
fc2782f
Bumping sha again
shirkevich Nov 19, 2024
a24f5b6
Trying to usr branch again
shirkevich Nov 19, 2024
beaa6c8
Use github oidc instead of credentials file
shirkevich Nov 19, 2024
d12ca5e
Fix condition for google backend
shirkevich Nov 19, 2024
44905e4
Fixing whitespaces
shirkevich Nov 19, 2024
3f6f1f3
Debugging action steps
shirkevich Nov 19, 2024
37789b9
Fix condition for google auth
shirkevich Nov 19, 2024
1860e7e
Specify firestore databaseId
shirkevich Nov 19, 2024
ade96d3
Fix google params naming
shirkevich Nov 20, 2024
2bd6eb0
Fix readme
shirkevich Nov 20, 2024
e8ea056
Don not pass values if they are nulls, to use defaults
shirkevich Nov 28, 2024
32b5821
Always passing Database and Collection name
shirkevich Nov 28, 2024
4840999
Merge branch 'main' into google-backend
shirkevich Dec 4, 2024
bcaf758
passing service account and workload identity provider from config
shirkevich Dec 5, 2024
a69fb20
Installing opentofu before get settings
shirkevich Dec 5, 2024
2a842cc
Fix gcp params for lockfile creation
shirkevich Dec 6, 2024
6d394be
Commenting tofu installation to check the error
shirkevich Dec 6, 2024
baddb26
Get settings now working without tofu installation
shirkevich Dec 6, 2024
0c3d187
Install terraform and opentofu before get all settings
shirkevich Dec 19, 2024
54db930
Merge branch 'main' into google-backend
shirkevich Dec 19, 2024
5527177
Do not get opentofu/terraform versions again
shirkevich Dec 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 28 additions & 7 deletions README.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,20 +44,24 @@ references:
usage: |-
### Prerequisites

This GitHub Action requires AWS access for two different purposes. This action will attempt to first run `terraform plan` against a given component and
then will use another role to save that given Terraform Plan to an S3 Bucket with metadata in a DynamoDB table. We recommend configuring
[OpenID Connect with AWS](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services)
to allow GitHub to assume roles in AWS and then deploying both a Terraform Plan role and a Terraform State role.
For Cloud Posse documentation on setting up GitHub OIDC, see our [`github-oidc-provider` component](https://docs.cloudposse.com/components/library/aws/github-oidc-provider/).
This GitHub Action requires cloud provider access for two different purposes. This action will attempt to first run `terraform plan` against a given component and
then will use credentials to save that given Terraform Plan to cloud storage with metadata.

In order to store Terraform State, we configure an S3 Bucket to store plan files and a DynamoDB table to track plan metadata. Both will need to be deployed before running
this action. For more on setting up those components, see the `gitops` component (__documentation pending__). This action will then use the [github-action-terraform-plan-storage](https://github.com/cloudposse/github-action-terraform-plan-storage) action to update these resources.
For AWS, we recommend configuring [OpenID Connect with AWS](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services)
to allow GitHub to assume roles in AWS and then deploying both a Terraform Plan role and a Terraform State role.
For Cloud Posse documentation on setting up GitHub OIDC with AWS, see our [`github-oidc-provider` component](https://docs.cloudposse.com/components/library/aws/github-oidc-provider/).

For Google Cloud, we recommend configuring [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation-with-GitHub-actions)
to allow GitHub Actions to authenticate with Google Cloud services. This will enable access to Google Cloud Storage for plan files and Firestore for metadata storage.

For AWS storage, we configure an S3 Bucket to store plan files and a DynamoDB table to track plan metadata. For Google Cloud storage, we use Google Cloud Storage buckets for plan files and Firestore for metadata. These resources will need to be deployed before running this action. For more on setting up those components, see the `gitops` component (__documentation pending__). This action will then use the [github-action-terraform-plan-storage](https://github.com/cloudposse/github-action-terraform-plan-storage) action to update these resources.

### Config

The action expects the atmos configuration file `atmos.yaml` to be present in the repository.
The config should have the following structure:

#### AWS
```yaml
integrations:
github:
Expand All @@ -77,6 +81,23 @@ usage: |-
sort-by: .stack_slug
group-by: .stack_slug | split("-") | [.[0], .[2]] | join("-")
```

#### Google Cloud
```yaml
integrations:
github:
gitops:
...
artifact-storage:
...
bucket: cptest-core-ue2-auto-gitops
google-service-account: [email protected]
google-workload-identity-provider: projects/project-id/locations/global/workloadIdentityPools/github-actions/providers/github-provider
google-project-id: cptest-core-ue2-auto-gitops
google-firestore-database-name: cptest-core-ue2-auto-gitops
google-firestore-collection-name: terraform-plan-storage

```

> [!IMPORTANT]
> **Please note!** This GitHub Action only works with `atmos >= 1.63.0`. If you are using `atmos < 1.63.0` please use `v1` version of this action.
Expand Down
67 changes: 59 additions & 8 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,15 @@ inputs:
not supplied by the user. When running this action on github.com, the default value is sufficient. When running on
GHES, you can pass a personal access token for github.com if you are experiencing rate limiting.
default: ${{ github.server_url == 'https://github.com' && github.token || '' }}

outputs:
summary:
description: "Summary"
value: "${{ steps.summary.outputs.result }}"

env:
ACTIONS_STEP_DEBUG: true

shirkevich marked this conversation as resolved.
Show resolved Hide resolved
runs:
using: "composite"
steps:
Expand Down Expand Up @@ -83,7 +87,13 @@ runs:
echo "opentofu-version=$(atmos describe config -f json | jq -r '.integrations.github.gitops["opentofu-version"]')" >> $GITHUB_OUTPUT
echo "terraform-version=$(atmos describe config -f json | jq -r '.integrations.github.gitops["terraform-version"]')" >> $GITHUB_OUTPUT
echo "enable-infracost=$(atmos describe config -f json | jq -r '.integrations.github.gitops["infracost-enabled"]')" >> $GITHUB_OUTPUT
echo "backend=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].backend')" >> $GITHUB_OUTPUT
echo "aws-region=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].region')" >> $GITHUB_OUTPUT
echo "google-project-id=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"]."google-project-id"')" >> $GITHUB_OUTPUT
echo "google-workload-identity-provider=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"]."google-workload-identity-provider"')" >> $GITHUB_OUTPUT
echo "google-service-account=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"]."google-service-account"')" >> $GITHUB_OUTPUT
echo "google-firestore-database-name=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"]."google-firestore-database-name"')" >> $GITHUB_OUTPUT
echo "google-firestore-collection-name=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"]."google-firestore-collection-name"')" >> $GITHUB_OUTPUT
echo "terraform-state-role=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].role')" >> $GITHUB_OUTPUT
echo "terraform-state-table=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].table')" >> $GITHUB_OUTPUT
echo "terraform-state-bucket=$(atmos describe config -f json | jq -r '.integrations.github.gitops["artifact-storage"].bucket')" >> $GITHUB_OUTPUT
Expand All @@ -101,20 +111,28 @@ runs:
with:
cache: true
config: |-
opentofu/opentofu:
opentofu/opentofu:
tag: ${{ startsWith(steps.config.outputs.opentofu-version, 'v') && steps.config.outputs.opentofu-version || format('v{0}', steps.config.outputs.opentofu-version) }}
skip: ${{ steps.config.outputs.opentofu-version == '' || steps.config.outputs.opentofu-version == 'null' }}
suzuki-shunsuke/tfcmt:
tag: v4.11.0
tag: v4.11.0

- name: Configure Plan AWS Credentials
if: ${{ steps.config.outputs.backend == 'aws' }}
uses: aws-actions/[email protected]
with:
aws-region: ${{ steps.config.outputs.aws-region }}
role-to-assume: ${{ steps.config.outputs.terraform-plan-role }}
role-session-name: "atmos-terraform-plan-gitops"
mask-aws-account-id: "no"

- name: Configure Google Credentials
if: ${{ steps.config.outputs.backend == 'google' }}
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ steps.config.outputs.google-workload-identity-provider }}
service_account: ${{ steps.config.outputs.google-service-account }}

- name: Get atmos settings
uses: cloudposse/github-action-atmos-get-setting@v1
id: component
Expand Down Expand Up @@ -265,18 +283,18 @@ runs:
rm -f ${TERRAFORM_OUTPUT_FILE}

- name: Configure State AWS Credentials
if: ${{ steps.atmos-plan.outputs.error == 'false' }}
if: ${{ steps.atmos-plan.outputs.error == 'false' && steps.config.outputs.backend == 'aws' }}
uses: aws-actions/[email protected]
with:
aws-region: ${{ steps.config.outputs.aws-region }}
role-to-assume: ${{ steps.config.outputs.terraform-state-role }}
role-session-name: "atmos-terraform-state-gitops"
mask-aws-account-id: "no"

- name: Store New Plan
if: ${{ steps.atmos-plan.outputs.error == 'false' }}
- name: Store New Plan (AWS)
if: ${{ steps.atmos-plan.outputs.error == 'false' && steps.config.outputs.backend == 'aws' }}
uses: cloudposse/github-action-terraform-plan-storage@v1
id: store-plan
id: store-plan-aws
with:
action: storePlan
commitSHA: ${{ inputs.sha }}
Expand All @@ -286,8 +304,25 @@ runs:
tableName: ${{ steps.config.outputs.terraform-state-table }}
bucketName: ${{ steps.config.outputs.terraform-state-bucket }}

- name: Store Lockfile for New Plan
if: ${{ steps.atmos-plan.outputs.error == 'false' }}
- name: Store New Plan (Google)
if: ${{ steps.atmos-plan.outputs.error == 'false' && steps.config.outputs.backend == 'google' }}
uses: shirkevich/github-action-terraform-plan-storage@google-cloud-backend
id: store-plan-google
with:
action: storePlan
commitSHA: ${{ inputs.sha }}
planPath: ${{ steps.vars.outputs.plan_file }}
component: ${{ inputs.component }}
stack: ${{ inputs.stack }}
planRepositoryType: gcs
metadataRepositoryType: firestore
bucketName: ${{ steps.config.outputs.terraform-state-bucket }}
gcpProjectId: ${{ steps.config.outputs.google-project-id }}
gcpFirestoreDatabaseName: ${{ steps.config.outputs.google-firestore-database-name }}
gcpFirestoreCollectionName: ${{ steps.config.outputs.google-firestore-collection-name }}

- name: Store Lockfile for New Plan (AWS)
if: ${{ steps.atmos-plan.outputs.error == 'false' && steps.config.outputs.backend == 'aws' }}
uses: cloudposse/github-action-terraform-plan-storage@v1
with:
action: storePlan
Expand All @@ -298,6 +333,22 @@ runs:
tableName: ${{ steps.config.outputs.terraform-state-table }}
bucketName: ${{ steps.config.outputs.terraform-state-bucket }}

- name: Store Lockfile for New Plan (Google)
if: ${{ steps.atmos-plan.outputs.error == 'false' && steps.config.outputs.backend == 'google' }}
uses: shirkevich/github-action-terraform-plan-storage@google-cloud-backend
with:
action: storePlan
commitSHA: ${{ inputs.sha }}
planPath: ${{ steps.vars.outputs.lock_file }}
component: ${{ inputs.component }}
stack: ${{ inputs.stack }}-lockfile
planRepositoryType: gcs
metadataRepositoryType: firestore
bucketName: ${{ steps.config.outputs.terraform-state-bucket }}
gcpProjectId: ${{ steps.config.outputs.google-project-id }}
gcpFirestoreDatabaseName: ${{ steps.config.outputs.google-firestore-database-name }}
gcpFirestoreCollectionName: ${{ steps.config.outputs.google-firestore-collection-name }}

- name: Setup Infracost
if: ${{ steps.config.outputs.enable-infracost == 'true' && steps.atmos-plan.outputs.changes == 'true' }}
uses: infracost/actions/setup@v3
Expand Down