Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: How to use pre-commit-terraform image to run pre-commit in CI #656

Merged
merged 6 commits into from
Apr 30, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions .github/workflows/pre-commit.yaml
MaxymVlasov marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -54,3 +54,44 @@ jobs:
SKIP: check-added-large-files,check-merge-conflict,check-vcs-permalinks,forbid-new-submodules,no-commit-to-branch,end-of-file-fixer,trailing-whitespace,check-yaml,check-merge-conflict,check-executables-have-shebangs,check-case-conflict,mixed-line-ending,detect-aws-credentials,detect-private-key,shfmt,shellcheck
with:
extra_args: --color=always --show-diff-on-failure --files ${{ steps.file_changes.outputs.files }}

pre-commit-container:
runs-on: ubuntu-latest
container:
image: ghcr.io/antonbabenko/pre-commit-terraform:latest # latest used here for simplicity, not recommended
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- run: |
git config --global --add safe.directory $GITHUB_WORKSPACE
git fetch --no-tags --prune --depth=1 origin +refs/heads/*:refs/remotes/origin/*
stevie- marked this conversation as resolved.
Show resolved Hide resolved

- name: Get changed files
id: file_changes
run: |
export DIFF=$(git diff --name-only origin/${{ github.base_ref }} ${{ github.sha }})
echo "Diff between ${{ github.base_ref }} and ${{ github.sha }}"
echo "files=$( echo "$DIFF" | xargs echo )" >> $GITHUB_OUTPUT

- name: fix tar dependency in alpine container image
run: |
apk --no-cache add tar
# check python modules installed versions
python -m pip freeze --local

- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
fetch-depth: 0
ref: ${{ github.event.pull_request.head.sha }}

- name: Cache pre-commit since we use pre-commit from container
uses: actions/cache@v4
with:
path: ~/.cache/pre-commit
key: pre-commit-3|${{ hashFiles('.pre-commit-config.yaml') }}

- name: Execute pre-commit
run: |
pre-commit run --color=always --show-diff-on-failure --files ${{ steps.file_changes.outputs.files }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, before we move forward, I'd like to have the answer to the next question:

Would you like to add a fully functional replacement for the current GH workflow[1], or just show the possibility of docker image usage in GHA[2]?

Btw, [1] little bit related to #373

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[2] - show both ways seems to be a good idea to me.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[2] - show both ways seems to be a good idea to me.

I guess the .github/workflows/pre-commit.yaml is among active workflows in this repo and hence running the same thing twice might not be desired 🤔

Copy link
Collaborator

@MaxymVlasov MaxymVlasov Apr 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. If just show that it is possible, please:

  1. Update the Readme example with the fixed version from .github/workflows/pre-commit.yaml
  2. Revert changes in .github/workflows/pre-commit.yaml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would mean, that you don't actively check the current container image within GHA.

Copy link
Collaborator

@MaxymVlasov MaxymVlasov Apr 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to your answer to my question above, you not plan to provide a " fully functional replacement for the current GH workflow[1]"

And we definitely won't to decrease the current test coverage. Current realization in this PR will fail on hadolint, shfmt, and shellcheck hooks as there are no such dependencies inside Docker image. (and these checks are vital for .sh and .dockerfile)

And also, it will not push fixes back to branch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: The install of additional tools should also work in the container version.

144 changes: 93 additions & 51 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,41 +25,43 @@ If you are using `pre-commit-terraform` already or want to support its developme

## Table of content

* [Sponsors](#sponsors)
* [Table of content](#table-of-content)
* [How to install](#how-to-install)
* [1. Install dependencies](#1-install-dependencies)
* [2. Install the pre-commit hook globally](#2-install-the-pre-commit-hook-globally)
* [3. Add configs and hooks](#3-add-configs-and-hooks)
* [4. Run](#4-run)
* [Available Hooks](#available-hooks)
* [Hooks usage notes and examples](#hooks-usage-notes-and-examples)
* [Known limitations](#known-limitations)
* [All hooks: Usage of environment variables in `--args`](#all-hooks-usage-of-environment-variables-in---args)
* [All hooks: Set env vars inside hook at runtime](#all-hooks-set-env-vars-inside-hook-at-runtime)
* [All hooks: Disable color output](#all-hooks-disable-color-output)
* [All hooks: Log levels](#all-hooks-log-levels)
* [Many hooks: Parallelism](#many-hooks-parallelism)
* [checkov (deprecated) and terraform\_checkov](#checkov-deprecated-and-terraform_checkov)
* [infracost\_breakdown](#infracost_breakdown)
* [terraform\_docs](#terraform_docs)
* [terraform\_docs\_replace (deprecated)](#terraform_docs_replace-deprecated)
* [terraform\_fmt](#terraform_fmt)
* [terraform\_providers\_lock](#terraform_providers_lock)
* [terraform\_tflint](#terraform_tflint)
* [terraform\_tfsec (deprecated)](#terraform_tfsec-deprecated)
* [terraform\_trivy](#terraform_trivy)
* [terraform\_validate](#terraform_validate)
* [terraform\_wrapper\_module\_for\_each](#terraform_wrapper_module_for_each)
* [terrascan](#terrascan)
* [tfupdate](#tfupdate)
* [terragrunt\_providers\_lock](#terragrunt_providers_lock)
* [Docker Usage](#docker-usage)
* [File Permissions](#file-permissions)
* [Download Terraform modules from private GitHub repositories](#download-terraform-modules-from-private-github-repositories)
* [Authors](#authors)
* [License](#license)
* [Additional information for users from Russia and Belarus](#additional-information-for-users-from-russia-and-belarus)
- [Collection of git hooks for Terraform to be used with pre-commit framework](#collection-of-git-hooks-for-terraform-to-be-used-with-pre-commit-framework)
- [Sponsors](#sponsors)
stevie- marked this conversation as resolved.
Show resolved Hide resolved
- [Table of content](#table-of-content)
- [How to install](#how-to-install)
- [1. Install dependencies](#1-install-dependencies)
- [2. Install the pre-commit hook globally](#2-install-the-pre-commit-hook-globally)
- [3. Add configs and hooks](#3-add-configs-and-hooks)
- [4. Run](#4-run)
- [Available Hooks](#available-hooks)
- [Hooks usage notes and examples](#hooks-usage-notes-and-examples)
- [Known limitations](#known-limitations)
- [All hooks: Usage of environment variables in `--args`](#all-hooks-usage-of-environment-variables-in---args)
- [All hooks: Set env vars inside hook at runtime](#all-hooks-set-env-vars-inside-hook-at-runtime)
- [All hooks: Disable color output](#all-hooks-disable-color-output)
- [All hooks: Log levels](#all-hooks-log-levels)
- [Many hooks: Parallelism](#many-hooks-parallelism)
- [checkov (deprecated) and terraform\_checkov](#checkov-deprecated-and-terraform_checkov)
- [infracost\_breakdown](#infracost_breakdown)
- [terraform\_docs](#terraform_docs)
- [terraform\_docs\_replace (deprecated)](#terraform_docs_replace-deprecated)
- [terraform\_fmt](#terraform_fmt)
- [terraform\_providers\_lock](#terraform_providers_lock)
- [terraform\_tflint](#terraform_tflint)
- [terraform\_tfsec (deprecated)](#terraform_tfsec-deprecated)
- [terraform\_trivy](#terraform_trivy)
- [terraform\_validate](#terraform_validate)
- [terraform\_wrapper\_module\_for\_each](#terraform_wrapper_module_for_each)
- [terrascan](#terrascan)
- [tfupdate](#tfupdate)
- [terragrunt\_providers\_lock](#terragrunt_providers_lock)
- [Docker Usage](#docker-usage)
- [File Permissions](#file-permissions)
- [Download Terraform modules from private GitHub repositories](#download-terraform-modules-from-private-github-repositories)
- [Github Actions](#github-actions)
- [Authors](#authors)
- [License](#license)
- [Additional information for users from Russia and Belarus](#additional-information-for-users-from-russia-and-belarus)

## How to install

Expand Down Expand Up @@ -102,7 +104,7 @@ All available tags [here](https://github.com/antonbabenko/pre-commit-terraform/p
**Build from scratch**:

> [!IMPORTANT]
> To build image you need to have [`docker buildx`](https://docs.docker.com/build/install-buildx/) enabled as default builder.
> To build image you need to have [`docker buildx`](https://docs.docker.com/build/install-buildx/) enabled as default builder.
stevie- marked this conversation as resolved.
Show resolved Hide resolved
> Otherwise - provide `TARGETOS` and `TARGETARCH` as additional `--build-arg`'s to `docker build`.

When hooks-related `--build-arg`s are not specified, only the latest version of `pre-commit` and `terraform` will be installed.
Expand Down Expand Up @@ -207,7 +209,7 @@ Otherwise, you can follow [this gist](https://gist.github.com/etiennejeanneaurev

Ensure your PATH environment variable looks for `bash.exe` in `C:\Program Files\Git\bin` (the one present in `C:\Windows\System32\bash.exe` does not work with `pre-commit.exe`)

For `checkov`, you may need to also set your `PYTHONPATH` environment variable with the path to your Python modules.
For `checkov`, you may need to also set your `PYTHONPATH` environment variable with the path to your Python modules.
E.g. `C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\Lib\site-packages`

</details>
Expand Down Expand Up @@ -360,10 +362,10 @@ Less verbose log levels will be implemented in [#562](https://github.com/antonba

### Many hooks: Parallelism

> All, except deprecated hooks: `checkov`, `terraform_docs_replace` and hooks which can't be paralleled this way: `infracost_breakdown`, `terraform_wrapper_module_for_each`.
> All, except deprecated hooks: `checkov`, `terraform_docs_replace` and hooks which can't be paralleled this way: `infracost_breakdown`, `terraform_wrapper_module_for_each`.
> Also, there's a chance that parallelism have no effect on `terragrunt_fmt` and `terragrunt_validate` hooks

By default, parallelism is set to `number of logical CPUs - 1`.
By default, parallelism is set to `number of logical CPUs - 1`.
If you'd like to disable parallelism, set it to `1`

```yaml
Expand Down Expand Up @@ -419,7 +421,7 @@ args:
- --hook-config=--parallelism-ci-cpu-cores=N
```

If you don't see code above in your `pre-commit-config.yaml` or logs - you don't need it.
If you don't see code above in your `pre-commit-config.yaml` or logs - you don't need it.
`--parallelism-ci-cpu-cores` used only in edge cases and is ignored in other situations. Check out its usage in [hooks/_common.sh](hooks/_common.sh)

### checkov (deprecated) and terraform_checkov
Expand Down Expand Up @@ -567,7 +569,7 @@ Unlike most other hooks, this hook triggers once if there are any changed files
* create a documentation file
* extend existing documentation file by appending markers to the end of the file (see item 1 above)
* use different filename for the documentation (default is `README.md`)
* use the same insertion markers as `terraform-docs` by default. It will be default in `v2.0`.
* use the same insertion markers as `terraform-docs` by default. It will be default in `v2.0`.
To migrate to `terraform-docs` insertion markers, run in repo root:

```bash
Expand All @@ -592,7 +594,7 @@ Unlike most other hooks, this hook triggers once if there are any changed files
- --args=--config=.terraform-docs.yml
```

> **Warning**
> **Warning**
> Avoid use `recursive.enabled: true` in config file, that can cause unexpected behavior.

5. If you need some exotic settings, it can be done too. I.e. this one generates HCL files:
Expand Down Expand Up @@ -743,7 +745,7 @@ To replicate functionality in `terraform_docs` hook:

3. `terraform_providers_lock` support passing custom arguments to its `terraform init`:

> **Warning**
> **Warning**
> DEPRECATION NOTICE: This is available only in `no-mode` mode, which will be removed in v2.0. Please provide this keys to [`terraform_validate`](#terraform_validate) hook, which, to take effect, should be called before `terraform_providers_lock`

```yaml
Expand Down Expand Up @@ -924,10 +926,10 @@ To replicate functionality in `terraform_docs` hook:
- --hook-config=--retry-once-with-cleanup=true # Boolean. true or false
```

> **Important**
> **Important**
> The flag requires additional dependency to be installed: `jq`.

> **Note**
> **Note**
> Reinit can be very slow and require downloading data from remote Terraform registries, and not all of that downloaded data or meta-data is currently being cached by Terraform.

When `--retry-once-with-cleanup=true`, in each failed directory the cached modules and providers from the `.terraform` directory will be deleted, before retrying once more. To avoid unnecessary deletion of this directory, the cleanup and retry will only happen if Terraform produces any of the following error messages:
Expand All @@ -938,7 +940,7 @@ To replicate functionality in `terraform_docs` hook:
* "Module not installed"
* "Could not load plugin"

> **Warning**
> **Warning**
> When using `--retry-once-with-cleanup=true`, problematic `.terraform/modules/` and `.terraform/providers/` directories will be recursively deleted without prompting for consent. Other files and directories will not be affected, such as the `.terraform/environment` file.

**Option 2**
Expand All @@ -957,7 +959,7 @@ To replicate functionality in `terraform_docs` hook:

`terraform_validate` hook will try to reinitialize them before running the `terraform validate` command.

> **Caution**
> **Caution**
> If you use Terraform workspaces, DO NOT use this option ([details](https://github.com/antonbabenko/pre-commit-terraform/issues/203#issuecomment-918791847)). Consider the first option, or wait for [`force-init`](https://github.com/antonbabenko/pre-commit-terraform/issues/224) option implementation.

1. `terraform_validate` in a repo with Terraform module, written using Terraform 0.15+ and which uses provider `configuration_aliases` ([Provider Aliases Within Modules](https://www.terraform.io/language/modules/develop/providers#provider-aliases-within-modules)), errors out.
Expand Down Expand Up @@ -1009,7 +1011,7 @@ To replicate functionality in `terraform_docs` hook:
[...]
```

> **Tip**
> **Tip**
> The latter method will leave an "aliased-providers.tf.json" file in your repo. You will either want to automate a way to clean this up or add it to your `.gitignore` or both.

### terraform_wrapper_module_for_each
Expand All @@ -1033,8 +1035,8 @@ Sample configuration:
- --args=--verbose # Verbose output
```

**If you use hook inside Docker:**
The `terraform_wrapper_module_for_each` hook attempts to determine the module's short name to be inserted into the generated `README.md` files for the `source` URLs. Since the container uses a bind mount at a static location, it can cause this short name to be incorrect.
**If you use hook inside Docker:**
The `terraform_wrapper_module_for_each` hook attempts to determine the module's short name to be inserted into the generated `README.md` files for the `source` URLs. Since the container uses a bind mount at a static location, it can cause this short name to be incorrect.
If the generated name is incorrect, set them by providing the `module-repo-shortname` option to the hook:

```yaml
Expand Down Expand Up @@ -1084,7 +1086,7 @@ If the generated name is incorrect, set them by providing the `module-repo-short
- --args=--version 2.5.0 # Will be pined to specified version
```

Check [`tfupdate` usage instructions](https://github.com/minamijoyo/tfupdate#usage) for other available options and usage examples.
Check [`tfupdate` usage instructions](https://github.com/minamijoyo/tfupdate#usage) for other available options and usage examples.
No need to pass `--recursive .` as it is added automatically.

### terragrunt_providers_lock
Expand Down Expand Up @@ -1167,6 +1169,46 @@ Finally, you can execute `docker run` with an additional volume mount so that th
docker run --rm -e "USERID=$(id -u):$(id -g)" -v ~/.netrc:/root/.netrc -v $(pwd):/lint -w /lint ghcr.io/antonbabenko/pre-commit-terraform:latest run -a
```

## Github Actions

You can use this hook in your GitHub Actions workflow togehther with [pre-commit](https://pre-commit.com). To easy up dependency management, you can use the managed [docker image](#docker-usage) within your workflow. Make sure to set the image tag to the version you want to use.

In this repository's pre-commit [workflow file](.github/workflows/pre-commit.yml) we also check the container image with pre-commit.
Here is another more simple example which includes caching of pre-commit dependencies and uses the `pre-commit` command to run the checks.
Copy link
Collaborator

@MaxymVlasov MaxymVlasov Apr 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Here is another more simple example which includes caching of pre-commit dependencies and uses the `pre-commit` command to run the checks.
Here is another example that includes caching of pre-commit dependencies and uses the `pre-commit` command to run the checks (but fixes will not automatically push back to your branch, when it possible):

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MaxymVlasov : but fixes will be not automatically pushed back to your branch (not sure about the gist of when clause though)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see, where the original workflow pushes changes found by pre-commit? Or do you want to be just very specific?

Copy link
Collaborator

@MaxymVlasov MaxymVlasov Apr 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- name: Execute pre-commit
uses: pre-commit/action@9b88afc9cd57fd75b655d5c71bd38146d07135fe # v2.0.3
env:
SKIP: no-commit-to-branch,hadolint
with:
token: ${{ secrets.GITHUB_TOKEN }}
extra_args: --color=always --show-diff-on-failure --files ${{ steps.file_changes.outputs.files }}
# Run only skipped checks
- name: Execute pre-commit check that have no auto-fixes
if: always()
uses: pre-commit/action@9b88afc9cd57fd75b655d5c71bd38146d07135fe # v2.0.3
env:
SKIP: check-added-large-files,check-merge-conflict,check-vcs-permalinks,forbid-new-submodules,no-commit-to-branch,end-of-file-fixer,trailing-whitespace,check-yaml,check-merge-conflict,check-executables-have-shebangs,check-case-conflict,mixed-line-ending,detect-aws-credentials,detect-private-key,shfmt,shellcheck
with:
extra_args: --color=always --show-diff-on-failure --files ${{ steps.file_changes.outputs.files }}

There 2 pre-commit runs.
The first one, is able to push fixes. second-one - not. The difference in availability of GITHUB_TOKEN.

That's done in this way because otherwise, hooks without fixes will produce exit code 1, and that stops from pushing fixed to the branch.

I can say GHA "ignore exit code 1", and fixes will be successfully pushed, but then if something will be detected by hooks that have no autofixes - there will be no notification about that, except inside logs in "successfully done" GHA

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I missed, that you don't use the latest version of the pre-commit/action which has no push support anymore.

So, if the 2nd run doesn't push changes, wouldn't this be a good candidate to be replaced by the container version?

Copy link
Collaborator

@MaxymVlasov MaxymVlasov Apr 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still no hadolint dependency inside image. It will check nothing from pre-commit-hooks (all of them, which we use here have autofixes or at least do not break the first check when fail) nor pre-commit-terraform repo hooks (as this repo doesn't include any tf code for testing).

If you'd like to make useful tests, check #373.


```yaml
name: pre-commit-terraform

on:
pull_request:
push:
branches: [main]

jobs:
pre-commit:
runs-on: ubuntu-latest
container:
image: ghcr.io/antonbabenko/pre-commit-terraform:v1.88.4 # make sure to use the correct version
steps:
- uses: actions/checkout@v4
- name: fixes and dependencys
shell: bash
run: |
# fix permissions in the container
git config --global --add safe.directory $GITHUB_WORKSPACE
apk --no-cache add tar
# check python modules installed versions
python -m pip freeze --local
- uses: actions/cache@v4
with:
path: ~/.cache/pre-commit
key: pre-commit-3|${{ hashFiles('.pre-commit-config.yaml') }}
- shell: bash
run: |
pre-commit run --show-diff-on-failure --color=always -a || cat $HOME/.cache/pre-commit/pre-commit.log

```

## Authors

This repository is managed by [Anton Babenko](https://github.com/antonbabenko) with help from these awesome contributors:
Expand Down
Loading