run-all --terragrunt-include-dir <directory> processing terragrunt.hcl in all directories, expected this to only target terragrunt.hcl in the targeted directory

[atheiman]

I have a project structure for hundreds of aws accounts like this:

terragrunt_root/
  root.hcl
  account-one/
    terragrunt.hcl
    main.tf
  account-two/
    terragrunt.hcl
    main.tf
  ...

Each account dir is >90% the same. Customizations to the account are handled by the terraform modules run in each account using tags on accounts and other data sources to do special actions in each account. terragrunt.hcl is copied into each account with an account directory generator script, and the entire contents is below:

# Include terragrunt_root/root.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
}

This setup works great for us. From the root directory (terragrunt_root/) I can plan / apply a handful of accounts or all accounts with terragrunt run-all ... optionally with --terragrunt-include-dir account-one.

However, I have noticed that run-all commands targeting a small number of accounts still processes terragrunt.hcl in accounts that are not targeted. This doesnt break anything, but it does make the command run for an unnecessarily long time. We have hundreds of account directories, and just processing terragrunt.hcl in each directory takes several minutes on my Windows desktop. I have added example run-all plan ... command output with --terragrunt-log-level debug output below showing the repeated processing of terragrunt.hcl in every directory. I did have to replace a lot of strings in the output for sensitivity, but the problem should be clear - terragrunt run-all ... spends a few minutes processing terragrunt.hcl in every directory under terragrunt_root/ although I only targeted a single directory. I realize for a single account I could use --terragrunt-working-dir account-one, but I often plan a handful of accounts with run-all.

Is processing every terragrunt.hcl in every subdirectory related to terragrunt dependency checking? I dont use terragrunt dependencies, so maybe I can disable that? Is there a way to prevent terragrunt from processing terragrunt.hcl in directories I did not target?

$ terragrunt run-all plan --terragrunt-include-dir account-one --terragrunt-log-level debug
08:31:04.025 DEBUG  [account-one] Did not find any locals block: skipping evaluation.
08:31:04.055 DEBUG  [account-one] Found locals block: evaluating the expressions.
08:31:04.058 DEBUG  [account-one] Running command: git rev-parse --show-toplevel
08:31:04.058 DEBUG  [account-one] Command output will be suppressed.
08:31:04.549 DEBUG  [account-one] git show-toplevel result:
C:/Users/austin/code/aws-infrastructure
08:31:04.565 DEBUG  [account-one] Evaluated 8 locals (remaining 1): local_one, local_two, ...
08:31:04.576 DEBUG  [account-one] Evaluated 1 locals (remaining 0): account_data
08:31:04.702 DEBUG  [account-one] [Partial] Included config C:/Users/austin/code/aws-infrastructure/terragrunt_root/root.hcl has strategy shallow merge: merging config in (shallow).
08:31:04.702 DEBUG  [account-one] Setting download directory for module .\account-one to C:/Users/austin/code/aws-infrastructure/terragrunt_root/account-one/.terragrunt-cache

08:31:05.025 DEBUG  [account-two] Did not find any locals block: skipping evaluation.
08:31:05.055 DEBUG  [account-two] Found locals block: evaluating the expressions.
08:31:05.058 DEBUG  [account-two] Running command: git rev-parse --show-toplevel
08:31:05.058 DEBUG  [account-two] Command output will be suppressed.
08:31:05.549 DEBUG  [account-two] git show-toplevel result:
C:/Users/austin/code/aws-infrastructure
08:31:05.565 DEBUG  [account-two] Evaluated 8 locals (remaining 1): local_one, local_two, ...
08:31:05.576 DEBUG  [account-two] Evaluated 1 locals (remaining 0): account_data
08:31:05.702 DEBUG  [account-two] [Partial] Included config C:/Users/austin/code/aws-infrastructure/terragrunt_root/root.hcl has strategy shallow merge: merging config in (shallow).
08:31:05.702 DEBUG  [account-two] Setting download directory for module .\account-two to C:/Users/austin/code/aws-infrastructure/terragrunt_root/account-two/.terragrunt-cache

08:31:06.025 DEBUG  [account-three] Did not find any locals block: skipping evaluation.
08:31:06.055 DEBUG  [account-three] Found locals block: evaluating the expressions.
08:31:06.058 DEBUG  [account-three] Running command: git rev-parse --show-toplevel
08:31:06.058 DEBUG  [account-three] Command output will be suppressed.
08:31:06.549 DEBUG  [account-three] git show-toplevel result:
C:/Users/austin/code/aws-infrastructure
08:31:06.565 DEBUG  [account-three] Evaluated 8 locals (remaining 1): local_one, local_two, ...
08:31:06.576 DEBUG  [account-three] Evaluated 1 locals (remaining 0): account_data
08:31:06.702 DEBUG  [account-three] [Partial] Included config C:/Users/austin/code/aws-infrastructure/terragrunt_root/root.hcl has strategy shallow merge: merging config in (shallow).
08:31:06.702 DEBUG  [account-three] Setting download directory for module .\account-three to C:/Users/austin/code/aws-infrastructure/terragrunt_root/account-three/.terragrunt-cache

[[[... REPEATED FOR EVERY ACCOUNT DIRECTORY ...]]]

08:31:40.273 DEBUG  [.] Stack at C:/Users/austin/code/aws-infrastructure/terragrunt_root:
  => Module C:/Users/austin/code/aws-infrastructure/terragrunt_root/account-one (excluded: true, assume applied: false, dependencies: [])
  => Module C:/Users/austin/code/aws-infrastructure/terragrunt_root/account-two (excluded: true, assume applied: false, dependencies: [])
  => Module C:/Users/austin/code/aws-infrastructure/terragrunt_root/account-three (excluded: true, assume applied: false, dependencies: [])
  [[[... REPEATED FOR EVERY ACCOUNT DIRECTORY ...]]]

[[[NOW WE ARE RUNNING THE PLAN FOR THE ONLY TARGETED ACCOUNT]]]

08:31:40.317 INFO   [.] The stack at C:/Users/austin/code/aws-infrastructure/terragrunt_root will be processed in the following order for command plan:
Group 1
- Module C:/Users/austin/code/aws-infrastructure/terragrunt_root/account-one

08:31:40.320 DEBUG  [account-one] Module C:/Users/austin/code/aws-infrastructure/terragrunt_root/account-one must wait for 0 dependencies to finish
08:31:40.322 DEBUG  [account-one] Running C:/Users/austin/code/aws-infrastructure/terragrunt_root/account-one
08:31:40.325 DEBUG  [account-one] Did not find any locals block: skipping evaluation.
08:31:40.367 DEBUG  [account-one] Found locals block: evaluating the expressions.
08:31:40.374 DEBUG  [account-one] Running command: git rev-parse --show-toplevel
08:31:40.374 DEBUG  [account-one] Command output will be suppressed.
08:31:40.880 DEBUG  [account-one] git show-toplevel result:
C:/Users/austin/code/aws-infrastructure
08:31:40.892 DEBUG  [account-one] Evaluated 8 locals (remaining 1): local_one, local_two, ...
08:31:40.904 DEBUG  [account-one] Evaluated 1 locals (remaining 0): account_data
08:31:40.961 DEBUG  [account-one] [Partial] Included config C:/Users/austin/code/aws-infrastructure/terragrunt_root/root.hcl has strategy shallow merge: merging config in (shallow).
08:31:40.964 DEBUG  [account-one] Running command: terraform --version
08:31:40.967 DEBUG  [account-one] Engine is not enabled, running command directly in .\account-one
08:31:43.087 DEBUG  [account-one] terraform version: 1.10.2
08:31:43.087 DEBUG  [account-one] Reading Terragrunt config file at C:/Users/austin/code/aws-infrastructure/terragrunt_root/account-one/terragrunt.hcl
08:31:43.092 DEBUG  [account-one] Did not find any locals block: skipping evaluation.
08:31:43.123 DEBUG  [account-one] Found locals block: evaluating the expressions.
08:31:43.130 DEBUG  [account-one] Running command: git rev-parse --show-toplevel
08:31:43.131 DEBUG  [account-one] Command output will be suppressed.
08:31:43.608 DEBUG  [account-one] git show-toplevel result:
C:/Users/austin/code/aws-infrastructure
08:31:43.621 DEBUG  [account-one] Evaluated 8 locals (remaining 1): local_one, local_two, ...
08:31:43.632 DEBUG  [account-one] Evaluated 1 locals (remaining 0): account_data
08:31:43.673 DEBUG  [account-one] [Partial] Included config C:/Users/austin/code/aws-infrastructure/terragrunt_root/root.hcl has strategy shallow merge: merging config in (shallow).
08:31:43.673 DEBUG  [account-one] Did not find any locals block: skipping evaluation.
08:31:43.706 DEBUG  [account-one] Found locals block: evaluating the expressions.
08:31:43.723 DEBUG  [account-one] Evaluated 8 locals (remaining 1): local_one, local_two
08:31:43.734 DEBUG  [account-one] Evaluated 1 locals (remaining 0): account_data
08:31:43.800 DEBUG  [account-one] Included config C:/Users/austin/code/aws-infrastructure/terragrunt_root/root.hcl has strategy shallow merge: merging config in (shallow) for dependency.
08:31:43.842 DEBUG  [account-one] Found locals block: evaluating the expressions.
08:31:43.857 DEBUG  [account-one] Evaluated 8 locals (remaining 1): local_one, local_two, ...
08:31:43.869 DEBUG  [account-one] Evaluated 1 locals (remaining 0): account_data
08:31:43.929 DEBUG  [account-one] Found locals block: evaluating the expressions.
08:31:43.947 DEBUG  [account-one] Evaluated 8 locals (remaining 1): local_one, local_two, ...
08:31:43.959 DEBUG  [account-one] Evaluated 1 locals (remaining 0): account_data
08:31:44.000 DEBUG  [account-one] Included config C:/Users/austin/code/aws-infrastructure/terragrunt_root/root.hcl has strategy shallow merge: merging config in (shallow).
08:31:44.000 DEBUG  [account-one] Detected 2 Hooks
08:31:44.008 DEBUG  [account-one] Generated file .\account-one\tg_generated_module_one.tf.
08:31:44.039 DEBUG  [account-one] Generated file .\account-one\tg_generated_module_two.tf.
08:31:44.144 DEBUG  [account-one] Detected 3 Hooks
08:31:44.144 INFO   [account-one] Executing hook: hook_one
08:31:44.144 DEBUG  [account-one] Running command: bash -c <hook-command>
08:31:50.232 INFO   [account-one] Executing hook: hook_two
08:31:50.232 DEBUG  [account-one] Running command: bash -c <hook-command>
08:31:50.885 DEBUG  [account-one] Running command: terraform plan -out=tfplan.binary -input=false
08:31:50.885 DEBUG  [account-one] Engine is not enabled, running command directly in .\account-one
08:33:49.176 STDOUT [account-one] terraform: module.module_one.module.vpc.data.external.external_data: Reading...

[yhakbar]

Yup! You can use the terragrunt-strict-include flag, @atheiman .

[atheiman]

@yhakbar I tried that and still have the same log output processing all directories

Edit: and Im on terragrunt release 70.2

[yhakbar]

Ah, I didn't understand what you were saying. You were asking, specifically for a parsing optimization to avoid parsing files that have been excluded (including when they are excluded by default, such as when using the terragrunt-strict-include flag).

Did I get that right?

[atheiman]

thats correct

[yhakbar]

We have a lot of HCL parsing optimization planned, and this would be included in that. It's non-trivial to change how we optimize parsing right now, so you may need to have patience while we find the right time and structure to address this optimization.