Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: exec, run commands implementation #3723

Open
wants to merge 81 commits into
base: main
Choose a base branch
from
Open

Conversation

levkohimins
Copy link
Contributor

@levkohimins levkohimins commented Jan 2, 2025

Description

  1. exec, run commands implementation.
  2. Sorting and renaming CLI flags.
  3. Automatic generation of environment variables.
  4. Code refactoring.

Relates to #3445 .

TODOs

Read the Gruntwork contribution guidelines.

  • Update the docs.
  • Run the relevant tests successfully, including pre-commit checks.
  • Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
  • Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added / Removed / Updated [X].

Migration Guide

@levkohimins levkohimins requested a review from denis256 January 4, 2025 13:07
Copy link
Collaborator

@yhakbar yhakbar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting pulled onto other things, but I thought I see enough that warrants changes before continuing with the review (I'm only like 37/206 through this review...). Will come back to this soon, but wanted to give you feedback early while I'm not looking at it.

.golangci.yml Outdated
@@ -79,6 +79,8 @@ linters:
- nolintlint
- wrapcheck
- varnamelen
- recvcheck
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we disabling this check? Shouldn't we have all the receiver types be consistent? I don't know why we'd want to mix and match pointers and non-pointers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At that time, the latest version of golangci-lint 1.62.2 did not have this fix. I just updated golangci-lint, now it has. I will enable recvcheck.

Copy link
Contributor Author

@levkohimins levkohimins Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, no, it is still not fixed in golangci-lint

pkg/log/level.go:83:6: the methods of "Level" use pointer receiver and non-pointer receiver. (recvcheck)
// UnmarshalText implements encoding.TextUnmarshaler.
func (level *Level) UnmarshalText(text []byte) error {

I will add a comment on why this check is disabled.

.golangci.yml Outdated Show resolved Hide resolved
cli/app.go Outdated Show resolved Hide resolved
Usage: "A key=value attribute to override in a provider block as part of the aws-provider-patch command. May be specified multiple times.",
},
}),
}
}

func NewCommand(opts *options.TerragruntOptions) *cli.Command {
return &cli.Command{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also hide this command? Who cares about it now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only moved the code and I do not know the answer. Technically, of course, it can be hidden.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to ignore for now, but we should get rid of it or hide it before 1.0. it's a waste of space in the CLI API.

cli/commands/default/command.go Outdated Show resolved Hide resolved
cli/commands/exec/command.go Outdated Show resolved Hide resolved
},
Flags: NewFlags(opts, cmdOpts).Sort(),
ErrorOnUndefinedFlag: true,
Hidden: true,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could I get confirmation on why we're hiding the command? Are we looking to dark launch it?

Copy link
Contributor Author

@levkohimins levkohimins Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mentioned it in the slack:

The new exec and run commands are temporarily hidden from help until we thoroughly test and document them in order not to delay the merge into main.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I add a comment or make them visible?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I would recommend is marking them as experiments.

The danger with just marking them as hidden is that we're updating a lot of documentation, etc and we have to in order for them to be used right.

By marking them as experiments and making them visible, we document both that they're not ready for production usage, and explain how they're supposed to be used.

UsageText: "terragrunt run [options] -- <tofu/terraform command>",
Description: "Run a command, passing arguments to a wrapped tofu/terraform binary.\n\nThis is the explicit, and most flexible form of running an IaC update with Terragrunt. Shortcuts can be found in \"terragrunt --help\" for common use-cases.",
Examples: []string{
"# Run a plan\nterragrunt run -- plan\n# Shortcut:\n# terragrunt plan",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we already have this shortcut supported?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shortcut is what we have now terragrunt plan, it is still supported, but with a deprecation warning. Should I disable the warning?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya, see this:
#3723 (comment)

Name: CommandName,
Usage: "Run an OpenTofu/Terraform command. Shortcuts for common `run` commands are provided below.",
UsageText: "terragrunt run [options] -- <tofu/terraform command>",
Description: "Run a command, passing arguments to a wrapped tofu/terraform binary.\n\nThis is the explicit, and most flexible form of running an IaC update with Terragrunt. Shortcuts can be found in \"terragrunt --help\" for common use-cases.",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Description: "Run a command, passing arguments to a wrapped tofu/terraform binary.\n\nThis is the explicit, and most flexible form of running an IaC update with Terragrunt. Shortcuts can be found in \"terragrunt --help\" for common use-cases.",
Description: "Run a command, passing arguments to an orchestrated tofu/terraform binary.\n\nThis is the explicit, and most flexible form of running an IaC command with Terragrunt. Shortcuts can be found in \"terragrunt --help\" for common use-cases.",

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we already support shortcuts?

Copy link
Contributor Author

@levkohimins levkohimins Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Responded here.

Description: "Run a command, passing arguments to a wrapped tofu/terraform binary.\n\nThis is the explicit, and most flexible form of running an IaC update with Terragrunt. Shortcuts can be found in \"terragrunt --help\" for common use-cases.",
Examples: []string{
"# Run a plan\nterragrunt run -- plan\n# Shortcut:\n# terragrunt plan",
"# Run output with -json flag\nterragrunt run -- output -json\n# Shortcut:\n# terragrunt output -json",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we already have this shortcut supported?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Responded here.

@@ -57,3 +57,12 @@ type MaxRetriesExceeded struct {
func (err MaxRetriesExceeded) Error() string {
return fmt.Sprintf("Exhausted retries (%v) for command %v %v", err.Opts.RetryMaxAttempts, err.Opts.TerraformPath, strings.Join(err.Opts.TerraformCliArgs, " "))
}

type RunAllDisabledErr struct {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is moved code, not mine. I found the following

// Known terraform commands that are explicitly not supported in run-all due to the nature of the command. This is
// tracked as a map that maps the terraform command to the reasoning behind disallowing the command in run-all.
var runAllDisabledCommands = map[string]string{
terraform.CommandNameImport: "terraform import should only be run against a single state representation to avoid injecting the wrong object in the wrong state representation.",
terraform.CommandNameTaint: "terraform taint should only be run against a single state representation to avoid using the wrong state address.",
terraform.CommandNameUntaint: "terraform untaint should only be run against a single state representation to avoid using the wrong state address.",
terraform.CommandNameConsole: "terraform console requires stdin, which is shared across all instances of run-all when multiple modules run concurrently.",
terraform.CommandNameForceUnlock: "lock IDs are unique per state representation and thus should not be run with run-all.",
// MAINTAINER'S NOTE: There are a few other commands that might not make sense, but we deliberately allow it for
// certain use cases that are documented here:
// - state : Supporting `state` with run-all could be useful for a mass pull and push operation, which can
// be done en masse with the use of relative pathing.
// - login / logout : Supporting `login` with run-all could be useful when used in conjunction with mise and
// multi-terraform version setups, where multiple terraform versions need to be configured.
// - version : Supporting `version` with run-all could be useful for sanity checking a multi-version setup.
}

cli/commands/scaffold/command.go Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the file called deprecated_flag now? Not a big deal, just a little strange.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would you like it to be called?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It used to be called deprecated_flags.go


// BoolWithDeprecatedFlag adds deprecated names with strict mode control for the given flag.
// If `oldNames` is not specified, names are taken from the given `flag` with adding `terragrunt-/TERRAGRUNT_` prefixes.
func BoolWithDeprecatedFlag(opts *options.TerragruntOptions, flag *cli.BoolFlag, oldNames ...string) cli.Flag { //nolint:ireturn
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not return cli.BoolFlag?

Copy link
Contributor Author

@levkohimins levkohimins Jan 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it returns *Flag struct.

cannot use &Flag{…} (value of type *Flag) as  *"github.com/gruntwork-io/terragrunt/internal/cli".BoolFlag value in return statement [IncompatibleAssign]

I will replace cli.Flag with *Flag

DeprecatedFlagNamePrefix = "terragrunt-"
)

// BoolWithDeprecatedFlag adds deprecated names with strict mode control for the given flag.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// BoolWithDeprecatedFlag adds deprecated names with strict mode control for the given flag.
// BoolWithDeprecatedFlag adds deprecated names when strict mode is disabled for the given flag.

Copy link
Contributor Author

@levkohimins levkohimins Jan 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure, it adds deprecated names regardless of strict mode settings. Strict mode only affects behavior, prohibits or warns not to use these aliases(flags).

internal/cloner/clone.go Outdated Show resolved Hide resolved
@@ -126,6 +130,10 @@ var StrictControls = Controls{
Error: errors.Errorf("The `%s` command is no longer supported. Use `terragrunt run-all validate` instead.", ValidateAll),
Warning: fmt.Sprintf("The `%s` command is deprecated and will be removed in a future version. Use `terragrunt run-all validate` instead.", ValidateAll),
},
DefaultCommand: {
Error: errors.New("The default command is no longer supported. Use `terragrunt run` instead."),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Error: errors.New("The default command is no longer supported. Use `terragrunt run` instead."),
Error: errors.New("Terragrunt no longer has a default command. Use `terragrunt run` to explicitly pass commands to OpenTofu/Terraform instead. e.g. `terragrunt run -- plan`"),

It might be too much, but I think it would be really nice if we could use the users incorrect command as a basis for this.

For example:

$ terragrunt foo
ERROR: The command `foo` is not a valid Terragrunt command. Use `terragrunt run` to explicitly pass commands to OpenTofu/Terraform instead. e.g. `terragrunt run -- foo`

@@ -142,6 +150,7 @@ var StrictControls = Controls{
Error: errors.Errorf("The `--%s` flag is no longer supported. Use `--terragrunt-log-format=json` instead.", TfLogJSON),
Warning: fmt.Sprintf("The `--%s` flag is deprecated and will be removed in a future version. Use `--terragrunt-log-format=json` instead.", TfLogJSON),
},
RenamedFlag: {},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • TODO: Populate this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RenamedFlag is a strict control for multiple flags, meaning it returns a message with different names. The current implementation of the strict control can not allow to do it. Therefore the message builder is moved to here.

options/options.go Show resolved Hide resolved
test/fixtures/exec-cmd/script.sh Outdated Show resolved Hide resolved
@yhakbar
Copy link
Collaborator

yhakbar commented Jan 7, 2025

I was playing with this functionality locally, and noticed these two things:

$ terragrunt exec ls
08:53:02.755 ERROR  target command not specified
08:53:02.758 ERROR  Unable to determine underlying exit code, so Terragrunt will exit with error code 1

That should work, given that terragrunt run plan works.

At the very least, a better error message should be thrown if we can't make it so that a user doesn't need --.

$ terragrunt exec -- ls
08:44:45.451 WARN   Detected that init is needed, but Auto-Init is disabled. Continuing with further actions, but subsequent terraform commands may fail.
main.tf
terragrunt.hcl

I'm also seeing issues with the --all flag of run:

$ terragrunt run --all plan
08:57:59.202 ERROR  stat ./terragrunt.hcl: no such file or directory
08:57:59.202 ERROR  Unable to determine underlying exit code, so Terragrunt will exit with error code 1
$ terragrunt run-all plan
08:58:02.838 INFO   The stack at . will be processed in the following order for command plan:
Group 1
- Module ./live/storage/external/first-external
- Module ./live/storage/internal/first-internal


08:58:02.880 INFO   [live/storage/external/first-external] tofu: Initializing the backend...
08:58:02.880 INFO   [live/storage/external/first-external] tofu: Initializing provider plugins...
08:58:02.880 INFO   [live/storage/external/first-external] tofu: OpenTofu has been successfully initialized!
08:58:02.880 INFO   [live/storage/external/first-external] tofu:
08:58:02.880 INFO   [live/storage/external/first-external] tofu: You may now begin working with OpenTofu. Try running "tofu plan" to see
08:58:02.880 INFO   [live/storage/external/first-external] tofu: any changes that are required for your infrastructure. All OpenTofu commands
08:58:02.880 INFO   [live/storage/external/first-external] tofu: should now work.
08:58:02.880 INFO   [live/storage/external/first-external] tofu: If you ever set or change modules or backend configuration for OpenTofu,
08:58:02.880 INFO   [live/storage/external/first-external] tofu: rerun this command to reinitialize your working directory. If you forget, other
08:58:02.880 INFO   [live/storage/external/first-external] tofu: commands will detect it and remind you to do so if necessary.
08:58:02.880 INFO   [live/storage/internal/first-internal] tofu: Initializing the backend...
08:58:02.880 INFO   [live/storage/internal/first-internal] tofu: Initializing provider plugins...
08:58:02.880 INFO   [live/storage/internal/first-internal] tofu: OpenTofu has been successfully initialized!
08:58:02.880 INFO   [live/storage/internal/first-internal] tofu:
08:58:02.880 INFO   [live/storage/internal/first-internal] tofu: You may now begin working with OpenTofu. Try running "tofu plan" to see
08:58:02.880 INFO   [live/storage/internal/first-internal] tofu: any changes that are required for your infrastructure. All OpenTofu commands
08:58:02.880 INFO   [live/storage/internal/first-internal] tofu: should now work.
08:58:02.880 INFO   [live/storage/internal/first-internal] tofu: If you ever set or change modules or backend configuration for OpenTofu,
08:58:02.880 INFO   [live/storage/internal/first-internal] tofu: rerun this command to reinitialize your working directory. If you forget, other
08:58:02.880 INFO   [live/storage/internal/first-internal] tofu: commands will detect it and remind you to do so if necessary.
08:58:02.901 STDOUT [live/storage/external/first-external] tofu: No changes. Your infrastructure matches the configuration.
08:58:02.901 STDOUT [live/storage/external/first-external] tofu: OpenTofu has compared your real infrastructure against your configuration and
08:58:02.901 STDOUT [live/storage/external/first-external] tofu: found no differences, so no changes are needed.
08:58:02.901 STDOUT [live/storage/internal/first-internal] tofu: No changes. Your infrastructure matches the configuration.
08:58:02.901 STDOUT [live/storage/internal/first-internal] tofu: OpenTofu has compared your real infrastructure against your configuration and
08:58:02.901 STDOUT [live/storage/internal/first-internal] tofu: found no differences, so no changes are needed.

@yhakbar
Copy link
Collaborator

yhakbar commented Jan 7, 2025

Another thing I'm noticing is that the help for both exec and run aren't working right now:

$ terragrunt run --help
OpenTofu has no command named "run".

To see all of OpenTofu's top-level commands, run:
  tofu -help

12:09:13.675 ERROR  Failed to execute "tofu run -help" in
OpenTofu has no command named "run".

To see all of OpenTofu's top-level commands, run:
  tofu -help


exit status 1
$ terragrunt exec --help
OpenTofu has no command named "exec".

To see all of OpenTofu's top-level commands, run:
  tofu -help

12:09:44.576 ERROR  Failed to execute "tofu exec -help" in
OpenTofu has no command named "exec".

To see all of OpenTofu's top-level commands, run:
  tofu -help


exit status 1

@yhakbar yhakbar mentioned this pull request Jan 7, 2025
4 tasks
@yhakbar yhakbar mentioned this pull request Jan 7, 2025
4 tasks
@levkohimins
Copy link
Contributor Author

levkohimins commented Jan 7, 2025

Another thing I'm noticing is that the help for both exec and run aren't working right now:

it works:

go run . exec --help
Usage: terragrunt exec [options] -- <command>

   Execute a command using Terragrunt.

Examples:
   # Utilize the AWS CLI.
   terragrunt exec -- aws s3 ls

   # Inspect `main.tf` file of module for Unit
   terragrunt exec -- cat main.tf

Options:
   --auth-provider-cmd                   Run the provided command and arguments to authenticate Terragrunt dynamically when necessary. [$TG_AUTH_PROVIDER_CMD]
   --config                              The path to the Terragrunt config file. Default is terragrunt.hcl. [$TG_CONFIG]
   --debug-inputs                        Write debug.tfvars to working folder to help root-cause issues. [$TG_DEBUG_INPUTS]
   --download-dir                        The path to download OpenTofu/Terraform modules into. Default is .cache in the working directory. [$TG_DOWNLOAD_DIR]
   --iam-assume-role                     Assume the specified IAM role before executing OpenTofu/Terraform. [$TG_IAM_ASSUME_ROLE]
   --iam-assume-role-duration            Session duration for IAM Assume Role session. [$TG_IAM_ASSUME_ROLE_DURATION]
   --iam-assume-role-session-name        Name for the IAM Assumed Role session. [$TG_IAM_ASSUME_ROLE_SESSION_NAME]
   --iam-assume-role-web-identity-token  For AssumeRoleWithWebIdentity, the WebIdentity token. [$TG_IAM_ASSUME_ROLE_WEB_IDENTITY_TOKEN]
   --in-download-dir                     Run the provided command in the download directory. [$TG_IN_DOWNLOAD_DIR]

Global Options:
   --experiment value         Enables specific experiments. For a list of available experiments, see https://terragrunt.gruntwork.io/docs/reference/experiment-mode . [$TG_EXPERIMENT]
   --experiment-mode          Enables experiment mode for Terragrunt. For more information, see https://terragrunt.gruntwork.io/docs/reference/experiment-mode . [$TG_EXPERIMENT_MODE]
   --log-custom-format value  Set the custom log formatting. [$TG_LOG_CUSTOM_FORMAT]
   --log-disable              Disable logging. [$TG_LOG_DISABLE]
   --log-format value         Set the log format. [$TG_LOG_FORMAT]
   --log-level value          Sets the logging level for Terragrunt. Supported levels: stderr, stdout, error, warn, info, debug, trace. (default: info) [$TG_LOG_LEVEL]
   --log-show-abs-paths       Show absolute paths in logs. [$TG_LOG_SHOW_ABS_PATHS]
   --no-color                 Disable color output. [$TG_NO_COLOR]
   --non-interactive          Assume "yes" for all prompts. [$TG_NON_INTERACTIVE]
   --strict-control value     Enables specific strict controls. For a list of available controls, see https://terragrunt.gruntwork.io/docs/reference/strict-mode . [$TG_STRICT_CONTROL]
   --strict-mode              Enables strict mode for Terragrunt. For more information, see https://terragrunt.gruntwork.io/docs/reference/strict-mode . [$TG_STRICT_MODE]
   --working-dir value        The path to the directory of Terragrunt configurations. Default is current directory. [$TG_WORKING_DIR]
   --help, -h                 Show help.
   --version, -v              Show terragrunt version.

@levkohimins
Copy link
Contributor Author

levkohimins commented Jan 7, 2025

I was playing with this functionality locally, and noticed these two things:

$ terragrunt exec ls
08:53:02.755 ERROR  target command not specified
08:53:02.758 ERROR  Unable to determine underlying exit code, so Terragrunt will exit with error code 1

That should work, given that terragrunt run plan works.

Ok, it will work.

At the very least, a better error message should be thrown if we can't make it so that a user doesn't need --.

What text should be displayed for the terragrunt exec command without arguments?

I'm also seeing issues with the --all flag of run:

--all and --graph are not implemented yet.

@yhakbar
Copy link
Collaborator

yhakbar commented Jan 8, 2025

What text should be displayed for the terragrunt exec command without arguments?

Something like the following:

The `exec` command requires arguments to be used.

e.g.

terragrunt exec -- echo "Hello from Terragrunt!"

In general, we should aim to always do the following with our errors in order of priority:

  1. Explain exactly what a user did wrong to cause an error, and if possible where they did it.
  2. Give them an example of proper usage.
  3. If possible, correct their improper usage using the example of proper usage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants