Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
* [ECS] Support 5 missing fields in TaskDefinition (#5424)

* Add missing fields in RegisterTaskDef

Signed-off-by: t-kikuc <[email protected]>

* remove two restrictions

Signed-off-by: t-kikuc <[email protected]>

---------

Signed-off-by: t-kikuc <[email protected]>
Signed-off-by: pipecd-bot <[email protected]>

* [ECS] Enabled to configure/update `PropagateTags` and `EnableECSManagedTags` (#5425)

* Enable updating PropagateTags and specifying other than SERVICE

Signed-off-by: t-kikuc <[email protected]>

* Enable updating EnableECSManagedTags

Signed-off-by: t-kikuc <[email protected]>

* Remove default PropagateTags: unnecessary

Signed-off-by: t-kikuc <[email protected]>

* support propagateTags other than SERVICE

Signed-off-by: t-kikuc <[email protected]>

---------

Signed-off-by: t-kikuc <[email protected]>
Signed-off-by: pipecd-bot <[email protected]>

* Enhance logging in EventWatcher (#5443)

* Add logger.Error() before returning for detailed logs

Signed-off-by: t-kikuc <[email protected]>

* Enhance error logging in event watcher with additional context for push and commit failures

Signed-off-by: t-kikuc <[email protected]>

---------

Signed-off-by: t-kikuc <[email protected]>
Signed-off-by: pipecd-bot <[email protected]>

* fixed to certainly store the startTime (#5461)

Signed-off-by: t-kikuc <[email protected]>
Signed-off-by: pipecd-bot <[email protected]>

* Add default envs in ScriptRunStage: SR_TRIGGERED_COMMANDER, SR_IS_ROLLBACK (#5464)

* Add TriggeredCommandar

Signed-off-by: t-kikuc <[email protected]>

* add isRollback

Signed-off-by: t-kikuc <[email protected]>

* update docs

Signed-off-by: t-kikuc <[email protected]>

* fix name->id

Signed-off-by: t-kikuc <[email protected]>

* fix env name

Signed-off-by: t-kikuc <[email protected]>

* fix: specify isRollback since deployment.Status is always Planned

Signed-off-by: t-kikuc <[email protected]>

* fix typo

Signed-off-by: t-kikuc <[email protected]>

* Add case of `pipectl sync`

Signed-off-by: t-kikuc <[email protected]>

---------

Signed-off-by: t-kikuc <[email protected]>
Signed-off-by: pipecd-bot <[email protected]>

* Release v0.50.1 (#5498)

Signed-off-by: Shinnosuke Sawada-Dazai <[email protected]>
Signed-off-by: pipecd-bot <[email protected]>

---------

Signed-off-by: t-kikuc <[email protected]>
Signed-off-by: pipecd-bot <[email protected]>
Signed-off-by: Shinnosuke Sawada-Dazai <[email protected]>
Co-authored-by: Tetsuya KIKUCHI <[email protected]>
Co-authored-by: Shinnosuke Sawada-Dazai <[email protected]>
  • Loading branch information
3 people authored Jan 21, 2025
1 parent cd3c295 commit 8ff25b1
Show file tree
Hide file tree
Showing 11 changed files with 78 additions and 34 deletions.
2 changes: 1 addition & 1 deletion RELEASE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Generated by `make release` command.
# DO NOT EDIT.
tag: v0.50.0
tag: v0.50.1

releaseNoteGenerator:
showCommitter: false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -531,15 +531,12 @@ There are some restrictions in configuring a service definition file.
- `deploymentController` is required and must be `EXTERNAL`.
- `loadBalancers` is not supported. Use `targetGroups` in [ECSDeploymentInput](#ecsdeploymentinput) instead.
- `platformFamily` is not supported.
- `propagateTags` is always set as `SERVICE`.
- `taskDefinition` is not supported. PipeCD uses the definition in `taskDefinitionFile` in [ECSDeploymentInput](#ecsdeploymentinput).

### Restrictions of Task Definition

There are some restrictions in configuring a task definition file.

- `placementConstraints` is not supported.
- `proxyConfiguration` is not supported.
- `tags` is not supported.

### ECSTargetGroupInput
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,12 @@ You can use the envrionment values related to the deployment.
|SR_APPLICATION_NAME| The application name | example |
|SR_TRIGGERED_AT| The timestamp when the deployment is triggered | 1719571113 |
|SR_TRIGGERED_COMMIT_HASH| The commit hash that triggered the deployment | 2bf969a3dad043aaf8ae6419943255e49377da0d |
|SR_TRIGGERED_COMMANDER| The ID of user who triggered the deployment via UI. This is APIKey's ID if it was triggered via `pipectl sync`. This is empty if it was triggered by your piped. | userid |
|SR_REPOSITORY_URL| The repository url configured in the piped config | [email protected]:org/repo.git, https://github.com/org/repo |
|SR_SUMMARY| The summary of the deployment | Sync with the specified pipeline because piped received a command from user via web console or pipectl|
|SR_CONTEXT_RAW| The json encoded string of above values | {"deploymentID":"877625fc-196a-40f9-b6a9-99decd5494a0","applicationID":"8d7609e0-9ff6-4dc7-a5ac-39660768606a","applicationName":"example","triggeredAt":1719571113,"triggeredCommitHash":"2bf969a3dad043aaf8ae6419943255e49377da0d","repositoryURL":"[email protected]:org/repo.git","labels":{"env":"example","team":"product"}} |
|SR_LABELS_XXX| The label attached to the deployment. The env name depends on the label name. For example, if a deployment has the labels `env:prd` and `team:server`, `SR_LABELS_ENV` and `SR_LABELS_TEAM` are registered. | prd, server |
|SR_IS_ROLLBACK| This is `true` if the deployment is rollbacking. Otherwise, this is `false`. | false |
### Use `SR_CONTEXT_RAW` with jq
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -531,15 +531,12 @@ There are some restrictions in configuring a service definition file.
- `deploymentController` is required and must be `EXTERNAL`.
- `loadBalancers` is not supported. Use `targetGroups` in [ECSDeploymentInput](#ecsdeploymentinput) instead.
- `platformFamily` is not supported.
- `propagateTags` is always set as `SERVICE`.
- `taskDefinition` is not supported. PipeCD uses the definition in `taskDefinitionFile` in [ECSDeploymentInput](#ecsdeploymentinput).

### Restrictions of Task Definition

There are some restrictions in configuring a task definition file.

- `placementConstraints` is not supported.
- `proxyConfiguration` is not supported.
- `tags` is not supported.

### ECSTargetGroupInput
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,10 +108,12 @@ You can use the envrionment values related to the deployment.
|SR_APPLICATION_NAME| The application name | example |
|SR_TRIGGERED_AT| The timestamp when the deployment is triggered | 1719571113 |
|SR_TRIGGERED_COMMIT_HASH| The commit hash that triggered the deployment | 2bf969a3dad043aaf8ae6419943255e49377da0d |
|SR_TRIGGERED_COMMANDER| The ID of user who triggered the deployment via UI. This is APIKey's ID if it was triggered via `pipectl sync`. This is empty if it was triggered by your piped. | userid |
|SR_REPOSITORY_URL| The repository url configured in the piped config | [email protected]:org/repo.git, https://github.com/org/repo |
|SR_SUMMARY| The summary of the deployment | Sync with the specified pipeline because piped received a command from user via web console or pipectl|
|SR_CONTEXT_RAW| The json encoded string of above values | {"deploymentID":"877625fc-196a-40f9-b6a9-99decd5494a0","applicationID":"8d7609e0-9ff6-4dc7-a5ac-39660768606a","applicationName":"example","triggeredAt":1719571113,"triggeredCommitHash":"2bf969a3dad043aaf8ae6419943255e49377da0d","repositoryURL":"[email protected]:org/repo.git","labels":{"env":"example","team":"product"}} |
|SR_LABELS_XXX| The label attached to the deployment. The env name depends on the label name. For example, if a deployment has the labels `env:prd` and `team:server`, `SR_LABELS_ENV` and `SR_LABELS_TEAM` are registered. | prd, server |
|SR_IS_ROLLBACK| This is `true` if the deployment is rollbacking. Otherwise, this is `false`. | false |
### Use `SR_CONTEXT_RAW` with jq
Expand Down
68 changes: 48 additions & 20 deletions pkg/app/piped/eventwatcher/eventwatcher.go
Original file line number Diff line number Diff line change
Expand Up @@ -126,14 +126,16 @@ func (w *watcher) Run(ctx context.Context) error {

workingDir, err := os.MkdirTemp("", "event-watcher")
if err != nil {
return fmt.Errorf("failed to create the working directory: %w", err)
w.logger.Error("failed to create the working directory", zap.Error(err))
return err
}
defer os.RemoveAll(workingDir)
w.workingDir = workingDir

for _, r := range w.config.Repositories {
repo, err := w.cloneRepo(ctx, r)
if err != nil {
w.logger.Error("failed to clone repository", zap.String("repo-id", r.RepoID), zap.Error(err))
return err
}
defer repo.Clean()
Expand Down Expand Up @@ -303,11 +305,13 @@ func (w *watcher) run(ctx context.Context, repo git.Repo, repoCfg config.PipedRe
func (w *watcher) cloneRepo(ctx context.Context, repoCfg config.PipedRepository) (git.Repo, error) {
dst, err := os.MkdirTemp(w.workingDir, repoCfg.RepoID)
if err != nil {
return nil, fmt.Errorf("failed to create a new temporary directory: %w", err)
w.logger.Error("failed to create a new temporary directory", zap.Error(err))
return nil, err
}
repo, err := w.gitClient.Clone(ctx, repoCfg.RepoID, repoCfg.Remote, repoCfg.Branch, dst)
if err != nil {
return nil, fmt.Errorf("failed to clone repository %s: %w", repoCfg.RepoID, err)
w.logger.Error("failed to clone repository", zap.String("repo-id", repoCfg.RepoID), zap.Error(err))
return nil, err
}
return repo, nil
}
Expand All @@ -317,11 +321,13 @@ func (w *watcher) execute(ctx context.Context, repo git.Repo, repoID string, eve
// Copy the repo to another directory to modify local file to avoid reverting previous changes.
tmpDir, err := os.MkdirTemp(w.workingDir, "repo")
if err != nil {
return fmt.Errorf("failed to create a new temporary directory: %w", err)
w.logger.Error("failed to create a new temporary directory", zap.Error(err))
return err
}
tmpRepo, err := repo.CopyToModify(filepath.Join(tmpDir, "tmp-repo"))
if err != nil {
return fmt.Errorf("failed to copy the repository to the temporary directory: %w", err)
w.logger.Error("failed to copy the repository to the temporary directory", zap.Error(err))
return err
}
// nolint: errcheck
defer tmpRepo.Clean()
Expand Down Expand Up @@ -368,7 +374,8 @@ func (w *watcher) execute(ctx context.Context, repo git.Repo, repoID string, eve
Labels: matcher.Labels,
})
if err != nil {
return fmt.Errorf("failed to get the latest event: %w", err)
w.logger.Error("failed to get the latest event", zap.Error(err))
return err
}
// The case where the latest event has already been handled.
if resp.Event.CreatedAt > latestEvent.CreatedAt {
Expand Down Expand Up @@ -429,7 +436,8 @@ func (w *watcher) execute(ctx context.Context, repo git.Repo, repoID string, eve
}
if len(outDatedEvents) > 0 {
if _, err := w.apiClient.ReportEventStatuses(ctx, &pipedservice.ReportEventStatusesRequest{Events: outDatedEvents}); err != nil {
return fmt.Errorf("failed to report event statuses: %w", err)
w.logger.Error("failed to report event statuses", zap.Error(err))
return err
}
w.logger.Info(fmt.Sprintf("successfully made %d events OUTDATED", len(outDatedEvents)))
}
Expand All @@ -449,8 +457,11 @@ func (w *watcher) execute(ctx context.Context, repo git.Repo, repoID string, eve
retry := backoff.NewRetry(retryPushNum, backoff.NewConstant(retryPushInterval))
for branch, events := range branchHandledEvents {
_, err = retry.Do(ctx, func() (interface{}, error) {
err := tmpRepo.Push(ctx, branch)
return nil, err
if err := tmpRepo.Push(ctx, branch); err != nil {
w.logger.Error("failed to push commits", zap.String("repo-id", repoID), zap.String("branch", branch), zap.Error(err))
return nil, err
}
return nil, nil
})

if err == nil {
Expand Down Expand Up @@ -493,11 +504,13 @@ func (w *watcher) updateValues(ctx context.Context, repo git.Repo, repoID string
// Copy the repo to another directory to modify local file to avoid reverting previous changes.
tmpDir, err := os.MkdirTemp(w.workingDir, "repo")
if err != nil {
return fmt.Errorf("failed to create a new temporary directory: %w", err)
w.logger.Error("failed to create a new temporary directory", zap.Error(err))
return err
}
tmpRepo, err := repo.CopyToModify(filepath.Join(tmpDir, "tmp-repo"))
if err != nil {
return fmt.Errorf("failed to copy the repository to the temporary directory: %w", err)
w.logger.Error("failed to copy the repository to the temporary directory", zap.Error(err))
return err
}
defer tmpRepo.Clean()

Expand Down Expand Up @@ -536,7 +549,8 @@ func (w *watcher) updateValues(ctx context.Context, repo git.Repo, repoID string
Labels: e.Labels,
})
if err != nil {
return fmt.Errorf("failed to get the latest event: %w", err)
w.logger.Error("failed to get the latest event", zap.Error(err))
return err
}
// The case where the latest event has already been handled.
if resp.Event.CreatedAt > latestEvent.CreatedAt {
Expand Down Expand Up @@ -577,7 +591,8 @@ func (w *watcher) updateValues(ctx context.Context, repo git.Repo, repoID string
}
if len(outDatedEvents) > 0 {
if _, err := w.apiClient.ReportEventStatuses(ctx, &pipedservice.ReportEventStatusesRequest{Events: outDatedEvents}); err != nil {
return fmt.Errorf("failed to report event statuses: %w", err)
w.logger.Error("failed to report event statuses", zap.Error(err))
return err
}
w.logger.Info(fmt.Sprintf("successfully made %d events OUTDATED", len(outDatedEvents)))
}
Expand All @@ -587,12 +602,16 @@ func (w *watcher) updateValues(ctx context.Context, repo git.Repo, repoID string

retry := backoff.NewRetry(retryPushNum, backoff.NewConstant(retryPushInterval))
_, err = retry.Do(ctx, func() (interface{}, error) {
err := tmpRepo.Push(ctx, tmpRepo.GetClonedBranch())
return nil, err
if err := tmpRepo.Push(ctx, tmpRepo.GetClonedBranch()); err != nil {
w.logger.Error("failed to push commits", zap.String("repo-id", repoID), zap.String("branch", tmpRepo.GetClonedBranch()), zap.Error(err))
return nil, err
}
return nil, nil
})
if err == nil {
if _, err := w.apiClient.ReportEventStatuses(ctx, &pipedservice.ReportEventStatusesRequest{Events: handledEvents}); err != nil {
return fmt.Errorf("failed to report event statuses: %w", err)
w.logger.Error("failed to report event statuses", zap.Error(err))
return err
}
w.milestoneMap.Store(repoID, maxTimestamp)
return nil
Expand All @@ -613,10 +632,12 @@ func (w *watcher) updateValues(ctx context.Context, repo git.Repo, repoID string
handledEvents[i].StatusDescription = fmt.Sprintf("Failed to push changed files: %v", err)
}
if _, err := w.apiClient.ReportEventStatuses(ctx, &pipedservice.ReportEventStatusesRequest{Events: handledEvents}); err != nil {
return fmt.Errorf("failed to report event statuses: %w", err)
w.logger.Error("failed to report event statuses: %w", zap.Error(err))
return err
}
w.milestoneMap.Store(repoID, maxTimestamp)
return fmt.Errorf("failed to push commits: %w", err)
w.logger.Error("failed to push commits", zap.Error(err))
return err
}

// commitFiles commits changes if the data in Git is different from the latest event.
Expand Down Expand Up @@ -647,14 +668,16 @@ func (w *watcher) commitFiles(ctx context.Context, latestEvent *model.Event, eve
newContent, upToDate, err = modifyText(path, r.Regex, latestEvent.Data)
}
if err != nil {
w.logger.Error("failed to modify file", zap.Error(err))
return "", err
}
if upToDate {
continue
}

if err := os.WriteFile(path, newContent, os.ModePerm); err != nil {
return "", fmt.Errorf("failed to write file: %w", err)
w.logger.Error("failed to write file", zap.Error(err))
return "", err
}
changes[filePath] = newContent
}
Expand All @@ -670,7 +693,12 @@ func (w *watcher) commitFiles(ctx context.Context, latestEvent *model.Event, eve
branch := makeBranchName(newBranch, eventName, repo.GetClonedBranch())
trailers := maps.Clone(latestEvent.Contexts)
if err := repo.CommitChanges(ctx, branch, commitMsg, newBranch, changes, trailers); err != nil {
return "", fmt.Errorf("failed to perform git commit: %w", err)
w.logger.Error("failed to perform git commit",
zap.String("branch", branch),
zap.Bool("make-new-branch", newBranch),
zap.Int("changed-files", len(changes)),
zap.Error(err))
return "", err
}
w.logger.Info(fmt.Sprintf("event watcher will update values of Event %q", eventName))
return branch, nil
Expand Down
2 changes: 1 addition & 1 deletion pkg/app/piped/executor/kubernetes/rollback.go
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ func (e *rollbackExecutor) ensureScriptRunRollback(ctx context.Context) model.St
}
}

ci := scriptrun.NewContextInfo(e.Deployment)
ci := scriptrun.NewContextInfo(e.Deployment, true)
ciEnv, err := ci.BuildEnv()
if err != nil {
e.LogPersister.Errorf("failed to build srcipt run context info: %w", err)
Expand Down
6 changes: 5 additions & 1 deletion pkg/app/piped/executor/scriptrun/script_run_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,21 +22,25 @@ func Test_ContextInfo_BuildEnv(t *testing.T) {
ApplicationName: "application-name",
TriggeredAt: 1234567890,
TriggeredCommitHash: "commit-hash",
TriggeredCommander: "commander",
RepositoryURL: "repo-url",
Labels: map[string]string{
"key1": "value1",
"key2": "value2",
},
Summary: "summary",
IsRollback: false,
Summary: "summary",
},
want: map[string]string{
"SR_DEPLOYMENT_ID": "deployment-id",
"SR_APPLICATION_ID": "application-id",
"SR_APPLICATION_NAME": "application-name",
"SR_TRIGGERED_AT": "1234567890",
"SR_TRIGGERED_COMMIT_HASH": "commit-hash",
"SR_TRIGGERED_COMMANDER": "commander",
"SR_REPOSITORY_URL": "repo-url",
"SR_SUMMARY": "summary",
"SR_IS_ROLLBACK": "false",
"SR_LABELS_KEY1": "value1",
"SR_LABELS_KEY2": "value2",
},
Expand Down
10 changes: 8 additions & 2 deletions pkg/app/piped/executor/scriptrun/scriptrun.go
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ func (e *Executor) executeCommand() model.StageStatus {
}
}

ci := NewContextInfo(e.Deployment)
ci := NewContextInfo(e.Deployment, false)
ciEnv, err := ci.BuildEnv()
if err != nil {
e.LogPersister.Errorf("failed to build srcipt run context info: %w", err)
Expand Down Expand Up @@ -136,22 +136,26 @@ type ContextInfo struct {
ApplicationName string `json:"applicationName,omitempty"`
TriggeredAt int64 `json:"triggeredAt,omitempty"`
TriggeredCommitHash string `json:"triggeredCommitHash,omitempty"`
TriggeredCommander string `json:"triggeredCommander,omitempty"`
RepositoryURL string `json:"repositoryURL,omitempty"`
Summary string `json:"summary,omitempty"`
Labels map[string]string `json:"labels,omitempty"`
IsRollback bool `json:"isRollback,omitempty"`
}

// NewContextInfo creates a new ContextInfo from the given deployment.
func NewContextInfo(d *model.Deployment) *ContextInfo {
func NewContextInfo(d *model.Deployment, isRollback bool) *ContextInfo {
return &ContextInfo{
DeploymentID: d.Id,
ApplicationID: d.ApplicationId,
ApplicationName: d.ApplicationName,
TriggeredAt: d.Trigger.Timestamp,
TriggeredCommitHash: d.Trigger.Commit.Hash,
TriggeredCommander: d.Trigger.Commander,
RepositoryURL: d.GitPath.Repo.Remote,
Summary: d.Summary,
Labels: d.Labels,
IsRollback: isRollback,
}
}

Expand All @@ -168,8 +172,10 @@ func (src *ContextInfo) BuildEnv() (map[string]string, error) {
"SR_APPLICATION_NAME": src.ApplicationName,
"SR_TRIGGERED_AT": strconv.FormatInt(src.TriggeredAt, 10),
"SR_TRIGGERED_COMMIT_HASH": src.TriggeredCommitHash,
"SR_TRIGGERED_COMMANDER": src.TriggeredCommander,
"SR_REPOSITORY_URL": src.RepositoryURL,
"SR_SUMMARY": src.Summary,
"SR_IS_ROLLBACK": strconv.FormatBool(src.IsRollback),
"SR_CONTEXT_RAW": string(b), // Add the raw json string as an environment variable.
}

Expand Down
2 changes: 1 addition & 1 deletion pkg/app/piped/executor/wait/wait.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ func (e *Executor) Execute(sig executor.StopSignal) model.StageStatus {
} else {
startTime = time.Now()
}
defer e.saveStartTime(sig.Context(), startTime)
e.saveStartTime(sig.Context(), startTime)

timer := time.NewTimer(duration)
defer timer.Stop()
Expand Down
12 changes: 10 additions & 2 deletions pkg/app/piped/platformprovider/ecs/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ func (c *client) CreateService(ctx context.Context, service types.Service) (*typ
PlacementConstraints: service.PlacementConstraints,
PlacementStrategy: service.PlacementStrategy,
PlatformVersion: service.PlatformVersion,
PropagateTags: types.PropagateTagsService,
PropagateTags: service.PropagateTags,
Role: service.RoleArn,
SchedulingStrategy: service.SchedulingStrategy,
Tags: service.Tags,
Expand Down Expand Up @@ -143,6 +143,8 @@ func (c *client) UpdateService(ctx context.Context, service types.Service) (*typ
PlacementStrategy: service.PlacementStrategy,
// TODO: Support update other properties of service.
// PlacementConstraints: service.PlacementConstraints,
PropagateTags: service.PropagateTags,
EnableECSManagedTags: aws.Bool(service.EnableECSManagedTags),
}

// If desiredCount is 0 or not set, keep current desiredCount because a user might use AutoScaling.
Expand Down Expand Up @@ -188,9 +190,15 @@ func (c *client) RegisterTaskDefinition(ctx context.Context, taskDefinition type
Volumes: taskDefinition.Volumes,
RuntimePlatform: taskDefinition.RuntimePlatform,
EphemeralStorage: taskDefinition.EphemeralStorage,
// Requires defined at task level in case Fargate is used.
// Cpu and Memory must be defined if Fargate is used.
Cpu: taskDefinition.Cpu,
Memory: taskDefinition.Memory,

InferenceAccelerators: taskDefinition.InferenceAccelerators,
IpcMode: taskDefinition.IpcMode,
PidMode: taskDefinition.PidMode,
PlacementConstraints: taskDefinition.PlacementConstraints,
ProxyConfiguration: taskDefinition.ProxyConfiguration,
// TODO: Support tags for registering task definition.
}
output, err := c.ecsClient.RegisterTaskDefinition(ctx, input)
Expand Down

0 comments on commit 8ff25b1

Please sign in to comment.