Skip to content

Commit

Permalink
Look for vault secret.sh in .whip dir
Browse files Browse the repository at this point in the history
gwillem committed Aug 30, 2024
1 parent 37f783a commit 271cbf4
Showing 9 changed files with 45 additions and 102 deletions.
22 changes: 13 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -19,27 +19,31 @@ Whip your servers into line. Chief Whip is a _fast_ and _simple_ Ansible replace
| templates | mysql | stat |
| vault | postgresql | debug |

# Install

```
# for now
curl -fsSL https://sansec.io/downloads/darwin-arm64/whip -O ~/bin/whip
```

# Demo

> [!NOTE]
> Keep this in mind.
# Philosophy

How will Chief Whip _stay_ fast and simple? By will only contain features that satisfy 95% of use cases. Convention over configuration.

Fast!
How will Chief Whip _stay_ fast and simple?

1. Eliminate unnecessary SSH round trips: Ansibles biggest delay is caused by tasks that are sent one by one. Chief Whip bundles tasks into a single job.
2. Implemented in Golang instead of Python
Simple ==> Only build features that satisfy 95% of use cases. Convention over configuration. Support top used modules only.

Simple!
Fast ==> Eliminate unnecessary SSH round trips: Ansibles biggest delay is caused by tasks that are sent one by one. Chief Whip bundles tasks into a single job. Also, Chief Whip is written in Golang which runs faster than Python.

# But why?

I really loved Ansible. Compared to the popular configuration management systems at the time (Puppet, Chef, CFEngine), it was a breeze of fresh air. Simple configuration files, easy to learn, effective documentation, simple push architecture. My team used it to manage some 2k+ servers without a fuss.
Curiously, Ansible also started out as fast and simple. Compared to the popular configuration management systems at the time (Puppet, Chef, CFEngine), it was a breeze of fresh air. Simple configuration files, easy to learn, effective documentation, simple push architecture. My team used it to manage some 2k+ servers without a fuss.

Until version 2 or so. After the RedHat acquisition, Ansible has quickly grown into commercial bloatware. It's funny how RedHat got rid of the old objectives page (Simple, Fast) and replaced it with a corporate bog of marketing fluff. The task parameter documentation is hidden behind compulsory white paper downloads. Core modules have grown to support 20 extra options to support esoteric use cases. And above all, its once legendary speed is gone. Ansible feels sluggish today.
Until version 2 or so. After the RedHat acquisition, Ansible grew into commercial bloatware. RedHat got rid of the old objectives page (Simple, Fast) and replaced it with corporate marketing fluff. The task parameter documentation is hidden behind white paper downloads. Core modules have grown to support 20 extra options to support esoteric use cases. And above all, its once legendary speed is gone. Ansible feels sluggish today.

Ansible has grown too complex, as illustrated by this Hacker News comment:

@@ -66,7 +70,7 @@ Ansible has grown too complex, as illustrated by this Hacker News comment:

The latter, however we stick to most of Ansible's verbiage to ease a transition.

#### Isn't everybody using Docker, Kubernetes etc these days?
#### Isn't everybody using Docker, Kubernetes and Kamal etc these days?

[Nope](https://trends.google.com/trends/explore?date=all&q=ansible).

11 changes: 10 additions & 1 deletion cmd/whip/main.go
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
package main

import (
"fmt"
"os"

"github.com/gwillem/go-buildversion"
log "github.com/gwillem/go-simplelog"

"github.com/gwillem/whip/internal/vault"
@@ -39,10 +41,17 @@ It aims to be stand-in replacement for Ansible for 90% of use cases.`,
}
},
}
versionCmd = &cobra.Command{
Use: "version",
Short: "Print the version number of Whip",
Run: func(_ *cobra.Command, _ []string) {
fmt.Println("whip", buildversion.String())
},
}
)

func init() {
rootCmd.AddCommand(vaultEditCmd, vaultConvertCmd)
rootCmd.AddCommand(vaultEditCmd, vaultConvertCmd, versionCmd)
rootCmd.CompletionOptions.HiddenDefaultCmd = true
rootCmd.PersistentFlags().CountP("verbose", "v", "verbose output")
}
29 changes: 8 additions & 21 deletions cmd/whip/whip.go
Original file line number Diff line number Diff line change
@@ -14,6 +14,7 @@ import (
"github.com/gwillem/go-buildversion"
log "github.com/gwillem/go-simplelog"
"github.com/gwillem/whip/internal/assets"
"github.com/gwillem/whip/internal/fsutil"
"github.com/gwillem/whip/internal/model"
"github.com/gwillem/whip/internal/playbook"
"github.com/gwillem/whip/internal/runners"
@@ -22,20 +23,21 @@ import (
)

const (
deputyPath = ".cache/whip/deputy"
defaultAssetPath = "files"
deputyPath = ".cache/whip/deputy"
defaultAssetPath = "files"
defaultPlaybookPath = ".whip/playbook.yml"
)

//go:embed deputies

Check failure on line 31 in cmd/whip/whip.go

GitHub Actions / build

pattern deputies: cannot embed directory deputies: contains no embeddable files

Check failure on line 31 in cmd/whip/whip.go

GitHub Actions / build

pattern deputies: cannot embed directory deputies: contains no embeddable files

Check failure on line 31 in cmd/whip/whip.go

GitHub Actions / build

pattern deputies: cannot embed directory deputies: contains no embeddable files

Check failure on line 31 in cmd/whip/whip.go

GitHub Actions / build

pattern deputies: cannot embed directory deputies: contains no embeddable files

Check failure on line 31 in cmd/whip/whip.go

GitHub Actions / build

pattern deputies: cannot embed directory deputies: contains no embeddable files
var deputies embed.FS

func runWhip(cmd *cobra.Command, args []string) {
var playbookPath string
if len(args) == 0 {
// Look for ".whip/playbook.yml" in current and parent directories
playbookPath = findPlaybookPath()
} else {
if len(args) > 0 {
playbookPath = args[0]
} else {
// Look for ".whip/playbook.yml" in current and parent directories
playbookPath = fsutil.FindAncestorPath(defaultPlaybookPath)
}

if playbookPath == "" {
@@ -156,21 +158,6 @@ func runWhip(cmd *cobra.Command, args []string) {
log.Ok(fmt.Sprintf("Finished whip in %.1fs", time.Since(whipStartTime).Seconds()))
}

func findPlaybookPath() string {
dir, _ := os.Getwd()
for {
path := filepath.Join(dir, ".whip", "playbook.yml")
if _, err := os.Stat(path); err == nil {
return path
}
parent := filepath.Dir(dir)
if parent == dir {
return ""
}
dir = parent
}
}

func runPlaybookAtHost(job model.Job, t model.TargetName, results chan<- model.TaskResult) {
runStart := time.Now()
if len(job.Playbook) == 0 {
59 changes: 0 additions & 59 deletions cmd/whip/whip_test.go
Original file line number Diff line number Diff line change
@@ -1,60 +1 @@
package main

import (
"os"
"path/filepath"
"testing"
)

func TestFindPlaybookPath(t *testing.T) {
// Create a temporary directory structure
tempDir, err := os.MkdirTemp("", "whip_test")
if err != nil {
t.Fatalf("Failed to create temp dir: %v", err)
}
defer os.RemoveAll(tempDir)

// Resolve any symlinks in tempDir
tempDir, err = filepath.EvalSymlinks(tempDir)
if err != nil {
t.Fatalf("Failed to resolve symlinks in temp dir: %v", err)
}

// Create a .whip directory with a playbook.yml file
whipDir := filepath.Join(tempDir, "subdir", "subsubdir", ".whip")
err = os.MkdirAll(whipDir, 0o755)
if err != nil {
t.Fatalf("Failed to create .whip dir: %v", err)
}
playbookPath := filepath.Join(whipDir, "playbook.yml")
err = os.WriteFile(playbookPath, []byte("dummy content"), 0o644)
if err != nil {
t.Fatalf("Failed to create playbook.yml: %v", err)
}

// Change to the deepest subdirectory
err = os.Chdir(filepath.Join(tempDir, "subdir", "subsubdir"))
if err != nil {
t.Fatalf("Failed to change directory: %v", err)
}

// Run the function
result := findPlaybookPath()

// Check the result
expected := playbookPath
if result != expected {
t.Errorf("Expected path %s, but got %s", expected, result)
}

// Test when no playbook is found
err = os.Chdir("/")
if err != nil {
t.Fatalf("Failed to change to root directory: %v", err)
}

result = findPlaybookPath()
if result != "" {
t.Errorf("Expected empty string when no playbook found, but got %s", result)
}
}
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -14,6 +14,7 @@ require (
github.com/gwillem/urlfilecache v0.0.0-20230402105623-8ef3b7b67c13
github.com/ieee0824/go-deepmerge v0.0.0-20170912170951-7ec7dbbd5a1f
github.com/karrick/gobls v1.3.5
github.com/klauspost/compress v1.17.8
github.com/mitchellh/mapstructure v1.5.0
github.com/nikolalohinski/gonja v1.5.0
github.com/pkg/sftp v1.13.5
@@ -37,7 +38,6 @@ require (
github.com/goph/emperror v0.17.1 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.17.8 // indirect
github.com/konsorten/go-windows-terminal-sequences v1.0.1 // indirect
github.com/kr/fs v0.1.0 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
2 changes: 0 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
@@ -157,8 +157,6 @@ github.com/goph/emperror v0.17.1 h1:6lOybhIvG/BB6VGoWfdv30FVZeZFBBZ9VvgzGXLVkyY=
github.com/goph/emperror v0.17.1/go.mod h1:+ZbQ+fUNO/6FNiUo0ujtMjhgad9Xa6fQL9KhH4LNHic=
github.com/gwillem/go-buildversion v0.0.0-20230402114832-b1addefa8764 h1:GL/gS9rv5F1oBgQwbT+pl0HefNbDZLjbcWmDNT/pUOU=
github.com/gwillem/go-buildversion v0.0.0-20230402114832-b1addefa8764/go.mod h1:yf2A6rXn1ptErQTutA0vpEDZJG6E0b/aUnMnXelY5tU=
github.com/gwillem/go-simplelog v0.3.2-0.20240416140709-da7b4f7d631f h1:4Ny6GTNmMdBMwZYTC9BQKKFZcIoHfg0WoGstBI9Qju8=
github.com/gwillem/go-simplelog v0.3.2-0.20240416140709-da7b4f7d631f/go.mod h1:VzaKnjEPW0JUVseBHP2c4w1kOxMG+TZXLeuT3xWpnbA=
github.com/gwillem/go-simplelog v0.3.2-0.20240425201514-40a6b7d1bcbb h1:Ofmn80/bLvFG9NGVj4sdUVC+n69V/hxPg1UaLuiJPfA=
github.com/gwillem/go-simplelog v0.3.2-0.20240425201514-40a6b7d1bcbb/go.mod h1:VzaKnjEPW0JUVseBHP2c4w1kOxMG+TZXLeuT3xWpnbA=
github.com/gwillem/urlfilecache v0.0.0-20230402105623-8ef3b7b67c13 h1:Kuu6BZRS1uaSatweU/S5pWKkwt+QFCMb2QjJrWcvTWA=
18 changes: 11 additions & 7 deletions internal/vault/age.go
Original file line number Diff line number Diff line change
@@ -9,11 +9,12 @@ import (

"filippo.io/age"
log "github.com/gwillem/go-simplelog"
"github.com/gwillem/whip/internal/fsutil"

Check failure on line 12 in internal/vault/age.go

GitHub Actions / build

no required module provides package github.com/gwillem/whip/internal/fsutil; to add it:

Check failure on line 12 in internal/vault/age.go

GitHub Actions / build

no required module provides package github.com/gwillem/whip/internal/fsutil; to add it:

Check failure on line 12 in internal/vault/age.go

GitHub Actions / build

no required module provides package github.com/gwillem/whip/internal/fsutil; to add it:

Check failure on line 12 in internal/vault/age.go

GitHub Actions / build

no required module provides package github.com/gwillem/whip/internal/fsutil; to add it:

Check failure on line 12 in internal/vault/age.go

GitHub Actions / build

no required module provides package github.com/gwillem/whip/internal/fsutil; to add it:

Check failure on line 12 in internal/vault/age.go

GitHub Actions / build

no required module provides package github.com/gwillem/whip/internal/fsutil; to add it:

Check failure on line 12 in internal/vault/age.go

GitHub Actions / build

no required module provides package github.com/gwillem/whip/internal/fsutil; to add it:
)

const (
ageEnv = "WHIP_KEY"
ageEnvScript = "whip-secret"
ageEnvScript = ".whip/secret.sh"
)

// headerGPG = []byte{0x85, 0x01, 0x8C, 0x03, 0x93, 0xE5, 0x4C, 0x74, 0x67, 0x58, 0x3E, 0x45}
@@ -77,10 +78,13 @@ func (v *ageVault) getID() (id *age.X25519Identity, err error) {
}
keyStr := os.Getenv(ageEnv)
if keyStr == "" {
keyStr, err = readFromScript(ageEnvScript)
if err != nil {
log.Error(err)
return nil, err
if sp := fsutil.FindAncestorPath(ageEnvScript); sp != "" {
log.Debug("Using script", sp, "to generate vault key")
keyStr, err = readFromScript(sp)
if err != nil {
log.Error(err)
return nil, err
}
}
}
if keyStr != "" {
@@ -92,7 +96,7 @@ func (v *ageVault) getID() (id *age.X25519Identity, err error) {
}

id, _ = age.GenerateX25519Identity()
return nil, fmt.Errorf("no $%s set, here's a new one: %s", ageEnv, id)
return nil, fmt.Errorf("no $%s set, here's a new one: %s\n(or create .whip/secret.sh to generate it dynamically)", ageEnv, id)
}

func (v *ageVault) Ready() bool {
@@ -113,7 +117,7 @@ func readFromScript(path string) (string, error) {
return "", fmt.Errorf("script %s is not executable", path)
}

data, err := exec.Command("./" + path).CombinedOutput()
data, err := exec.Command(path).CombinedOutput()
data = bytes.TrimSpace(data)
// fmt.Printf("got '%s'\n", string(data))
if err != nil {
2 changes: 1 addition & 1 deletion internal/vault/convert.go
Original file line number Diff line number Diff line change
@@ -23,7 +23,7 @@ func ConvertAnsibleToWhip(root string) error {
counter := 0

// walk over the root and find ansible encrypted files
err := fsutil.Walk(root, func(path string, info os.FileInfo, err error) error {
err := afs.Walk(root, func(path string, info os.FileInfo, err error) error {
if err != nil || info.IsDir() {
return err
}
2 changes: 1 addition & 1 deletion internal/vault/vault.go
Original file line number Diff line number Diff line change
@@ -21,7 +21,7 @@ var (
allVaulters = []Vaulter{&ageVault{}, &ansibleVault{}}
magicSize = findMagicSize()
fs = afero.NewOsFs()
fsutil = afero.Afero{Fs: fs}
afs = afero.Afero{Fs: fs}
)

const (

0 comments on commit 271cbf4

Please sign in to comment.