Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to encrypt and decrypt from stdin #1690

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Allow to encrypt and decrypt from stdin #1690

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,33 +359,33 @@ Encrypting and decrypting from other programs
When using ``sops`` in scripts or from other programs, there are often situations where you do not want to write
encrypted or decrypted data to disk. The best way to avoid this is to pass data to SOPS via stdin, and to let
SOPS write data to stdout. By default, the encrypt and decrypt operations write data to stdout already. To pass
data via stdin, you need to pass ``/dev/stdin`` as the input filename. Please note that this only works on
Unix-like operating systems such as macOS and Linux. On Windows, you have to use named pipes.
data via stdin, you need to not provide an input filename. For encrpytion, you also must provide the
``--filename-override`` option with the file's filename. The filename will be used to determine the input and output
types, and to select the correct creation rule.

To decrypt data, you can simply do:

.. code:: sh

$ cat encrypted-data | sops decrypt /dev/stdin > decrypted-data
$ cat encrypted-data | sops decrypt --filename-override filename.yaml > decrypted-data

To control the input and output format, pass ``--input-type`` and ``--output-type`` as appropriate. By default,
``sops`` determines the input and output format from the provided filename, which is ``/dev/stdin`` here, and
``sops`` determines the input and output format from the provided filename, which is the empty string here, and
thus will use the binary store which expects JSON input and outputs binary data on decryption.

For example, to decrypt YAML data and obtain the decrypted result as YAML, use:

.. code:: sh

$ cat encrypted-data | sops decrypt --input-type yaml --output-type yaml /dev/stdin > decrypted-data
$ cat encrypted-data | sops decrypt --input-type yaml --output-type yaml > decrypted-data

To encrypt, it is important to note that SOPS also uses the filename to look up the correct creation rule from
``.sops.yaml``. Likely ``/dev/stdin`` will not match a creation rule, or only match the fallback rule without
``path_regex``, which is usually not what you want. For that, ``sops`` provides the ``--filename-override``
parameter which allows you to tell SOPS which filename to use to match creation rules:
``.sops.yaml``. Therefore, you must provide the ``--filename-override`` parameter which allows you to tell
SOPS which filename to use to match creation rules:

.. code:: sh

$ echo 'foo: bar' | sops encrypt --filename-override path/filename.sops.yaml /dev/stdin > encrypted-data
$ echo 'foo: bar' | sops encrypt --filename-override path/filename.sops.yaml > encrypted-data

SOPS will find a matching creation rule for ``path/filename.sops.yaml`` in ``.sops.yaml`` and use that one to
encrypt the data from stdin. This filename will also be used to determine the input and output store. As always,
Expand All @@ -394,7 +394,7 @@ the input store type can be adjusted by passing ``--input-type``, and the output

.. code:: sh

$ echo foo=bar | sops encrypt --filename-override path/filename.sops.yaml --input-type dotenv /dev/stdin > encrypted-data
$ echo foo=bar | sops encrypt --filename-override path/filename.sops.yaml --input-type dotenv > encrypted-data


Encrypting using Hashicorp Vault
Expand Down Expand Up @@ -1237,7 +1237,7 @@ When operating on stdin, use the ``--input-type`` and ``--output-type`` flags as

.. code:: sh

$ cat myfile.json | sops decrypt --input-type json --output-type json /dev/stdin
$ cat myfile.json | sops decrypt --input-type json --output-type json

JSON and JSON_binary indentation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down
28 changes: 22 additions & 6 deletions cmd/sops/common/common.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package common

import (
"fmt"
"io"
"os"
"path/filepath"
"time"
Expand Down Expand Up @@ -130,11 +131,20 @@ func EncryptTree(opts EncryptTreeOpts) error {
return nil
}

// LoadEncryptedFile loads an encrypted SOPS file, returning a SOPS tree
func LoadEncryptedFile(loader sops.EncryptedFileLoader, inputPath string) (*sops.Tree, error) {
fileBytes, err := os.ReadFile(inputPath)
if err != nil {
return nil, NewExitError(fmt.Sprintf("Error reading file: %s", err), codes.CouldNotReadInputFile)
// LoadEncryptedFileEx loads an encrypted SOPS file from a file or stdin, returning a SOPS tree
func LoadEncryptedFileEx(loader sops.EncryptedFileLoader, inputPath string, readFromStdin bool) (*sops.Tree, error) {
var fileBytes []byte
var err error
if readFromStdin {
fileBytes, err = io.ReadAll(os.Stdin)
if err != nil {
return nil, NewExitError(fmt.Sprintf("Error reading from stdin: %s", err), codes.CouldNotReadInputFile)
}
} else {
fileBytes, err = os.ReadFile(inputPath)
if err != nil {
return nil, NewExitError(fmt.Sprintf("Error reading file: %s", err), codes.CouldNotReadInputFile)
}
}
path, err := filepath.Abs(inputPath)
if err != nil {
Expand All @@ -145,6 +155,11 @@ func LoadEncryptedFile(loader sops.EncryptedFileLoader, inputPath string) (*sops
return &tree, err
}

// LoadEncryptedFile loads an encrypted SOPS file, returning a SOPS tree
func LoadEncryptedFile(loader sops.EncryptedFileLoader, inputPath string) (*sops.Tree, error) {
return LoadEncryptedFileEx(loader, inputPath, false)
}

// NewExitError returns a cli.ExitError given an error (wrapped in a generic interface{})
// and an exit code to represent the failure
func NewExitError(i interface{}, exitCode int) *cli.ExitError {
Expand Down Expand Up @@ -227,6 +242,7 @@ type GenericDecryptOpts struct {
Cipher sops.Cipher
InputStore sops.Store
InputPath string
ReadFromStdin bool
IgnoreMAC bool
KeyServices []keyservice.KeyServiceClient
DecryptionOrder []string
Expand All @@ -235,7 +251,7 @@ type GenericDecryptOpts struct {
// LoadEncryptedFileWithBugFixes is a wrapper around LoadEncryptedFile which includes
// check for the issue described in https://github.com/mozilla/sops/pull/435
func LoadEncryptedFileWithBugFixes(opts GenericDecryptOpts) (*sops.Tree, error) {
tree, err := LoadEncryptedFile(opts.InputStore, opts.InputPath)
tree, err := LoadEncryptedFileEx(opts.InputStore, opts.InputPath, opts.ReadFromStdin)
if err != nil {
return nil, err
}
Expand Down
12 changes: 7 additions & 5 deletions cmd/sops/decrypt.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ type decryptOpts struct {
InputStore sops.Store
OutputStore sops.Store
InputPath string
ReadFromStdin bool
IgnoreMAC bool
Extract []interface{}
KeyServices []keyservice.KeyServiceClient
Expand All @@ -27,11 +28,12 @@ type decryptOpts struct {

func decryptTree(opts decryptOpts) (tree *sops.Tree, err error) {
tree, err = common.LoadEncryptedFileWithBugFixes(common.GenericDecryptOpts{
Cipher: opts.Cipher,
InputStore: opts.InputStore,
InputPath: opts.InputPath,
IgnoreMAC: opts.IgnoreMAC,
KeyServices: opts.KeyServices,
Cipher: opts.Cipher,
InputStore: opts.InputStore,
InputPath: opts.InputPath,
ReadFromStdin: opts.ReadFromStdin,
IgnoreMAC: opts.IgnoreMAC,
KeyServices: opts.KeyServices,
})
if err != nil {
return nil, err
Expand Down
26 changes: 18 additions & 8 deletions cmd/sops/encrypt.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package main

import (
"fmt"
"io"
"os"
"path/filepath"

Expand All @@ -27,11 +28,12 @@ type encryptConfig struct {
}

type encryptOpts struct {
Cipher sops.Cipher
InputStore sops.Store
OutputStore sops.Store
InputPath string
KeyServices []keyservice.KeyServiceClient
Cipher sops.Cipher
InputStore sops.Store
OutputStore sops.Store
InputPath string
ReadFromStdin bool
KeyServices []keyservice.KeyServiceClient
encryptConfig
}

Expand Down Expand Up @@ -78,9 +80,17 @@ func metadataFromEncryptionConfig(config encryptConfig) sops.Metadata {

func encrypt(opts encryptOpts) (encryptedFile []byte, err error) {
// Load the file
fileBytes, err := os.ReadFile(opts.InputPath)
if err != nil {
return nil, common.NewExitError(fmt.Sprintf("Error reading file: %s", err), codes.CouldNotReadInputFile)
var fileBytes []byte
if opts.ReadFromStdin {
fileBytes, err = io.ReadAll(os.Stdin)
if err != nil {
return nil, common.NewExitError(fmt.Sprintf("Error reading from stdin: %s", err), codes.CouldNotReadInputFile)
}
} else {
fileBytes, err = os.ReadFile(opts.InputPath)
if err != nil {
return nil, common.NewExitError(fmt.Sprintf("Error reading file: %s", err), codes.CouldNotReadInputFile)
}
}
branches, err := opts.InputStore.LoadPlainFile(fileBytes)
if err != nil {
Expand Down
61 changes: 39 additions & 22 deletions cmd/sops/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -707,8 +707,8 @@ func main() {
},
{
Name: "decrypt",
Usage: "decrypt a file, and output the results to stdout",
ArgsUsage: `file`,
Usage: "decrypt a file, and output the results to stdout. If no filename is provided, stdin will be used.",
ArgsUsage: `[file]`,
Flags: append([]cli.Flag{
cli.BoolFlag{
Name: "in-place, i",
Expand Down Expand Up @@ -736,7 +736,7 @@ func main() {
},
cli.StringFlag{
Name: "filename-override",
Usage: "Use this filename instead of the provided argument for loading configuration, and for determining input type and output type",
Usage: "Use this filename instead of the provided argument for loading configuration, and for determining input type and output type. Required when reading from stdin.",
},
cli.StringFlag{
Name: "decryption-order",
Expand All @@ -748,19 +748,24 @@ func main() {
if c.Bool("verbose") {
logging.SetLevel(logrus.DebugLevel)
}
if c.NArg() < 1 {
return common.NewExitError("Error: no file specified", codes.NoFileSpecified)
if c.NArg() == 0 && c.Bool("in-place") {
return common.NewExitError("Error: cannot use --in-place when reading from stdin", codes.ErrorConflictingParameters)
}
warnMoreThanOnePositionalArgument(c)
if c.Bool("in-place") && c.String("output") != "" {
return common.NewExitError("Error: cannot operate on both --output and --in-place", codes.ErrorConflictingParameters)
}
fileName, err := filepath.Abs(c.Args()[0])
if err != nil {
return toExitError(err)
}
if _, err := os.Stat(fileName); os.IsNotExist(err) {
return common.NewExitError(fmt.Sprintf("Error: cannot operate on non-existent file %q", fileName), codes.NoFileSpecified)
readFromStdin := c.NArg() == 0
var fileName string
var err error
if !readFromStdin {
fileName, err = filepath.Abs(c.Args()[0])
if err != nil {
return toExitError(err)
}
if _, err := os.Stat(fileName); os.IsNotExist(err) {
return common.NewExitError(fmt.Sprintf("Error: cannot operate on non-existent file %q", fileName), codes.NoFileSpecified)
}
}
fileNameOverride := c.String("filename-override")
if fileNameOverride == "" {
Expand Down Expand Up @@ -791,6 +796,7 @@ func main() {
OutputStore: outputStore,
InputStore: inputStore,
InputPath: fileName,
ReadFromStdin: readFromStdin,
Cipher: aes.NewCipher(),
Extract: extract,
KeyServices: svcs,
Expand Down Expand Up @@ -832,8 +838,8 @@ func main() {
},
{
Name: "encrypt",
Usage: "encrypt a file, and output the results to stdout",
ArgsUsage: `file`,
Usage: "encrypt a file, and output the results to stdout. If no filename is provided, stdin will be used.",
ArgsUsage: `[file]`,
Flags: append([]cli.Flag{
cli.BoolFlag{
Name: "in-place, i",
Expand Down Expand Up @@ -911,26 +917,36 @@ func main() {
},
cli.StringFlag{
Name: "filename-override",
Usage: "Use this filename instead of the provided argument for loading configuration, and for determining input type and output type",
Usage: "Use this filename instead of the provided argument for loading configuration, and for determining input type and output type. Required when reading from stdin.",
},
}, keyserviceFlags...),
Action: func(c *cli.Context) error {
if c.Bool("verbose") {
logging.SetLevel(logrus.DebugLevel)
}
if c.NArg() < 1 {
return common.NewExitError("Error: no file specified", codes.NoFileSpecified)
if c.NArg() == 0 {
if c.Bool("in-place") {
return common.NewExitError("Error: cannot use --in-place when reading from stdin", codes.ErrorConflictingParameters)
}
if c.String("filename-override") == "" {
return common.NewExitError("Error: must specify --filename-override when reading from stdin", codes.ErrorConflictingParameters)
}
}
warnMoreThanOnePositionalArgument(c)
if c.Bool("in-place") && c.String("output") != "" {
return common.NewExitError("Error: cannot operate on both --output and --in-place", codes.ErrorConflictingParameters)
}
fileName, err := filepath.Abs(c.Args()[0])
if err != nil {
return toExitError(err)
}
if _, err := os.Stat(fileName); os.IsNotExist(err) {
return common.NewExitError(fmt.Sprintf("Error: cannot operate on non-existent file %q", fileName), codes.NoFileSpecified)
readFromStdin := c.NArg() == 0
var fileName string
var err error
if !readFromStdin {
fileName, err = filepath.Abs(c.Args()[0])
if err != nil {
return toExitError(err)
}
if _, err := os.Stat(fileName); os.IsNotExist(err) {
return common.NewExitError(fmt.Sprintf("Error: cannot operate on non-existent file %q", fileName), codes.NoFileSpecified)
}
}
fileNameOverride := c.String("filename-override")
if fileNameOverride == "" {
Expand All @@ -955,6 +971,7 @@ func main() {
OutputStore: outputStore,
InputStore: inputStore,
InputPath: fileName,
ReadFromStdin: readFromStdin,
Cipher: aes.NewCipher(),
KeyServices: svcs,
encryptConfig: encConfig,
Expand Down
Loading
Loading