Credstash - Upstream changes #144

nathan-muir · 2017-03-13T23:38:15Z

Hi there,

I started a fork of credstash a few months ago, so I could quickly iterate on a few concepts.

I've been using it in production for a while now, and thought it would be good to contribute back some of the features that were developed.

The main goal for the fork:

Allow the KMS & DynamoDB config to persist in a config file; eg. /etc/credsmash.cfg
Add support for template interpolation with secret data (see credsmash render-templates)

On the way: (See History.md)

Add support for binary data
Switch from pycrypto to cryptography & fix a few issues in aes_ctr (already upstreamed in Cryptography - Switch from pycrypto to cryptography #116 )
Added support for aes-gcm
Converted the CLI from argparse to click
Pluggable Key Services
Pluggable Storage Services
Created a better API for using directly in python

Obviously there was a big departure from the credstash codebase, so I did start work on re-using the argparse implementation with the new internal api's - 3stack-software#4

So there's a lot going on here, it would be great to discuss and see what can be made suitable to upstream.

My biggest fear would be for a change like #136 to cause unnecessary divergence.

The goal was to clean up some of the CLI, and to modularise the api for calling kms/dynamodb. This will let us add new features like setting default options from a configuration file, and writing configuration templates. Fixes fugue#93 allowing boto3/botocore to configure itself. Fixes fugue#79 as file input is the default for put/put-many Closes fugue#88 as `put` only allows for files , so you must use `printf`/`echo` to put any value.

Providing `--table-name` or `--key-id` will take precedence over the `credsmash.cfg`.

Manually adding `--context key value` will only append each key-value pair.

This was necessary so we can add support for other encryption schemes, and fix-up support for binary files; without breaking existing stored secrets.

Fixes fugue#96 Also, we now expose the `nonce` being used for CTR mode- see fugue#75. To be resolved in a future update.

…dom nonces. Fixes fugue#73 by no longer treating all values as unicode. Fixes fugue#75 by generating & storing a unique nonce.

Also move configuration of encryption to `credsmash.cfg`, so the `--digest` parameter has been removed from `put`.

…sh` data. Adds the new `render-template` and `render-templates` commands.

…_service`.

….storage_service`.

…has changed. You can avoid this behaviour by using `--version`, or `--no-compare`.

… the template renders successfully.

The new naming makes it's purpose clearer.

You can now have full control over the output of your templates.

Reproduces #3

Credsmash was using the wrong output key. Resolves #3

davbo · 2017-03-22T17:09:36Z

@nathan-muir Thanks for flagging up the risk of backward incompatibility from #136.

Let me know if I can contribute anything to your fork to allow for such a change. I'm keen to see some default authenticated data included in credstash, either directly contributed to credstash or by way adding it to your fork.

nathan-muir · 2017-03-23T01:53:11Z

@davbo Initially I was a little hesitant to rely on another AWS/KMS feature - in favour of supporting AEAD in AES-GCM directly, but after some thought there isn't any substantial reasons to suggest that change.

After a bit of review, I would only suggest the following changes:

Don't use a padded-int for the version field. (In this fork) it is an implementation detail of DynamoDB storage, and the rest of the code sees a regular int.
Rather than using try/except to detect the format of secrets, consider adding an attribute to the object, eg. {"aead": "v1"}/ {"protocol": "2"} which tells us how to properly decode it.

Relying on exceptions in a specific component becomes hard to refactor later into clean/separate components.

collisdigital · 2017-04-11T11:26:53Z

We're looking to use credstash and support for AES-GCM-256 would be very welcome along with some of the other changes from this fork. @alex-luminal this is quite a large PR; is it likely to merge in the near future or would we be better splitting out some specific things (like GCM support) into a smaller PR to get those things in sooner?

`pip` can behave unexpectedly with some version constraints. 1. It will abort when constraints aren't satisfiable using the highest version 2. It will not consider multiple constraints on a single dependency In general, it is safer to remove constraints from setup.py, and allow users to ensure install which-ever dependencies work.

gene1wood · 2017-09-17T22:55:36Z

@collisdigital I would suspect the new features in this PR would be much more likely to get merged if they were pulled out into their own PRs. If there's a feature in this PR that you'd like to see, breaking it into it's own PR is probably a good path.

collisdigital · 2017-09-18T07:49:23Z

@gene1wood thanks, to be honest this has fallen off our radar now as we decided to go with a different solution in the end.

As per documentation https://cryptography.io/en/latest/hazmat/primitives/cryptographic-hashes/

wayne-luminal · 2017-10-25T19:15:41Z

@nathan-muir Apologies for taking so long to get back to you. Thanks for the awesome work! I'm in the process of reviewing all these PRs and my gameplan for all these PRs is basically to bring everything into an integration branch (assuming it's suitable to merge upstream) and ultimately release a new major version of credstash. I am documenting breaking changes like you correctly pointed out for #136 and intend to provide upgrade instructions.

Overall your PR looks like it's something we can pull most of as it has a few things I intended to do anyway.

- Escape for systemd envfiles with `env` - Minify a JSON block with `minjson`

Consolidate api/cli into a single core module. It's now much easier to load & call credsmash via python. Core - Perform version/compare before sealing new secret. DynamoDB - Create separate `get_one` and `get_latest`. This also removes the `name` & `version` from the ciphertext output, as it is an implementation detail of how DynamoDB stores data.

Crypto - Support additional authenticated data on key service

nathan-muir added 30 commits September 28, 2016 13:26

CLI - Allow options to override configuration file

ed8912d

Providing `--table-name` or `--key-id` will take precedence over the `credsmash.cfg`.

CLI - Allow encryption-context to be read from configuration file.

f50c662

Manually adding `--context key value` will only append each key-value pair.

Cryptography - Extract encryption scheme into credsmash.aes_ctr

031db4e

This was necessary so we can add support for other encryption schemes, and fix-up support for binary files; without breaking existing stored secrets.

Cryptography - Switch from pycrypto to cryptography

2299a05

Fixes fugue#96 Also, we now expose the `nonce` being used for CTR mode- see fugue#75. To be resolved in a future update.

Cleanup get_secret

4e06ac4

Cryptography - Add new aes-ctr mode that supports binary & uses ran…

dc9db5a

…dom nonces. Fixes fugue#73 by no longer treating all values as unicode. Fixes fugue#75 by generating & storing a unique nonce.

Cryptography - Add new aes-gcm mode.

7021485

Also move configuration of encryption to `credsmash.cfg`, so the `--digest` parameter has been removed from `put`.

Update requirements.txt

228d8c8

credsmash put - Ensure that unicode is converted to bytes / str

5409d4a

Support writing shell & json script templates, directly from `credsma…

63547d3

…sh` data. Adds the new `render-template` and `render-templates` commands.

Improve setup.py and add deployment configuration to travis-ci

88e124e

Add test to show crypto working

1245e0a

Travis-CI - Upgrade pip

4cdb88a

Bump version: 2.0.0rc.0 → 2.0.0

2348e9d

CLI - use stdout for list

881a34d

CLI - Autodetect input & output formats where possible (from filename)

1b99568

YAML - Use safe_dump to prevent !!python/unicode annotations.

7651966

Bring highest_version lookup into put.

a3eb7ee

Move seal/open code into credsmash.crypto

c7491fa

Allow for alternative key services via the entry point `credsmash.key…

cfc7a2f

…_service`.

Allow for alternative storage services via the entry point `credsmash…

081ae66

….storage_service`.

CLI - credsmash put will only auto-increment a secret if the value …

c7a9fb4

…has changed. You can avoid this behaviour by using `--version`, or `--no-compare`.

Use entry points to allow additional cli commands to be loaded.

0f26134

Bump version: 2.0.0 → 2.1.0

d56796a

Ensure render-templates only creates/truncates the "destination" if…

b15a1e8

… the template renders successfully.

Allow render-templates to write binary secrets.

7608380

Rename option --data-file to --secrets-file on render-template

f2fda49

The new naming makes it's purpose clearer.

Add option --template-vars to render-template.

d4adced

You can now have full control over the output of your templates.

Templates - Add support for in operator.

e402afb

nathan-muir added 6 commits October 13, 2016 15:07

Bump version: 2.1.0 → 2.2.0

dc7ea77

Fix bug with render-template

b651e01

Bump version: 2.2.0 → 2.2.1

f3d4bfe

Add test using digest-method for aes-ctr

3ac534f

Reproduces #3

Fix issue when using AES-CTR with non-default digest method

6a5f59f

Credsmash was using the wrong output key. Resolves #3

Bump version: 2.2.1 → 2.2.2

2e8f1e4

nathan-muir added 2 commits July 18, 2017 10:20

Bump version: 2.2.2 → 2.2.3

1eb37df

nathan-muir added 2 commits October 12, 2017 09:16

Remove all deprecated hashes

fdc9cfc

As per documentation https://cryptography.io/en/latest/hazmat/primitives/cryptographic-hashes/

Bump version: 2.2.3 → 2.2.4

c9931c5

This was referenced Oct 31, 2017

A configuration file to store settings to address different vaults #131

Open

Allow for custom backends #11

Open

nathan-muir added 5 commits August 22, 2018 17:28

Update .travis.yml

8be7425

templates: add env & minjson filters

88ad55d

- Escape for systemd envfiles with `env` - Minify a JSON block with `minjson`

Bump version: 2.2.4 → 2.2.5

654b59c

put: check version/comparison before sealing new secret

ef4273b

dynamodb: add support for large table scans

59beffe

nathan-muir force-pushed the credstash branch from 03b0dd3 to 5931378 Compare March 13, 2019 05:06

nathan-muir added 5 commits March 13, 2019 15:41

Fix python3 compatibility issues

b29a909

Bump version: 2.2.5 → 2.3.0

ca49f10

Update KMS to support Additional Authenticated Data

b232908

Crypto - Support additional authenticated data on key service

Create a credstash command that is compatible with the original CLI

6d26ce3

nathan-muir force-pushed the credstash branch from 5931378 to 6d26ce3 Compare March 13, 2019 05:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Credstash - Upstream changes #144

Credstash - Upstream changes #144

nathan-muir commented Mar 13, 2017 •

edited

Loading

davbo commented Mar 22, 2017

nathan-muir commented Mar 23, 2017 •

edited

Loading

collisdigital commented Apr 11, 2017

gene1wood commented Sep 17, 2017

collisdigital commented Sep 18, 2017

wayne-luminal commented Oct 25, 2017

Credstash - Upstream changes #144

Are you sure you want to change the base?

Credstash - Upstream changes #144

Conversation

nathan-muir commented Mar 13, 2017 • edited Loading

davbo commented Mar 22, 2017

nathan-muir commented Mar 23, 2017 • edited Loading

collisdigital commented Apr 11, 2017

gene1wood commented Sep 17, 2017

collisdigital commented Sep 18, 2017

wayne-luminal commented Oct 25, 2017

nathan-muir commented Mar 13, 2017 •

edited

Loading

nathan-muir commented Mar 23, 2017 •

edited

Loading