Cpd 101 #102

ashcrow · 2017-03-02T19:27:22Z

For discussion per #101

ashcrow · 2017-03-02T19:28:31Z

mbarnes · 2017-03-03T18:37:43Z

doc/cpd/101.rst

+---------
+The likelihood of having a ``Storage`` system that is used only for Commissaire seems
+low. More than likely the same instance will be used for other applications as well.
+By adding encryption to the keys we mitigate could mitigate access from those with


I think "keys we mitigate" is supposed to be "keys we store"?

I should have been 'to the keys, we could mitigate'. Will update.

mbarnes · 2017-03-03T19:01:21Z

doc/cpd/101.rst

+------
+The server(s) would house a GnuPG configuration and keys allowing them to encrypt
+and decrypt data. The servers would either need to have the same key across all
+instances or each server would need a copy of other services public keys.


Can you give more details about what the configuration might look like?

I don't think there would be any configuration updates. However, the public/private keys for use would need to be imported into the keyring at /etc/commissaire/$PLACEWEDECIDEON

mbarnes · 2017-03-03T19:07:39Z

doc/cpd/101.rst

+the key encrypted as long as possible.
+
+The HTTP Server would need to handle decrypting the key back into a base64 encoded
+string before handing it back to the authenticated requestor.


ClusterExecService, InvestigatorService and WatcherService must also be able to decrypt the keys, right?

Hrm, that's a good point. A few things come immediately to mind. We could:

Have the key decrypted at the StorageService level. (Easiest but removes some of the protection since the key may flow over the same backend service)

Have the services request the decrypted key from the REST server

Have services generate and register a gpg key with the server (this would require a new endpoint but would be doable and probably cleaner)

The more I think about it the more I feel it makes sense for StorageService to do the decryption. We'd need to explicitly note in the documentation that the bus level would need to be trusted.

Kombu supports SSL/TLS at the Connection layer for some transports:

SSL compatibility SSL currently only works with the py-amqp, amqplib, and qpid transports. For other transports you can use stunnel.

Perhaps we could pass SSL/TLS options for kombu.Connection from a config file?

source

@mbarnes Yeah, I think that would make sense. We will need to add a quick example on how to use stunnel as well but that is pretty trivial.

ashcrow · 2017-03-06T16:15:30Z

Updated.

ashcrow · 2017-03-16T14:09:45Z

Updated. I need to re-add the stunnel stuff back to it though.

ashcrow · 2017-03-16T14:13:10Z

⬆️

ashcrow · 2017-03-21T14:47:56Z

@mbarnes ⬆️

mbarnes · 2017-03-21T15:32:16Z

doc/cpd/101.rst

-string before handing it back to the requesting service.
+The StorageService would need to to know when to use Custodia versus the configured
+``StorageHandler``(s). It would look at the ``_secrets`` attribute on the instance and,
+if set to ``True`` would use the secrets handler.


This sentence was confusing at first since the _secrets attribute isn't defined until a few paragraphs below.

Fair enough. On next update I'll try to clarify.

mbarnes · 2017-03-21T15:42:18Z

doc/cpd/101.rst

+The models would match based on their primary keys: ``address``.
+
+The ``Model`` class would also gain a new attribute to denote the model is secret. A secret model would
+denote ``_secrets = True``.


Instead of an attribute, I suggest a class hierarchy.

Model | +------+------+ | | | SecretModel | | Host HostCreds

If nothing else, it would make for clearer documentation in APIs where we expect a secret model as opposed to any type of model.

:param secret_model: A model containing sensitive data :type secret_model: commissaire.models.SecretModel

I like that. I'll update.

mbarnes · 2017-03-21T16:10:27Z

doc/cpd/101.rst

+New HTTP Handler
+~~~~~~~~~~~~~~~~
+A new handler called ``secrets`` could be added. This would proxy requests back to the
+Custodia instance.


Okay, so if the "secrets" handler is using a Unix socket to proxy to Custodia, does that imply Custodia has to be running on the same machine as commissaire-server? In the example below, it looks like commissaire-storage-service has the Custodia config, and that service may be running on a different machine than commissaire-server.

Would it make sense for commissaire-storage-service to call back to our own /api/v0/secrets endpoint, instead of accessing Custodia directly?

I feel like I'm not seeing a piece of this clearly.

commissaire-storage-service would call back to /api/v0/secrets. We'd be proxying the requests back to custodia on the server side.

Got'cha. So in that case do we still need the "custodia_api_id" and "custodia_api_key" things in the storage config? ~~It's not clear to me what those are for.~~ Wait, I think I get it now. That's linkage with the Custodia config, right? Didn't look at that part closely.

Correct. Those would be used as the auth into custodia through our endpoint. Since we'd just be proxying access and allowing custodia to handle it's own access we'd need to add the auth changes listed.

Admittedly there are a lot of small changes to get this to work.

True. I think I understand now how this all fits together, conceptually at least.

That's linkage with the Custodia config, right? Didn't look at that part closely.

Yup!

One other piece to this is how commissaire-storage-service learns where the /api/v0/secrets endpoint is. Might be nice if commissaire-http could slip this info to it somehow so we can avoid another line item in storage.conf.

(... or bring back the configuration-in-etcd concept from MVP ...)

ashcrow · 2017-03-21T20:00:54Z

⬆️

mbarnes · 2017-03-21T20:35:46Z

doc/cpd/101.rst

@@ -180,7 +178,7 @@ These are examples and likely will not work without modification.
        handler is used. Raises KeyError if no handler is registered
        for that type of model.
        """
-        if model._secret:
+        if issubclass(model.__class__, models.SecretModel):


simpler: if isinstance(model, models.SecretModel):

mbarnes · 2017-03-21T20:36:44Z

Looking good! 👍

ashcrow · 2017-03-22T13:50:31Z

@mbarnes is there anything else you can think of that needs to be addressed in the CPD?

Of course I'd like to get another eye on it as well before it's ratified /cc @gbraad @portdirect

mbarnes · 2017-03-29T13:13:52Z

(3) [multi-host-mgr-2] Add Custodia to E2E test environment

mbarnes · 2017-03-29T13:18:58Z

(3) [multi-host-mgr-2] Proxy Custodia's REST API

mbarnes · 2017-03-29T13:21:55Z

[spike] [multi-host-mgr-2] Investigate Custodia integration

mbarnes · 2017-03-29T18:46:01Z

Verify a card for installing Custodia is created

@ashcrow: About that ⬆️ ... now that I'm taking a closer look at Custodia, it's not clear to me whether the responsibility of installing Custodia and configuring it correctly falls on Commissaire or our users. Who will own Custodia's config file and how to you envision this all getting set up?

Should we create a container image for commissaire-http + Custodia, configured the way we want (Unix socket, encrypted-etcd backend, etc)? That might significantly simplify the setup instructions.

ashcrow · 2017-03-29T19:33:30Z

@mbarnes I believe the version we use internally should be installed by us ... either via container, pip, or whatever. Exposing other instances would be up ops folks who want to provide them as extra services.

ashcrow · 2017-04-04T15:24:35Z

Per CPD-1 let's go ahead and vote on this. I proposed it so I obviously 👍

@mbarnes PTAL

mbarnes · 2017-04-04T18:37:03Z

Some details may need clarified or changed as we get into the implementation, but I think the CPD looks good as an initial design document. 👍

ashcrow · 2017-04-05T19:32:39Z

By rule this passes. @mbarnes please merge and update to accepted!

ashcrow · 2017-04-12T18:15:16Z

@mbarnes ⬆️

ashcrow · 2017-04-17T13:50:20Z

I'll go ahead and mark it accepted and merge since @mbarnes 👍'd it

mbarnes · 2017-06-07T13:30:42Z

(3) [multi-host-mgr-2] Store service forwards secrets to Custodia

ashcrow added docs enhancement labels Mar 2, 2017

ashcrow requested a review from mbarnes March 2, 2017 19:27

mbarnes reviewed Mar 3, 2017

View reviewed changes

ashcrow mentioned this pull request Mar 13, 2017

DO NOT MERGE: Encryption example for review. #111

Closed

mbarnes reviewed Mar 21, 2017

View reviewed changes

mbarnes mentioned this pull request Mar 24, 2017

Add ListModel subclass #119

Merged

ashcrow force-pushed the cpd-101 branch from 652950e to a75f012 Compare April 17, 2017 13:50

ashcrow added 2 commits April 17, 2017 09:50

doc: Fix CPD-61 version.

1be7b15

doc: Added CPD-101 documentation

3b2f42e

doc: Marking cpd-101 as accepted

53d9b14

ashcrow force-pushed the cpd-101 branch from a75f012 to 53d9b14 Compare April 17, 2017 13:50

ashcrow merged commit 972affd into projectatomic:master Apr 17, 2017

Cpd 101 #102

Cpd 101 #102

Conversation

ashcrow commented Mar 2, 2017 • edited Loading

ashcrow commented Mar 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashcrow commented Mar 6, 2017

ashcrow commented Mar 16, 2017

ashcrow commented Mar 16, 2017

ashcrow commented Mar 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashcrow Mar 21, 2017 • edited Loading

Choose a reason for hiding this comment

mbarnes Mar 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbarnes Mar 21, 2017 • edited Loading

Choose a reason for hiding this comment

ashcrow commented Mar 21, 2017

Choose a reason for hiding this comment

mbarnes commented Mar 21, 2017

ashcrow commented Mar 22, 2017

mbarnes commented Mar 29, 2017

mbarnes commented Mar 29, 2017

mbarnes commented Mar 29, 2017

mbarnes commented Mar 29, 2017 • edited Loading

ashcrow commented Mar 29, 2017

ashcrow commented Apr 4, 2017

mbarnes commented Apr 4, 2017

ashcrow commented Apr 5, 2017

ashcrow commented Apr 12, 2017

ashcrow commented Apr 17, 2017

mbarnes commented Jun 7, 2017

ashcrow commented Mar 2, 2017 •

edited

Loading

ashcrow Mar 21, 2017 •

edited

Loading

mbarnes Mar 21, 2017 •

edited

Loading

mbarnes Mar 21, 2017 •

edited

Loading

mbarnes commented Mar 29, 2017 •

edited

Loading