Vault token (and credentials) active renew of lease during lifetime of Pod to avoid TTL expiration?

[thomas-riccardi]

The vault-agent-in-sidecar does active renewal of token (and/ credentials) lease for the lifetime of the Pod.
cf https://www.vaultproject.io/docs/platform/k8s/injector#renewals-and-updating-secrets and https://www.vaultproject.io/docs/agent/template#renewals-and-updating-secrets

That's especially useful as it allows using workload without any modification: it can read the secrets as env var at startup as usual and never be bothered to dynamically rotate them.

I could not find anything like that in this secret csi provider.
(the closest was #64 talking about token bound to the pod lifetime, but as I understand it it's about kubernetes token, not vault token)

Am I missing something? Is it something that is planned to be added in the future?

Thanks!

[tomhjp]

Hi @thomas-riccardi, thanks for the feedback! We are indeed planning something in this area. The likely route is that we will do #82 first, and then start deploying Vault Agent as a sidecar to the vault-csi-provider daemonset and route all requests though the Agent. Then we can use the same exact mechanism as the sidecar injector.

[jonpjenkins]

To add, as alluded to by @thomas-riccardi , this would be really useful with regards to vault provided dynamic credentials, which can be renewable but would need to be handled by an agent. Thanks much for the work!

[jonpjenkins]

@tomhjp checking in on this - now that #138 closed #82 , is this work able to go forward? This feature would be really useful for our use case.

[tomhjp]

@jonpjenkins yep I've been thinking about this quite a bit lately. Adding lease renewal should now be relatively straightforward, and I think it probably won't even require any further changes in this repo - it should all be deployment changes which we'll help automate in hashicorp/vault-helm.

One slight wrinkle as far as caching is concerned though: the cache entry key depends on the Vault token, but because we generate a fresh k8s token on every mount request, the provider will never get a cache hit from Agent. That means when auto rotation is used (via the flag on the driver), a fresh secret will be fetched on every re-mount. I'm wondering about a new Agent feature to help fix this, which could perhaps rely on JWT validation because even if the token isn't stable, the identity is, so that could be a verifiably correct way to allow cache hits from different tokens. But I haven't explored that in depth yet.

[tomhjp]

For anyone following this issue, #163 and hashicorp/vault-helm#749 are relevant. It's not quite the whole picture, but they should be good steps forward when merged.

[LosAngeles971]

Hello @tomhjp,
hope everything is fine.
I am also interested on the issue raised by @thomas-riccardi; if possibile, is there a time estimation for the realization of that feature?
Thanks so much,
Angelo.

[jonpjenkins]

@tomhjp I wanted to bump the above ask - the question of time estimation. I am on the cusp of writing a work-around for now until this is resolved.

[jonpjenkins]

@tomhjp Thanks for getting #163 taken care of 🎉 🥇

Is that #749 WIP close to being complete?

[tomhjp]

Thanks for the ping, it is! I've just pushed some updates to that PR and requested review.

[jonpjenkins]

@tomhjp I saw that, thanks so much!

[jonpjenkins]

@tomhjp I see the referenced PR is merged, and the agent is included by default now -- thanks! 🎉 🙌

Would you be wiling to offer your thoughts on what would be left yet to do on this front?

[tomhjp]

I think #202 probably wraps up this issue. Alongside the Agent sidecar deployed by vault-helm, it will mean that tokens and secret leases are always renewed by Vault Agent for as long as possible, and each pod will only have one dynamic lease generated at a time per requested secret. There are some other TTL-related issues like #149 and #151 that won't be fixed, but as this issue is primarily about TTL renewal, I think I'll close this with #202.