Consult

Generate configuration and secrets for Rails apps automatically from Consul & Vault.

Background

This gem is a spiritual sibling to Consul Template, but specifically intended for use in Ruby/Rails environments. It does not have the same features as Consul Template; it is intended for simpler scenarios. Most importantly, leases and configuration changes are not watched to automatically re-render. Consult is intended for more static or medium-to-long lived application configuration.

We use Consul Template for server level configuration, but application level configuration is more tricky. It is difficult to solve the problem of fetching configuration and secrets in a consistent way in development, staging, and production. For example, we wanted to avoid having Consul Template used in production, but some other custom solution in development.

With Consult the process is the same in all environments.

Installation

Add this line to your application's Gemfile:

gem 'consult'

And then execute:

$ bundle

Or install it yourself as:

$ gem install consult

Usage

Using Consult requires a configuration YAML file and a series of template files. The configuration file serves as a manifest of templates and their settings, along with optional connection settings to Vault and Consul.

Pre-existing copies of files generated by Consult (such as secrets.yml, database.yml, etc) should be removed from your app's source control and added to your .gitignore. Only keep your templates in source control, not the generated files!

If this gem is included in a Rails, the templates will render on Rails boot. Configuration or credential changes can be picked up by restarting your app.

CLI

Render templates on demand with the CLI. By default, this will bypass template TTLs to force rendering and provide verbose output. See consult --help for options.

$ bundle exec consult
Consult: Rendered my_config
Consult: Rendered secrets

The Consult CLI is also available via Docker:

$ docker run --rm -v .:/app veracross/consult:latest --directory /app

If your templates reference localhost (such as the templates in the spec directory of this repo), add --net host to the command.

Configuration

# Optional; Consult will render this specific environment, if set
# Defaults to ENV['RAILS_ENV'] or Rails.env if Rails is present
env: test

# "shared" is the base configuration used for all environments by default
# note: you do NOT need to use yaml merge syntax to have shared configuration included for specific environments
shared:
  # Optional
  consul:
    # Prefers `CONSUL_HTTP_ADDR` environment variable
    address: http://0.0.0.0:8500
    # Prefers `CONSUL_HTTP_TOKEN` environment variable, or a ~/.consul-token file.
    # Setting a token here is not best practice because consul tokens should have a relatively short TTL
    # and be read from the environment, but this is convenient for testing.
    token: 5d3f1c66-d405-4ad1-b634-ea30be4fb539

  # Optional
  vault:
    # Prefers `VAULT_ADDR` environment variable
    address: http://0.0.0.0:8200
    # Prefers `VAULT_TOKEN` environment variable, or a ~/.vault-token file
    # Setting a token here is not best practice because vault tokens should have a relatively short TTL
    # and be read from the environment, but this is convenient for testing.
    token: 8fcd5aed-3eb9-412d-8923-1397af7aede2

  # Enumerate the templates.
  templates:
    database:
      # Relative paths are assumed to be in #{Rails.root}.
      # Path to the template
      path: config/templates/database.yml
      # Destination for the rendered template
      dest: config/database.yml
      # If the file is less than this old, do not re-render
      ttl: 3600 # seconds

# environment specific configuration
# NOTE: environment keys will be deep merged with the "shared" configuration
test:
  templates:
    secrets:
      path: config/templates/secrets.yml
      dest: config/secrets.yml
      # vars can be defined on a per-template basis
      vars:
        test_specific_key: and_the_value

    extra_test_config:
      # normally there's an error for missing templates, but this can be allowed via config
      skip_missing_template: true
      # config files are also processed through ERB, so paths can be made dynamic
      path: config/templates/<%= ENV['extra_test_file'] %>.yml
      dest: config/extra_test_config.yml

production:
  # vars can be defined at the environment level, which are available to these templates
  vars:
    hello: world

  templates:
    # You can concatenate multiple files together
    my_config:
      paths:
        - config/templates/one.yml
        - config/templates/two.yml
      dest: config/my_config.yml

    # Templates can come from Consul
    your_config:
      consul_keys:
        - some/consul/key
        - another/consul/key
      dest: config/your_config.txt

Templates

Templates files are processed with ERB. As such, they can do anything ERB can do. Consult also provides a few helper functions.

Note that under the hood, Consult is using Diplomat and the Vault Gem. Consul objects are therefore Diplomat objects, and likewise Vault objects are Vault Gem objects. See their API docs for more information. Diplomat generally returns structs with title cased properties.

Consul Functions

service(name) - Fetch the nodes for the specified service.

<% service("redis").each do |node| %>
host: <%= node.Address %>
port: <%= node.ServicePort %>
<% end %>

returns

host: redis1.local
port: 6379

query(name_or_id, options: nil) - Execute the specified prepared Query by name or ID

<% query('pg-production').tap do |result| %>
  service: <%= result.Service %>
  nodes:
  <% result.Nodes.each do |node| %>
    address: <%= node['Node']['Address']
  <% end %>
<% end %>

query_nodes(name_or_id, options: nil) - Return only the nodes from a prepared query

<% query_nodes('pg-production').each do |node| %>
<%= node['Node'] %>:
  host: <%= node['Address'] %>
  datacenter: <%= node['Datacenter'] %>
<% end %>

pg1:
  host: 10.0.100.101
  datacenter: us-east-1
pg2:
  host: 10.0.100.102
  datacenter: us-east-2

key(key, options: nil, not_found: :reject, found: :return) - Return value of the given key

'<% key('apps/infrastructure/node/dns') %>':
<<: *common
  host: <%= key('apps/infrastructure/node/dns') %>
  port: 1433

'db1':
<<: *common
  host: db1
  port: 1433

Vault Functions

secret(path) - Fetch a secret at the given path.

# Vault KV v2
username: <%= secret('secret/data/credentials').data.dig(:data, :username) %>

# Vault KV v1
username: <%= secret('secret/credentials').data[:username] %>

yields

username: kylo.ren

secrets(path) - List all secrets at the given path

<% secrets('secret').each do |path| %>
  <%= path %>
<% end %>

yields

foo
bar
baz

Utility Functions

timestamp - Renders the current utc timestamp.

<%= timestamp %>

renders

2018-02-23 14:20:29 UTC

indent(string, level, separator = '\n') - Indents a multi-line string by level

keys:
  multi_line: |
<%= indent secret('secret/keys/multi_line).data[:value], 4 %>

renders

keys:
  multi_line: |
    30ada39cccf79aadbd1d870bc15f0086
    7ea8d734e81e9c6710faa15b0aff516c
    27778ab3b1e10db2028352f12c3c07bb
    e7ec40d1e45834681b4dc3548230d1ca

with(whatever) - takes whatever and yields it back. Equivalent to tap, but provided as a bridge from [Consul Template]/Go template conventions.

<% with secret "secrets/credentials" do |s| %>
username: <%= s.data[:username] %>
password: <%= s.data[:password] %>
<% end %>

More Full Examples

Render multiple servers into a database.yml file, keyed by their name.

# database.yml
<% service("postgres").each do |node| %>
'<%= node.Node %>':
  host: <%= node.Address %>
  port: <%= node.ServicePort %>
  <%- with secret "secret/base/sql-server/#{node.Node}/web" do |s| -%>
  # Credential lease good until <%= (timestamp + s.lease_duration).to_s %>
  username: <%= s.data[:username] %>
  password: <%= s.data[:password] %>
  <% end -%>
<% end %>

Yields something like

# database.yml
'db1':
  host: 10.0.100.101
  port: 5432
  # Credential lease good until 2018-02-24 16:08:29 UTC
  username: foo
  password: bar
'db2':
  host: 10.0.100.102
  port: 5432
  # Credential lease good until 2018-02-24 16:08:29 UTC
  username: baz
  password: qux

Secrets

# secrets.yml
shared:
  rollbar_token: <%= secret('secrets/third_party').data[:rollbar] %>
  scout_token: <%= secret('secrets/third_party').data[:scout] %>

development:
  secret_key_base: abcd1234....

production:
  secret_key_base: <%= secret('secret/apps/myapp').data[:secret_key_base] %>

Then reference secrets in your app with Rails.application.secrets.

# config/initializers/rollbar.rb
Rollbar.configure do |config|
  config.access_token = Rails.application.secrets.rollbar_token
end

Development

After checking out the repo, run bin/setup to install dependencies. You can also run bin/console for an interactive prompt that will allow you to experiment. See below for testing instructions.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Testing

Testing is easiest by running Consul and Vault in Docker. Just boot up their minimal containers:

$ docker-compose up

Then run bundle exec rspec, or bundle exec guard.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/veracross/consult.

License

The gem is available as open source under the terms of the MIT License.