Vault Secret Management System

Vault is a secret management system provided by Hashicorp.  Like many Hashi tools it seeks to fill a hole in the infrastructure management space that can be used with their other tools or independent of them.

With Vault you can write arbitrary key value pairs to a path in the system and they will be encrypted.  Policies can be implemented around paths which restrict access to them by different parameters.  There is also a plugin system for authentication which leverages the most popular authentication mechanisms available.  

Secret Backends

While the system can manage key value pairs it can do something more interesting than that.  You can mount different Vault backends which actually provision access keys when they are queried.  So instead of storing your AWS access keys in the Vault database you can have the AWS backend create them for you when requested.  The keys returned can have a TTL associated with them so that you have dynamically rotating keys managed automatically for you.  These secret backends support most major databases as well as certificates and ssh.  

Storage Backends

The key value store can be backed by several different engines as well.  Consul from Hashi is the most common but a simple file can be used, etcd, zookeeper, and many others.


As it stands today Vault doesn’t support custom backends so if there is a special type of infrastructure that you want to support you must get them to implement it for you.  They are thinking about a plugin system but are focused on the high security of the system and have put plugins off for later.


Simple Key Values

For the Vault POC I set up Vault in dev mode, which stores all data in memory, and first tried out setting and getting keys to a path.  Simple and easy.  

AWS Secret Backend

Then I tried out the AWS secret backend.  Backends are mounted like a file system.

$ vault mount aws
Successfully mounted 'aws' at 'aws'!

By default backends are mounted to a path that matches their name.  You can mount them to a different path though and this is important if you want to have different configurations for the same backend.  For instance, supporting different AWS accounts.

Once the backend is mounted you load an AWS key into it that has privileges to make the type of key you want it to produce later.  

$ vault write aws/config/root \
    access_key=AKIAJWVN5Z4FOFT7NLNA \
    secret_key=R4nm063hgMVo4BTT5xOs5nHLeLXA6lar7ZJ3Nt0i \

Now that Vault has permission to make us new keys we can add a role to the system.  A role is a path that has an IAM policy bound to it.  When queried the role will provide an access key with a policy bound to the role.

$ vault read aws/creds/deploy
Key             Value
lease_id        aws/creds/deploy/7cb8df71-782f-3de1-79dd-251778e49f58
lease_duration  3600
secret_key      BK9++oBABaBvRKcT5KEF69xQGcH7ZpPRF3oqVEv7
security_token  <nil>

When the lease is up on the role the keys will be deleted.

Postgresql Secret Backend

Vault can do a similar thing with databases.  Once the Postgresql backend is mounted you can configure it.  Here you will want to mount the backend to a specific path so that you can support multiple databases within a single server.  Here we mount to pg_blnd for the blnd database.

$ vault mount -path=pg_blnd postgresql

Now we can configure the backend with a connection string that has GRANT privileges on this database.

$ vault write postgresql/roles/blend sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"

Vault passes the username and password to this template to create the database user and grant it access privileges.

Of course most apps that use Postgresql don’t know how to query Vault for data so we need to get this information into a configuration file.  Consul-Template is a system that can query Vault (or Consul) and then render the data into a template.  Here is an example template that uses the postgresql backend and the secret backend.

DATABASES['default'] = {
   'ENGINE': 'django.db.backends.postgresql_psycopg2',
   'NAME': 'blnd',
{{ with secret "pg_blnd/creds/blend" }}
   'USER': '{{ .Data.username }}',
   'PASSWORD': '{{ .Data.password }}',
{{ end }}
{{ with secret "secret/blend/database/default" }}
   'HOST': '{{ .Data.hostname }}',
{{ end }}
   'PORT': '',
   'OPTIONS': {
       'sslmode': 'require',

Authenticating Against Vault

Ok so now we have all our secrets out of the code and into Vault how do we get them back out?  Vault has multiple authentication backends that grant access and an authorization system to control access.

For human access GitHub and username + password can be used.  These backends allow developers or systems admins to add and update values into the vault system.

For automated script access there are several options.  The simplest way to authenticate against Vault is using a token.  You can limit the rights of this token but it’s still the gateway to many other secrets.

AppRole is a little more sophisticated it has a RoleID and a SecretID.  It can limit based on source IP and can auto generate a SecretID based on the roleID.  This uniquely identifies a particular vault client for logging purposes.

TLS certificates can be used that have been signed by a CA or self-signed.  For a web server or anything with an SSL cert this might be a good option.

The AWS auth backend is really cool.  It allows you to use AWS as a trusted party using the cryptographically signed dynamic metadata that each EC2 instance has as identity.  While are lots of details around this system at a high level it makes sure that clients accessing vault are coming from inside your AWS account.  

Vault vs Barbican

Vault and Barbican are both open source projects with vault being made by a commercial open source company and Barbican under the OpenStack foundation.  Barbican is trying to solve similar problems to Vault but is less mature and less feature rich.  The dynamic secret backends that Vault has for AWS, certificate provisioning and and databases is very powerful.   The Consul-template Vault  integration that let’s configuration files be filled in with authentication information allows legacy applications to be integrated in a very straightforward manner.