More and more organizations use dedicated software to safely handle the creation and management of secrets (for example SSL certificate keys, private variables and passwords). Three 'well known' solutions are Square's Keywhiz, Hashicorp's Vault and crypt in combination with etcd or consul.
As with all security solutions the roll-out can be quite cumbersome. The correct implementation (think key management, think audit trails, think key recovery) of any one of these solutions is difficult. And difficult means that most people won't use it, at least not right away (remember SELinux ?).
There are a number of tools available to encrypt secrets within (Ansible) repositories. One of them for instance is Ansible Vault (look here for a more in-depth review). Although the idea of selectively encrypting data is a good one, text-oriented version control systems like git or Subversion aren't meant to store binary blobs of encrypted data. Moreover you still run the risk of accidentally uploading or sharing unencrypted files. Mitigations like adding filenames of unencrypted secrets to a .gitignore file are error-prone.
How to facilitate developers and system operators to store secrets in a safe place, outside the repositories where Ansible playbooks and configuration files are kept ?
This article describes a possible workflow for Ansible users who want to safely store secrets outside of the Ansible repository for use within their playbooks. It assumes that current operators already have access to the correct secrets: Developers have access to the development environment secrets for example, and system operators have access to production environment secrets. How you store these secrets is outside the scope of this article - this is about structuring (Ansible) variables in such a way to facilitate the safe storage of secrets.
The underlying goals are:
- Make it as easy as possible for humans to use secrets and store them outside the repository (don't disrupt the current workflow)
- Have different security levels depending on the environment (Development, Test, Acceptance or Production)
File structure
This is an example of a file structure, where /ansible is a git repository, and /protected/storage is a protected storage file system. This can be anything. The protected development storage can be for instance a private git repository shared amongst developers (low security clearance), and the production storage could be an encrypted network mount that is only accessible by operators (high security clearance). They don't have to share the same filesystem (see goal 2).
|-ansible |---/roles |-----/vars |-------public_vars |-----/files |-------public_file |-protected |---/storage |-----/development |-------secret_file |-------secret_vars.yml |-----/production |-------secret_file |-------secret_vars.yml
Ansible server works great from Vagrant for example, where the various Vagrant guest directories are mapped to host directories.
Ansible setup
The design pattern for Ansible is to create an inventory file which groups the hosts into environments having different (security) restrictions. This can be one inventory file or separate ones:
# file: inventory [development] development.domain.com [test] test.domain.com [acceptance] acceptance.domain.com [production] production.domain.com
This means that Ansible will look for variables for the development environment in the current directory under group_vars/development and in /etc/ansible/group_vars/development. Regardless where you store these files (in the current directory or on the ansible server in /etc/ansible), now you can include other files based on the environment. See for example the file group_vars/development...
--- # file: group_vars/development protected_storage: /protected/storage/development
....and group_vars/production
--- # file: group_vars/production protected_storage: /protected/storage/production
The variable protected_storage is set according to the environment. This means that by utilizing this pattern you can store any secret outside of the current Ansible repository, and the secrets will be loaded based on the environment. This can be used not only for variables, but files as well. By using the name protected_storage developers of Ansible playbooks are 'forced' to store secret files inside of that directory. Below is a configuration example to deploy SSL keys:
- name: Deploy private keys
copy:
src={{ protected_storage }}/{{ item }}
dest=/etc/ssl/private/{{ item }}
owner=root
group=root
mode=600
with_items: secret_file
This deploys the correct SSL keys based on the environment from the protected storage to the server. You can also include files containing secret variables in playbooks:
# file: webservers.yml
- hosts: all
roles:
- role: webtier
tags: webtier
tasks:
- include: roles/webtier/tasks/php.yml
tags: php
- include: roles/webtier/tasks/wordpress.yml
tags: wordpress
vars_files:
- "{{ protected_storage }}/wordpress.yml"``
When you run this playbook for the development environment, the correct secrets will be included from /protected/storage/development
--- # file: protected-storage/development/wordpress.yml wp_sites: - name: www.domain.com db_name: database_name db_user: database_user db_password: secret_password plugins: - akismet
This structure is relatively easy to implement, doesn't interfere with the current workflow, and separates secrets from the public variables and files. Hopefully this will last until your company does a complete roll-out of Keywhiz, Vault or crypt...
Comments
comments powered by Disqus