Skip to main content
Contact our team to know more about our services
select webform
By submitting, you acknowledge that you've read and agree to our privacy policies, and Opcito may use the information provided for business purposes.
Become a part of our team
select webform
One file only.
1.5 GB limit.
Allowed types: gif, jpg, jpeg, png, bmp, eps, tif, pict, psd, txt, rtf, html, odf, pdf, doc, docx, ppt, pptx, xls, xlsx, xml, avi, mov, mp3, mp4, ogg, wav, bz2, dmg, gz, jar, rar, sit, svg, tar, zip.
By submitting, you acknowledge that you've read and agree to our privacy policies, and Opcito may use the information provided for business purposes.
How to set up High Availability Vault with Consul backend?
12 Sep 2019

How to set up High Availability Vault with Consul backend?

Usable security, trusted delivery, and infrastructure independence are the three pillars of Docker container security that result in safer apps. Another important aspect of this security is that apps must communicate with other apps and systems in safe and secure ways. Docker secrets is a container-based solution that ensures secret distribution within container platforms. A secret literally means anything that is not meant to be seen or known by unauthorized personnel and demands tokens, API keys, passwords, and certificates, among others, to unlock. Docker secrets is an integral part of container security that strengthens its “trusted delivery” component. In this blog, I am going to talk about how you can strengthen the security of your Docker containers using Docker secrets and a HashiCorp duo of Vault and Consul. I will focus mostly on achieving Vault High Availability with Consul Cluster. But before moving toward that, let us first have a look at Vault and Consul as individual components of the system that we are going to build.

Vault

Vault provides high-level policy management, secret leasing, audit logging, and automatic revocation to protect sensitive information using UI, CLI, or HTTP API. All the secrets stored in it have an associated lease to enable key usage auditing, perform key rolling, and ensure automatic revocation. It provides multiple revocation mechanisms to give operators a clear “break glass” procedure in case of a potential compromise. It consists of a unified interface to store secrets while providing tight access control and recording a detailed audit log. It can be deployed to practically any environment and does not require any special hardware such as physical HSMs (Hardware Security Modules). Vault is an open-source tool, and you can get it here.

Consul Storage Backend

The Consul storage backend is used to retain Vault's data in Consul's key-value store. In addition to providing durable storage, the inclusion of this backend will also register Vault as a service in Consul with a default health check. The Consul storage backend supports high availability and is backed by HashiCorp.

Consul is a tool that can aid in service discovery or, basically, the registration of apps to your system. It also can be used for a key/value configuration storage system and can store app authentication, configuration flags, etc. When Vault is coupled with a Consul back-end, it makes your key/value configuration more secure. The tool itself requires authentication to be able to access all the secret goodies that are stored inside.

The token management system of Vault allows you to create temporary access tokens to Vault as per time and usage. Hence, applications can have only temporary access to those secrets. For example, a lot of applications need a username and password for some DB authorization parameters upon startup. These credentials are then stored in the memory. By creating limited authentication tokens, you can prevent the application from requesting credentials while it is running.

Consul is a distributed, highly available, and data-center-aware service discovery & configuration system. It has a flexible and powerful interface that allows clients to view the current state of the infrastructure, including service & node discovery mechanisms, a tagging system, health checks, consensus-based election routines, system-wide key/value storage, and more. With Consul, you can easily build a new level of awareness into your applications and services. Now, let us see how to achieve high availability using Consul and Vault.

Consul Storage Backend architecture

High Availability

A Vault cluster is a highly-available unit of deployment within a data center. It is recommended to have three Vault servers with a Consul storage backend. With this configuration, during a Vault server outage, failover is addressed immediately without human intervention. To learn more about setting up your Vault servers in HA mode, you can refer the Vault HA with Consul guide. High availability is achieved using Performance Standby Nodes and data locality across datacenters that are exclusively available in Vault Enterprise.

Now, let's build new images and spin up containers:

###Docker Network

$ docker network create -d overlay --attachable 

###Deploy Stack

$ docker stack deploy -c docker-stack.yml 

###Remove Stack

$ docker stack rm 

###List Docker Services

$ docker service ls

###Access Vault UI

open http://:8200 URL in the browser

Network Connectivity Details

Consul Network Connectivity Details Architecture

The following is a sample configuration block for Vault HA with Consul cluster:

docker-stack.yml

version: '3.6'
services:
server-bootstrap:
image: consul:1.2.1
ports:
- 8500:8500
command: "agent -server -bootstrap -ui -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"
server:
image: consul:1.2.1
command: "agent -server -retry-join server-bootstrap -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"
deploy:
replicas: 2
depends_on:
- server-bootstrap
client:
image: consul:1.2.1
command: "agent -retry-join server-bootstrap -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"
deploy:
replicas: 2
depends_on:
- server-bootstrap
vault:
depends_on:
- consul
image: vault
container_name: "vault"
hostname: "vault"
restart: always
cap_add:
- IPC_LOCK
ports:
- 8200:8200
environment:
VAULT_LOCAL_CONFIG: '{"backend": {"consul": {"address": "http://server-bootstrap:8500", "path": "vault"}}, "listener": {"tcp": {"address": "0.0.0.0:8200", "tls_disable": "1"}}, "default_lease_ttl": "24h", "ui":true, "max_lease_ttl": "720h", "disable_mlock":true}'
VAULT_ADDR: "http://0.0.0.0:8200"
command:
- server
ulimits:
nproc: 65535
cap_add:
- ALL
privileged: true
deploy:
restart_policy:
condition: on-failure
delay: 5s
max_attempts: 3
window: 120s
networks:
default:
external:
name: network_name

Setup Consul Client Agents on Vault Nodes

Let’s set up the initial set of master keys that you may need in case of emergency -

Setup Consul Client Agents on Vault Nodes

The above screenshot shows the first screen that appears after the initial deployment. Key Shares is basically the number of keys that you want and Threshold is the number of keys needed to be entered to unseal your vault. At least 5 Key Sharez and 2 Key Thresholds are recommended for good security. Also, make sure to download this content securely somewhere. It also provides the root authentication token.

Root authentication token and master key portion in consul

Vault has “seal” and “unseal” features. The auto unseal feature is available only in the Enterprise version. In case of a compromise, this can be your “kill switch”. Seal your vault, and everything gets restricted.

###Access Consul UI

Open http://:8500 URL in the browser
Access Consul UI

 

Since we used Consul as the Vault backend, all Vault data is stored in Consul KV, and everything is encrypted.

Basic Key-Value Example

Writing Static credential to the vault using CLI:

vault write secret/staticcreds/username my-value=N7OVkOH6yQGfkGciySsek9knZfSzKj

Read Static credential from vault using CLI by key name:

vault read secret/staticcreds/username
Vault Read Secret

Once you’ve completed this, Vault will be up and running in HA!

[root@host1 ~]# vault status

Sealed: false

Key Shares: 5

Key Threshold: 3

Unseal Progress: 0

Unseal Nonce:

Version: 0.7.3

Cluster Name: vault-cluster-975049e2

Cluster ID: 6efde828-3c74-a994-ed47-32e5d8451f3d

High-Availability Enabled: true
Mode: active
Leader: http://host1:8200
[root@host2 ~]# vault status
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce:
Version: 0.7.3
Cluster Name: vault-cluster-975049e2
Cluster ID: 6efde828-3c74-a994-ed47-32e5d8451f3d
High-Availability Enabled: true
Mode: standby
Leader: http://host2:8200

Well, that was a quick look at how you can achieve Vault High Availability with Consul Cluster. To gain a further in-depth understanding of the topic, I am giving some additional references in the links mentioned below:

I hope this piece added value to your understanding of the topic. If you have any queries or suggestions, feel free to write them down in the comments section.

Subscribe to our feed

select webform