How to set up High Availability Vault with Consul backend?

September 12, 2019 0 20
Share
How to set up High Availability Vault with Consul backend?

Usable security, trusted delivery, and infrastructure independence are the three pillars of Docker container security that result in safer apps. Another important aspect of this security is that apps must communicate with other apps and systems in safe and secure ways. Docker secrets is a container-based solution that ensures secret distribution within container platforms. A secret literally means anything that is not meant to be seen or known by unauthorized personnel and demands tokens, API keys, passwords, and certificates, among others to unlock. Docker secrets is an integral part of container security that strengthens its “trusted delivery” component. In this blog, I am going to talk about how you can strengthen the security of your Docker containers using Docker secrets and a HashiCorp duo of Vault and Consul. I will focus mostly on achieving Vault High Availability with Consul Cluster. But before moving toward that, let us first have a look at Vault and Consul as individual components of the system that we are going to build.

Vault

Vault provides high-level policy management, secret leasing, audit logging, and automatic revocation to protect sensitive information using UI, CLI, or HTTP API. All the secrets stored in it have an associated lease to enable key usage auditing, perform key rolling, and ensure automatic revocation. It provides multiple revocation mechanisms to give operators a clear “break glass” procedure in case of a potential compromise. It consists of a unified interface to store secrets while providing tight access control and recording a detailed audit log. It can be deployed to practically any environment and does not require any special hardware such as physical HSMs (Hardware Security Modules). Vault is open-source tool and you can get it here.

Consul Storage Backend

The Consul storage backend is used to retain Vault’s data in Consul’s key-value store. In addition to providing durable storage, the inclusion of this backend will also register Vault as a service in Consul with a default health check. The Consul storage backend supports high availability and is backed by HashiCorp.

Consul is a tool that can aide in service discovery or basically the registration of apps to your system. It also can be used for a key/value configuration storage system and can store app authentication, configuration flags, etc. When Vault is coupled with a Consul back-end, it makes your key/value configuration more secure. The tool itself requires authentication to be able to access all the secret goodies that are stored inside.

The token management system of Vault allows you to create temporary access tokens to Vault as per time and usage. Hence, applications can have only temporary access to those secrets. For example, a lot of applications need a username and password for some DB authorization parameters upon startup. These credentials are then stored in the memory. By creating limited authentication tokens, you can prevent the application from requesting credentials while it is running.

Consul is a distributed, highly available, and data-center-aware service discovery & configuration system. It has a flexible and powerful interface that allows clients to view the current state of the infrastructure including service & node discovery mechanisms, a tagging system, health checks, consensus-based election routines, system-wide key/value storage, and more. With Consul, you can easily build a new level of awareness into your applications and services. Now, let us see how to achieve high availability using Consul and Vault.

Consul Storage Backend

High Availability

A Vault cluster is the highly-available unit of deployment within a data center. It is recommended to have three Vault servers with a Consul storage backend. With this configuration, during a Vault server outage, failover is addressed immediately without human intervention. To learn more about setting up your Vault servers in HA mode, you can refer the Vault HA with Consul guide. High availability is achieved using Performance Standby Nodes and data locality across datacenters that are exclusively available in Vault Enterprise.

Now, let’s build new images and spin up containers:

###Docker Network

$ docker network create -d overlay --attachable 

###Deploy Stack

$ docker stack deploy -c docker-stack.yml 

###Remove Stack

$ docker stack rm 

###List Docker Services

$ docker service ls

###Access Vault UI

open http://:8200 URL in the browser

Network Connectivity Details

Network Connectivity Details

The following is a sample configuration block for Vault HA with Consul cluster:

docker-stack.yml


version: '3.6'

services:

  server-bootstrap:
    image: consul:1.2.1
    ports:
      - 8500:8500
    command: "agent -server -bootstrap -ui -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"

  server:
    image: consul:1.2.1
    command: "agent -server -retry-join server-bootstrap -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"
    deploy:
      replicas: 2
    depends_on:
      - server-bootstrap

  client:
    image: consul:1.2.1
    command: "agent -retry-join server-bootstrap -client 0.0.0.0 -bind '{{ GetInterfaceIP \"eth0\" }}'"
    deploy:
      replicas: 2
    depends_on:
      - server-bootstrap

  vault:
    depends_on:
      - consul
    image: vault
    container_name: "vault"
    hostname: "vault"
    restart: always
    cap_add:
      - IPC_LOCK
    ports:
      - 8200:8200
    environment:
      VAULT_LOCAL_CONFIG: '{"backend": {"consul": {"address": "http://server-bootstrap:8500", "path": "vault"}}, "listener": {"tcp": {"address": "0.0.0.0:8200", "tls_disable": "1"}}, "default_lease_ttl": "24h", "ui":true, "max_lease_ttl": "720h", "disable_mlock":true}'
      VAULT_ADDR: "http://0.0.0.0:8200"
    command:
      - server
    ulimits:
      nproc: 65535
    cap_add:
      - ALL
    privileged: true
    deploy:
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s

networks:
  default:
    external:
      name: network_name

Setup Consul Client Agents on Vault Nodes

Let’s set up the initial set of master keys that you may need in case of emergency –

Setup Consul Client Agents on Vault Nodes

The above screenshot shows the first screen that appears after the initial deployment. Key Shares is basically the number of keys that you want and Threshold is the number of keys needed to be entered to unseal your vault. At least 5 Key Sharez and 2 Key Thresholds are recommended for good security. Also, make sure to download this content securely somewhere. It also provides the root authentication token.

root authentication token

master key portion

Vault has “seal” and “unseal” features. The auto unseal feature is available only in the Enterprise version. In case of a compromise, this can be your “kill switch”. Seal your vault and everything gets restricted.

###Access Consul UI

Open http://:8500 URL in the browser

Access Consul UI
Access Consul UI

Since we used Consul as Vault backend, all Vault data is stored in Consul KV and everything is encrypted.

Basic Key-Value Example

Writing Static credential to the vault using CLI:

vault write secret/staticcreds/username my-value=N7OVkOH6yQGfkGciySsek9knZfSzKj

Read Static credential from vault using CLI by key name:

vault read secret/staticcreds/username

Vault

Once you’ve completed this, Vault is up and running in HA!

[root@host1 ~]# vault status
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce:
Version: 0.7.3
Cluster Name: vault-cluster-975049e2
Cluster ID: 6efde828-3c74-a994-ed47-32e5d8451f3d

High-Availability Enabled: true
	Mode: active
	Leader: http://host1:8200

[root@host2 ~]# vault status
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce:
Version: 0.7.3
Cluster Name: vault-cluster-975049e2
Cluster ID: 6efde828-3c74-a994-ed47-32e5d8451f3d

High-Availability Enabled: true
	Mode: standby
	Leader: http://host2:8200

Well, that was a quick look at how you can achieve Vault High Availability with Consul Cluster. To gain further in-depth understanding of the topic, I am giving some additional references in the links mentioned below:

I hope this piece added value to your understanding of the topic. If you have any queries or suggestions, feel free to write them down in the comments section.

Tags
Leave Comment

Your email address will not be published.