HashiCorp Vault with Docker Compose: Three-Node Cluster Setup (Part 3)

In Part 1, you set up a simple Vault instance in development mode, and in Part 2, you added persistent storage, manual unsealing, and MySQL dynamic secrets. While functional, a single node lacks high availability (HA). Now, in Part 3 of our series, you’ll create a three-node Vault cluster using Docker Compose to achieve high availability (HA). A cluster ensures Vault remains accessible if a node fails, making it more production-ready. This guide walks you through setting up three Vault nodes with individual configuration files and manual joining, keeping it beginner-to-intermediate friendly.

Cover Image: A digital padlock on a circuit board, symbolizing secure secrets management. (Source: Unsplash)

---

Prerequisites

Before starting, ensure you have:

• Docker (version 20.10 or later). Install from Docker’s guide.

• Docker Compose (version 2.0 or later). See Docker Compose installation.

• Vault CLI for CLI access. Download from releases.hashicorp.com/vault or use a package manager (e.g., brew install vault on macOS).

• A text editor (e.g., VS Code).

• Familiarity with Parts 1 and 2 (development mode, persistent storage, unsealing).

• Commands are tested on Linux/macOS; Windows users may need to use set instead of export for environment variables.

---

Setting Up a Three-Node Vault Cluster

A Vault cluster consists of multiple nodes sharing the same storage backend, with one node acting as the leader and others as followers. If the leader fails, a follower takes over. Below is a diagram of the setup:

Step 1: Create the Docker Compose File

Create a file named docker-compose.yml in a new directory (e.g., vault-cluster):

services:
  vault1:
    image: hashicorp/vault:latest
    container_name: vault1
    ports:
      - "8200:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault1-data:/vault/file
      - ./vault-config/vault1:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault2:
    image: hashicorp/vault:latest
    container_name: vault2
    ports:
      - "8201:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault2-data:/vault/file
      - ./vault-config/vault2:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault3:
    image: hashicorp/vault:latest
    container_name: vault3
    ports:
      - "8202:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault3-data:/vault/file
      - ./vault-config/vault3:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

volumes:
  vault1-data:
  vault2-data:
  vault3-data:

networks:
  vault-net:
    driver: bridge

Key Notes:

• Separate volumes (vault1-data, vault2-data, vault3-data) ensure Raft storage is independent per node.

• Each node maps a unique config directory (vault-config/vault1, etc.) and exposes a different host port (8200, 8201, 8202).

• vault-net: A custom network ensures nodes can communicate.

• Inside containers, Vault listens on port 8200, mapped to unique host ports.

Step 2: Create Vault Configuration Files

To simplify cluster setup, we could manually join nodes using vault operator raft join, but this approach is tedious and error-prone, requiring careful coordination for each node. Instead, we’ll use retry_join to automate node discovery, making the process faster, more reliable, and less boring. This configuration allows each node to automatically find and join the cluster by contacting other nodes, streamlining setup and improving resilience.

Moving to Raft storage

In Part 2, we used file storage with a shared volume across containers for simplicity, enabling a single-node Vault to persist data easily. In Part 3, we switch to Raft storage, which is integrated into Vault and designed for high availability clustering. Raft ensures each node maintains its own consistent state and supports leader election and replication, making it ideal for a resilient, multi-node cluster. Additionally, unlike Part 2’s shared volume, Part 3 uses separate storage volumes for each node (vault1-data, vault2-data, vault3-data), eliminating shared storage dependencies and aligning with Raft’s requirement for independent node data.

Create a vault-config directory with subdirectories: vault1, vault2, vault3. Add vault.hcl files following the specified pattern:

vault-config/vault1/vault.hcl:

ui = true
api_addr = "http://vault1:8200"
cluster_addr = "http://vault1:8201"

storage "raft" {
  path = "/vault/file"
  node_id = "vault1"

  retry_join {
    leader_api_addr = "http://vault2:8200"
    leader_api_addr = "http://vault3:8200"
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

The retry_join configuration automates cluster formation by enabling each Vault node to attempt joining other nodes’ API addresses (e.g., vault1 tries vault2 and vault3) until successful. This resilient mechanism retries on failure and excludes the current node to avoid self-joining, ensuring robust clustering. By specifying multiple leader_api_addr entries, retry_join simplifies setup and enhances reliability compared to manual joining.

vault-config/vault2/vault.hcl:

ui = true
api_addr = "http://vault2:8200"
cluster_addr = "http://vault2:8201"

storage "raft" {
  path = "/vault/file"
  node_id = "vault2"

  retry_join {
    leader_api_addr = "http://vault1:8200"
    leader_api_addr = "http://vault3:8200"
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

vault-config/vault3/vault.hcl:

ui = true
api_addr = "http://vault3:8200"
cluster_addr = "http://vault3:8201"

storage "raft" {
  path = "/vault/file"
  node_id = "vault3"

  retry_join {
    leader_api_addr = "http://vault1:8200"
    leader_api_addr = "http://vault2:8200"
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

Key Notes:

• storage "raft": Uses Raft for HA, with a unique node_id per node.

• retry_join: Excludes the current node (e.g., vault1 joins vault2 and vault3), improving resilience.

• tls_disable = 1: Disabled for simplicity; production requires TLS.

• Production Warning: Use multiple unseal keys and TLS in production.

• api_addr: Specifies the node’s API address for client communication.

• cluster_addr: Defines the address for cluster communication (port 8201 to avoid conflicts with 8200).

Step 3: Start and Initialize the Cluster

1. Navigate to vault-cluster and start the containers:

    docker-compose up -d
   

2. Verify the containers are running:

    docker ps
   

   You should see vault1, vault2, and vault3.

3. Initialize the leader node (vault1):

    export VAULT_ADDR=http://localhost:8200
    vault operator init -key-shares=1 -key-threshold=1
   
    Unseal Key 1: <unseal-key>
    Initial Root Token: <root-token>
   
    Vault initialized with 1 key shares and a key threshold of 1.
    Please securely distribute the key shares printed above.
   

4. This outputs one unseal key and a root token. Save the unseal key and root token securely (e.g., in a password manager). As in Part 2, we’re using a single key for simplicity, but production clusters typically use multiple keys (e.g., three out of five) for security. Do not use a single key in production.

5. Unseal vault1:

    vault operator unseal <unseal-key>
   

6. Log in with the root token:

    export VAULT_TOKEN=<root-token>
    vault login
   

7. Join vault2 to the cluster:

    export VAULT_ADDR=http://localhost:8201
    vault operator init -key-shares=1 -key-threshold=1
    vault operator unseal <vault2-unseal-key>
   

8. Join vault3 to the cluster:

    export VAULT_ADDR=http://localhost:8202
    vault operator init -key-shares=1 -key-threshold=1
    vault operator unseal <vault3-unseal-key>
   

Step 4: Test the Cluster

1. Store a secret on vault1:

    export VAULT_ADDR=http://localhost:8200
    vault kv put secret/my-app db_password=supersecret
   

2. Retrieve it from vault2:

    export VAULT_ADDR=http://localhost:8201
    vault kv get secret/my-app
   

3. Check cluster status:

    vault operator raft list-peers
   
    Node      Address          State       Voter
    ----      -------          -----       -----
    vault1    vault1:8201      leader      true
    vault2    vault2:8201      follower    true
    vault3    vault3:8201      follower    true
   

   This shows all three nodes, with vault1 as the leader.

Step 5: Access Vault

• Web UI: Open http://localhost:8200, http://localhost:8201, or http://localhost:8202. Log in with the root token. Navigate to “Secrets” to view or create secrets.

• CLI: Use any node’s address to interact with the cluster (e.g., VAULT_ADDR=http://localhost:8200).

• Secrets persist via separate Raft storage volumes.

Step 6: Stop the Cluster

To stop the containers:

docker-compose down

On restart, unseal each node using the same unseal key.

---

Best Practices

• Use Multiple Keys in Production: Configure multiple unseal keys (e.g., three out of five) for security. Avoid a single key.

• Monitor Retry Join: Ensure retry_join targets are stable. Use cloud metadata or multiple addresses in production.

• Backup Raft Storage: Regularly back up vault1-data, vault2-data, and vault3-data volumes.

• Secure Unseal Key and Root Token: Store the unseal key and root token in a secure location (e.g., password manager or HSM). In production, use multiple keys (e.g., three out of five) per node.

• Ensure Network Reliability: Nodes must communicate via cluster_addr. Use a stable network (e.g., vault-net) and monitor connectivity.

• Monitor Leader Election: Check vault operator raft list-peers to ensure a leader is active. If the leader fails, a follower takes over automatically.

• Enable TLS: In production, set tls_disable = 0 in vault.hcl and use certificates for secure communication.

• Backup Storage: Regularly back up the vault-data volume to prevent data loss.

---

Troubleshooting

• Node Fails to Join: Verify vault1 is unsealed and accessible (curl http://vault1:8200/v1/sys/health). Check Docker network (vault-net) with docker network inspect vault-net.

• Raft Issues: Ensure separate volumes are mounted (docker inspect vault1). Check logs (docker logs vault1).

• Unseal Fails: Ensure the unseal key is correct. Re-run vault operator init -key-shares=1 -key-threshold=1 on vault1 if lost (this resets the cluster).

• Cluster Not Responding: Check logs (docker logs vault1) for errors. Ensure ports 8200-8202 are free (lsof -i :8200-8202).

---

What’s Next?

You’ve built a high-availability Vault cluster with Raft and retry_join! In Part 4, we’ll eliminate manual unsealing by implementing transit-based auto-unseal with a dedicated Vault instance. Try:

• Stop vault1 and verify a new leader is elected (vault operator raft list-peers).

• Adapting MySQL secrets from Part 2.

• Store a secret on one node and retrieve it from another.

For dynamic secrets (e.g., MySQL), revisit Part 2 and adapt the configuration for this cluster. Share your progress in the comments or join the HashiCorp Community Forum!

---

Resources

• HashiCorp Vault Documentation: vaultproject.io

• Vault HA Documentation: vaultproject.io/docs/concepts/ha

• Raft Storage Docs: vaultproject.io/docs/configuration/storage/raft

• Docker Compose Documentation: docs.docker.com/compose

Note: For questions or setup help, comment below or check the Vault documentation.