# HashiCorp Vault with Docker Compose: Three-Node Cluster Setup (Part 3)

In [Part 1](hashicorp-vault-docker-compose-part1), you set up a simple Vault instance in development mode, and in [Part 2](hashicorp-vault-docker-compose-part2), you added persistent storage, manual unsealing, and MySQL dynamic secrets. While functional, a single node lacks high availability (HA). Now, in Part 3 of our series, you’ll create a **three-node Vault cluster** using Docker Compose to achieve **high availability** (HA). A cluster ensures Vault remains accessible if a node fails, making it more production-ready. This guide walks you through setting up three Vault nodes with individual configuration files and manual joining, keeping it beginner-to-intermediate friendly.

*Cover Image*: A digital padlock on a circuit board, symbolizing secure secrets management. (Source: [Unsplash](https://images.unsplash.com/photo-1550751827-4bd374c3f58b?ixlib=rb-4.0.3&auto=format&fit=crop&w=1350&q=80))

---

## Prerequisites

Before starting, ensure you have:

* **Docker** (version 20.10 or later). Install from [Docker’s guide](https://docs.docker.com/get-docker/).
    
* **Docker Compose** (version 2.0 or later). See [Docker Compose installation](https://docs.docker.com/compose/install/).
    
* **Vault CLI** for CLI access. Download from [releases.hashicorp.com/vault](https://releases.hashicorp.com/vault/) or use a package manager (e.g., `brew install vault` on macOS).
    
* A text editor (e.g., VS Code).
    
* Familiarity with Parts 1 and 2 (development mode, persistent storage, unsealing).
    
* Commands are tested on Linux/macOS; Windows users may need to use `set` instead of `export` for environment variables.
    

---

## Setting Up a Three-Node Vault Cluster

A Vault cluster consists of multiple nodes sharing the same storage backend, with one node acting as the **leader** and others as **followers**. If the leader fails, a follower takes over. Below is a diagram of the setup:

```mermaid
graph TD
    A[Client: Browser/CLI] --> B[Docker Compose]
    B --> C[Vault Node 1: Port 8200]
    B --> D[Vault Node 2: Port 8201]
    B --> E[Vault Node 3: Port 8202]
    C --> F[Shared Storage]
    D --> F
    E --> F
```

### Step 1: Create the Docker Compose File

Create a file named `docker-compose.yml` in a new directory (e.g., `vault-cluster`):

```yaml
services:
  vault1:
    image: hashicorp/vault:latest
    container_name: vault1
    ports:
      - "8200:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault1-data:/vault/file
      - ./vault-config/vault1:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault2:
    image: hashicorp/vault:latest
    container_name: vault2
    ports:
      - "8201:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault2-data:/vault/file
      - ./vault-config/vault2:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault3:
    image: hashicorp/vault:latest
    container_name: vault3
    ports:
      - "8202:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault3-data:/vault/file
      - ./vault-config/vault3:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

volumes:
  vault1-data:
  vault2-data:
  vault3-data:

networks:
  vault-net:
    driver: bridge
```

**Key Notes**:

* Separate volumes (vault1-data, vault2-data, vault3-data) ensure Raft storage is independent per node.
    
* Each node maps a unique config directory (`vault-config/vault1`, etc.) and exposes a different host port (`8200`, `8201`, `8202`).
    
* `vault-net`: A custom network ensures nodes can communicate.
    
* Inside containers, Vault listens on port `8200`, mapped to unique host ports.
    

### Step 2: Create Vault Configuration Files

To simplify cluster setup, we could manually join nodes using `vault operator raft join`, but this approach is tedious and error-prone, requiring careful coordination for each node. Instead, we’ll use retry\_join to automate node discovery, making the process faster, more reliable, and less boring. This configuration allows each node to automatically find and join the cluster by contacting other nodes, streamlining setup and improving resilience.

### Moving to Raft storage

In Part 2, we used file storage with a shared volume across containers for simplicity, enabling a single-node Vault to persist data easily. In Part 3, we switch to Raft storage, which is integrated into Vault and designed for high availability clustering. Raft ensures each node maintains its own consistent state and supports leader election and replication, making it ideal for a resilient, multi-node cluster. Additionally, unlike Part 2’s shared volume, Part 3 uses separate storage volumes for each node (vault1-data, vault2-data, vault3-data), eliminating shared storage dependencies and aligning with Raft’s requirement for independent node data.

Create a vault-config directory with subdirectories: `vault1`, `vault2`, `vault3`. Add `vault.hcl` files following the specified pattern:

**vault-config/vault1/vault.hcl**:

```plaintext
ui = true
api_addr = "http://vault1:8200"
cluster_addr = "http://vault1:8201"

storage "raft" {
  path = "/vault/file"
  node_id = "vault1"

  retry_join {
    leader_api_addr = "http://vault2:8200"
    leader_api_addr = "http://vault3:8200"
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}
```

*The retry\_join configuration automates cluster formation by enabling each Vault node to attempt joining other nodes’ API addresses (e.g.,* `vault1` *tries* `vault2` *and* `vault3`*) until successful. This resilient mechanism retries on failure and excludes the current node to avoid self-joining, ensuring robust clustering. By specifying multiple* `leader_api_addr` *entries,* `retry_join` *simplifies setup and enhances reliability compared to manual joining.*

**vault-config/vault2/vault.hcl**:

```plaintext
ui = true
api_addr = "http://vault2:8200"
cluster_addr = "http://vault2:8201"

storage "raft" {
  path = "/vault/file"
  node_id = "vault2"

  retry_join {
    leader_api_addr = "http://vault1:8200"
    leader_api_addr = "http://vault3:8200"
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}
```

**vault-config/vault3/vault.hcl**:

```plaintext
ui = true
api_addr = "http://vault3:8200"
cluster_addr = "http://vault3:8201"

storage "raft" {
  path = "/vault/file"
  node_id = "vault3"

  retry_join {
    leader_api_addr = "http://vault1:8200"
    leader_api_addr = "http://vault2:8200"
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}
```

**Key Notes**:

* `storage "raft"`: Uses Raft for HA, with a unique node\_id per node.
    
* `retry_join`: Excludes the current node (e.g., vault1 joins vault2 and vault3), improving resilience.
    
* `tls_disable = 1`: Disabled for simplicity; production requires TLS.
    
* **Production Warning**: Use multiple unseal keys and TLS in production.
    
* `api_addr`: Specifies the node’s API address for client communication.
    
* `cluster_addr`: Defines the address for cluster communication (port `8201` to avoid conflicts with `8200`).
    

### Step 3: Start and Initialize the Cluster

1. Navigate to `vault-cluster` and start the containers:
    
    ```bash
    docker-compose up -d
    ```
    
2. Verify the containers are running:
    
    ```bash
    docker ps
    ```
    
    You should see `vault1`, `vault2`, and `vault3`.
    
3. Initialize the leader node (`vault1`):
    
    ```bash
    export VAULT_ADDR=http://localhost:8200
    vault operator init -key-shares=1 -key-threshold=1
    
    Unseal Key 1: <unseal-key>
    Initial Root Token: <root-token>
    
    Vault initialized with 1 key shares and a key threshold of 1.
    Please securely distribute the key shares printed above.
    ```
    
4. This outputs **one unseal key** and a **root token**. Save the **unseal key** and **root token** securely (e.g., in a password manager). As in Part 2, we’re using a single key for simplicity, but production clusters typically use multiple keys (e.g., three out of five) for security. **Do not use a single key in production**.
    
5. Unseal `vault1`:
    
    ```bash
    vault operator unseal <unseal-key>
    ```
    
6. Log in with the root token:
    
    ```bash
    export VAULT_TOKEN=<root-token>
    vault login
    ```
    
7. Join `vault2` to the cluster:
    
    ```bash
    export VAULT_ADDR=http://localhost:8201
    vault operator init -key-shares=1 -key-threshold=1
    vault operator unseal <vault2-unseal-key>
    ```
    
8. Join `vault3` to the cluster:
    
    ```bash
    export VAULT_ADDR=http://localhost:8202
    vault operator init -key-shares=1 -key-threshold=1
    vault operator unseal <vault3-unseal-key>
    ```
    

### Step 4: Test the Cluster

1. Store a secret on `vault1`:
    
    ```bash
    export VAULT_ADDR=http://localhost:8200
    vault kv put secret/my-app db_password=supersecret
    ```
    
2. Retrieve it from `vault2`:
    
    ```bash
    export VAULT_ADDR=http://localhost:8201
    vault kv get secret/my-app
    ```
    
3. Check cluster status:
    
    ```bash
    vault operator raft list-peers
    
    Node      Address          State       Voter
    ----      -------          -----       -----
    vault1    vault1:8201      leader      true
    vault2    vault2:8201      follower    true
    vault3    vault3:8201      follower    true
    ```
    
    This shows all three nodes, with `vault1` as the leader.
    

### Step 5: Access Vault

* **Web UI**: Open `http://localhost:8200`, `http://localhost:8201`, or `http://localhost:8202`. Log in with the root token. Navigate to “Secrets” to view or create secrets.
    
* **CLI**: Use any node’s address to interact with the cluster (e.g., `VAULT_ADDR=http://localhost:8200`).
    
* Secrets persist via separate Raft storage volumes.
    

### Step 6: Stop the Cluster

To stop the containers:

```bash
docker-compose down
```

On restart, unseal each node using the same unseal key.

---

## Best Practices

* **Use Multiple Keys in Production**: Configure multiple unseal keys (e.g., three out of five) for security. Avoid a single key.
    
* **Monitor Retry Join**: Ensure retry\_join targets are stable. Use cloud metadata or multiple addresses in production.
    
* **Backup Raft Storage**: Regularly back up vault1-data, vault2-data, and vault3-data volumes.
    
* **Secure Unseal Key and Root Token**: Store the unseal key and root token in a secure location (e.g., password manager or HSM). In production, use multiple keys (e.g., three out of five) per node.
    
* **Ensure Network Reliability**: Nodes must communicate via `cluster_addr`. Use a stable network (e.g., `vault-net`) and monitor connectivity.
    
* **Monitor Leader Election**: Check `vault operator raft list-peers` to ensure a leader is active. If the leader fails, a follower takes over automatically.
    
* **Enable TLS**: In production, set `tls_disable = 0` in `vault.hcl` and use certificates for secure communication.
    
* **Backup Storage**: Regularly back up the `vault-data` volume to prevent data loss.
    

---

## Troubleshooting

* **Node Fails to Join**: Verify `vault1` is unsealed and accessible (`curl http://vault1:8200/v1/sys/health`). Check Docker network (`vault-net`) with `docker network inspect vault-net`.
    
* **Raft Issues**: Ensure separate volumes are mounted (docker inspect vault1). Check logs (docker logs vault1).
    
* **Unseal Fails**: Ensure the unseal key is correct. Re-run `vault operator init -key-shares=1 -key-threshold=1` on `vault1` if lost (this resets the cluster).
    
* **Cluster Not Responding**: Check logs (`docker logs vault1`) for errors. Ensure ports `8200-8202` are free (`lsof -i :8200-8202`).
    

---

## What’s Next?

You’ve built a high-availability Vault cluster with Raft and retry\_join! In **Part 4**, we’ll eliminate manual unsealing by implementing transit-based auto-unseal with a dedicated Vault instance. Try:

* Stop `vault1` and verify a new leader is elected (`vault operator raft list-peers`).
    
* Adapting MySQL secrets from Part 2.
    
* Store a secret on one node and retrieve it from another.
    

For dynamic secrets (e.g., MySQL), revisit [Part 2](hashicorp-vault-docker-compose-part2) and adapt the configuration for this cluster. Share your progress in the comments or join the [HashiCorp Community Forum](https://discuss.hashicorp.com)!

---

## Resources

* **HashiCorp Vault Documentation**: [vaultproject.io](https://www.vaultproject.io)
    
* **Vault HA Documentation**: [vaultproject.io/docs/concepts/ha](https://www.vaultproject.io/docs/concepts/ha)
    
* **Raft Storage Docs**: vaultproject.io/docs/configuration/storage/raft
    
* **Docker Compose Documentation**: [docs.docker.com/compose](https://docs.docker.com/compose)
    

*Note*: For questions or setup help, comment below or check the Vault documentation.