HashiCorp Vault with Docker Compose: Advanced Auto-Unseal with Transit (Part 4)

Subtitle: Transition from manual unsealing to auto-unseal using a dedicated transit Vault

In Part 3, you built a three-node Vault cluster using Raft storage and retry_join for automated clustering, with separate storage volumes for high availability (HA). However, each node required manual unsealing, which is cumbersome. In this final, comprehensive part of our series, we’ll enhance the cluster by implementing auto-unseal using the transit secrets engine hosted on a dedicated Vault container, eliminating manual key entry. For simplicity, we use transit, but production environments should use a Key Management Service (KMS) (e.g., AWS KMS, Azure Key Vault) or Hardware Security Module (HSM) for enhanced security. This intermediate-to-advanced guide builds on Parts 1-3, delivering a production-like setup.

---

Prerequisites

Before starting, ensure you have:

• Docker (version 20.10 or later). Install from Docker’s guide.

• Docker Compose (version 2.0 or later). See Docker Compose installation.

• Vault CLI. Download from releases.hashicorp.com/vault or use a package manager (e.g., brew install vault on macOS).

• jq for parsing JSON (optional, for token extraction). Install via brew install jq (macOS) or equivalent.

• A text editor (e.g., VS Code).

• Familiarity with Parts 1-3 (development mode, persistent storage, Raft clustering with retry_join).

• Commands are tested on Linux/macOS; Windows users may need to use set instead of export.

---

Reviewing the Cluster with Manual Unseal

In Part 3, you set up a three-node cluster (vault1, vault2, vault3) using Raft storage for HA, with retry_join for automated node discovery and separate volumes (vault1-data, vault2-data, vault3-data) to support Raft’s independent storage needs. As explained in Part 3, this transitioned from Part 2’s shared file storage to enable clustering. Each node requires manual unsealing with a key, which ensures security but is time-consuming. Let’s verify the setup before transitioning to auto-unseal.

Diagram

Step 1: Verify the Existing Cluster

Assuming you have the Part 3 setup, verify the cluster:

1. Navigate to vault-cluster and check containers:

    docker ps
   

   You should see vault1, vault2, and vault3.

2. Check cluster status on vault1:

    export VAULT_ADDR=http://localhost:8200
    export VAULT_TOKEN=<root-token-from-part3>
    vault operator raft list-peers
   

   Sample Response:

    Node      Address          State       Voter
    ----      -------          -----       -----
    vault1    vault1:8201      leader      true
    vault2    vault2:8201      follower    true
    vault3    vault3:8201      follower    true
   

3. Test manual unseal by restarting:

    docker-compose down
    docker-compose up -d
   

4. Unseal each node (using keys from Part 3):

    export VAULT_ADDR=http://localhost:8200
    vault operator unseal <vault1-unseal-key>
    export VAULT_ADDR=http://localhost:8201
    vault operator unseal <vault2-unseal-key>
    export VAULT_ADDR=http://localhost:8202
    vault operator unseal <vault3-unseal-key>
   

5. Verify a secret:

    export VAULT_ADDR=http://localhost:8200
    vault kv get secret/my-app
   

This manual process underscores the need for automation.

---

Transit-Based Auto-Unseal with Dedicated Vault

Manual unsealing is tedious, time-consuming, and prone to errors, especially in a production environment where downtime matters. Auto-unseal offers a better way by automating the process, making Vault startup faster and more reliable. Now, we’ll introduce a completely separate vault-transit container to host the transit secrets engine, enabling auto-unseal for vault1, vault2, and vault3. This standalone vault-transit instance ensures isolation from the cluster. We’ll set up vault-transit first, then reconfigure the cluster to use auto-unseal, eliminating manual key entry.

Diagram

Step 2: Update Docker Compose

Update docker-compose.yml to add vault-transit, keeping separate volumes:

services:
  vault1:
    image: hashicorp/vault:latest
    container_name: vault1
    ports:
      - "8200:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault1-data:/vault/file
      - ./vault-config/vault1:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault2:
    image: hashicorp/vault:latest
    container_name: vault2
    ports:
      - "8201:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault2-data:/vault/file
      - ./vault-config/vault2:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault3:
    image: hashicorp/vault:latest
    container_name: vault3
    ports:
      - "8202:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault3-data:/vault/file
      - ./vault-config/vault3:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault-transit:
    image: hashicorp/vault:latest
    container_name: vault-transit
    ports:
      - "8203:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault-transit-data:/vault/file
      - ./vault-config/vault-transit:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

volumes:
  vault1-data:
  vault2-data:
  vault3-data:
  vault-transit-data:

networks:
  vault-net:
    driver: bridge

Key Notes:

• Retains Part 3’s separate volumes (vault1-data, vault2-data, vault3-data) for Raft.

• Adds vault-transit-data for vault-transit.

• vault-net ensures communication.

Step 3: Configure Transit Vault

1. Create a vault-transit subdirectory in vault-config and add vault.hcl:

   vault-config/vault-transit/vault.hcl:

    ui = true
    api_addr = "http://vault-transit:8200"
   
    storage "file" {
      path = "/vault/file"
    }
   
    listener "tcp" {
      address = "0.0.0.0:8200"
      tls_disable = 1
    }
   

2. Start containers:

    docker-compose up -d
   

3. Initialize vault-transit:

    export VAULT_ADDR=http://localhost:8203
    vault operator init -key-shares=1 -key-threshold=1
   

   Sample Response:

    Unseal Key 1: <transit-unseal-key>
    Initial Root Token: s.<transit-root-token>
   
    Vault initialized with 1 key shares and a key threshold of 1.
    Please securely distribute the key shares printed above.
   

4. Unseal vault-transit:

    vault operator unseal <transit-unseal-key>
   

5. Log in:

    export VAULT_TOKEN=<transit-root-token>
    vault login
   

6. Enable transit engine:

    vault secrets enable -path=transit transit
   

   Sample Response:

    Success! Enabled the transit secrets engine at: transit/
   

7. Create an unseal key:

    vault write -f transit/keys/unseal_key
   

   Sample Response:

    Success! Data written to: transit/keys/unseal_key
   

8. Create a policy:

    vault policy write transit-unseal - <<EOF
    path "transit/encrypt/unseal_key" {
      capabilities = ["update"]
    }
    path "transit/decrypt/unseal_key" {
      capabilities = ["update"]
    }
    EOF
   

9. Create a transit token:

    vault token create -policy=transit-unseal -format=json | jq -r .auth.client_token > transit-token.txt
   

   Sample Response:

    {
      "auth": {
        "client_token": "<transit-token>",
        "policies": ["default", "transit-unseal"],
        "lease_duration": 2764800,
        ...
      }
    }
   

Step 4: Update Cluster for Auto-Unseal

1. Update vault1/vault.hcl, vault2/vault.hcl, and vault3/vault.hcl to add the seal stanza with the transit token from transit-token.txt:

   vault-config/vault1/vault.hcl:

    ui = true
    api_addr = "http://vault1:8200"
    cluster_addr = "http://vault1:8201"
   
    storage "raft" {
      path = "/vault/file"
      node_id = "vault1"
      retry_join {
        leader_api_addr = "http://vault2:8200"
        leader_api_addr = "http://vault3:8200"
      }
    }
   
    listener "tcp" {
      address = "0.0.0.0:8200"
      tls_disable = 1
    }
   
    seal "transit" {
      address = "http://vault-transit:8200"
      token = "<transit-token>"
      mount_path = "transit/"
    }
   

   vault-config/vault2/vault.hcl:

    ui = true
    api_addr = "http://vault2:8200"
    cluster_addr = "http://vault2:8201"
   
    storage "raft" {
      path = "/vault/file"
      node_id = "vault2"
      retry_join {
        leader_api_addr = "http://vault1:8200"
        leader_api_addr = "http://vault3:8200"
      }
    }
   
    listener "tcp" {
      address = "0.0.0.0:8200"
      tls_disable = 1
    }
   
    seal "transit" {
      address = "http://vault-transit:8200"
      token = "<transit-token>"
      mount_path = "transit/"
    }
   

   vault-config/vault3/vault.hcl:

    ui = true
    api_addr = "http://vault3:8200"
    cluster_addr = "http://vault3:8201"
   
    storage "raft" {
      path = "/vault/file"
      node_id = "vault3"
      retry_join {
        leader_api_addr = "http://vault1:8200"
        leader_api_addr = "http://vault2:8200"
      }
    }
   
    listener "tcp" {
      address = "0.0.0.0:8200"
      tls_disable = 1
    }
   
    seal "transit" {
      address = "http://vault-transit:8200"
      token = "<transit-token>"
      mount_path = "transit/"
    }
   

2. Restart cluster nodes:

    docker-compose restart vault1 vault2 vault3
   

3. Initialize cluster nodes:

    export VAULT_ADDR=http://localhost:8200
    vault operator init -key-shares=1 -key-threshold=1
   

   Sample Response:

    Success! Vault is initialized
    Initial Root Token: <cluster-root-token>
   
    Vault initialized with 1 key shares and a key threshold of 1.
    Please securely distribute the key shares printed above.
   

   If you’re continuing from Part 3, you might wonder why reinitialization is needed. Switching from manual unsealing to transit-based auto-unseal requires a new keyring compatible with the transit secrets engine, as the existing Raft data is tied to manual unseal keys. Reinitializing applies the new seal configuration while preserving your data in the Raft storage volumes (vault1-data, vault2-data, vault3-data). Initializing one node (e.g., vault1) is sufficient, as Raft replicates the state to vault2 and vault3 via retry_join. All nodes auto-unseal automatically using vault-transit, so no unseal keys or additional initialization is needed.

Step 5: Test the Cluster

1. Log in to vault1:

    export VAULT_ADDR=http://localhost:8200
    export VAULT_TOKEN=<cluster-root-token>
    vault login
   

2. Store a secret:

    vault kv put secret/my-app db_password=supersecret
   

3. Retrieve from vault2:

    export VAULT_ADDR=http://localhost:8201
    vault kv get secret/my-app
   

4. Check cluster status:

    vault operator raft list-peers
   

   Sample Response:

    Node      Address          State       Voter
    ----      -------          -----       -----
    vault1    vault1:8201      leader      true
    vault2    vault2:8201      follower    true
    vault3    vault3:8201      follower    true
   

Step 6: Access Vault

• Web UI: Open http://localhost:8200, 8201, or 8202 for the cluster; 8203 for vault-transit. Use the cluster or transit root token, respectively.

• CLI: Use any cluster node’s address (e.g., VAULT_ADDR=http://localhost:8200).

• Secrets persist via separate Raft storage volumes.

Step 7: Stop the Cluster

docker-compose down

On restart, unseal vault-transit manually; cluster nodes auto-unseal.

---

Best Practices

• Use KMS or HSM in Production: Transit is for learning. Use a KMS (e.g., AWS KMS, Google Cloud KMS) or HSM for auto-unseal. See Vault Auto-Unseal Docs.

• Secure Tokens: Store transit and root tokens in a password manager or HSM. Rotate transit tokens:

    vault token revoke <transit-token>
    vault token create -policy=transit-unseal
  

• Multiple Keys in Production: Use multiple unseal keys (e.g., three out of five) for vault-transit.

• Enable TLS: Set tls_disable = 0 and use certificates.

• Monitor Raft and Retry Join: Ensure Raft storage is backed up and retry_join targets are stable.

• Backup Storage: Back up vault1-data, vault2-data, vault3-data, and vault-transit-data volumes.

• Isolate Transit Vault: The separate vault-transit instance enhances security.

---

Troubleshooting

• Retry Join Fails: Verify target nodes (curl http://vault1:8200/v1/sys/health). Check vault-net (docker network inspect vault-net).

• Auto-Unseal Fails: Ensure vault-transit is unsealed and the transit token is valid. Check logs (docker logs vault1).

• Raft Issues: Verify separate volumes (docker inspect vault1). Check logs (docker logs vault1).

• Token Errors: Confirm the transit token matches transit-token.txt and has the transit-unseal policy.

---

What’s Next?

You’ve transitioned your Vault cluster to transit-based auto-unseal! To extend your learning:

• Adapt MySQL dynamic secrets from Part 2.

• Test leader failover by stopping vault1 and checking vault operator raft list-peers.

• Explore KMS-based auto-unseal using Vault Auto-Unseal Docs.

Share your progress in the comments or join the HashiCorp Community Forum!

---

Resources

• HashiCorp Vault Documentation: vaultproject.io

• Vault Auto-Unseal Docs: vaultproject.io/docs/configuration/seal

• Vault HA Documentation: vaultproject.io/docs/concepts/ha

• Docker Compose Documentation: docs.docker.com/compose

Note: For questions or setup help, comment below or check the Vault documentation.