# HashiCorp Vault with Docker Compose: Advanced Clustering and Auto-Unseal (Part 4) # HashiCorp Vault with Docker Compose: Advanced Auto-Unseal with Transit (Part 4) **Subtitle**: Transition from manual unsealing to auto-unseal using a dedicated transit Vault In [Part 3](hashicorp-vault-docker-compose-part3), you built a three-node Vault cluster using Raft storage and `retry_join` for automated clustering, with separate storage volumes for high availability (HA). However, each node required manual unsealing, which is cumbersome. In this final, comprehensive part of our series, we’ll enhance the cluster by implementing **auto-unseal** using the **transit secrets engine** hosted on a dedicated Vault container, eliminating manual key entry. For simplicity, we use transit, but **production environments should use a Key Management Service (KMS)** (e.g., AWS KMS, Azure Key Vault) or **Hardware Security Module (HSM)** for enhanced security. This intermediate-to-advanced guide builds on Parts 1-3, delivering a production-like setup. --- ## Prerequisites Before starting, ensure you have: * **Docker** (version 20.10 or later). Install from [Docker’s guide](https://docs.docker.com/get-docker/). * **Docker Compose** (version 2.0 or later). See [Docker Compose installation](https://docs.docker.com/compose/install/). * **Vault CLI**. Download from [releases.hashicorp.com/vault](https://releases.hashicorp.com/vault/) or use a package manager (e.g., `brew install vault` on macOS). * **jq** for parsing JSON (optional, for token extraction). Install via `brew install jq` (macOS) or equivalent. * A text editor (e.g., VS Code). * Familiarity with Parts 1-3 (development mode, persistent storage, Raft clustering with `retry_join`). * Commands are tested on Linux/macOS; Windows users may need to use `set` instead of `export`. --- ## Reviewing the Cluster with Manual Unseal In Part 3, you set up a three-node cluster (`vault1`, `vault2`, `vault3`) using Raft storage for HA, with `retry_join` for automated node discovery and separate volumes (`vault1-data`, `vault2-data`, `vault3-data`) to support Raft’s independent storage needs. As explained in Part 3, this transitioned from Part 2’s shared file storage to enable clustering. Each node requires manual unsealing with a key, which ensures security but is time-consuming. Let’s verify the setup before transitioning to auto-unseal. ### Diagram ```mermaid graph TD A[Client: Browser/CLI] --> B[Docker Compose] B --> C[Vault Node 1: Port 8200
Manual Unseal] B --> D[Vault Node 2: Port 8201
Manual Unseal] B --> E[Vault Node 3: Port 8202
Manual Unseal] C --> F[Vault1 Data] D --> G[Vault2 Data] E --> H[Vault3 Data] ``` ### Step 1: Verify the Existing Cluster Assuming you have the Part 3 setup, verify the cluster: 1. Navigate to `vault-cluster` and check containers: ```bash docker ps ``` You should see `vault1`, `vault2`, and `vault3`. 2. Check cluster status on `vault1`: ```bash export VAULT_ADDR=http://localhost:8200 export VAULT_TOKEN= vault operator raft list-peers ``` **Sample Response**: ```bash Node Address State Voter ---- ------- ----- ----- vault1 vault1:8201 leader true vault2 vault2:8201 follower true vault3 vault3:8201 follower true ``` 3. Test manual unseal by restarting: ```bash docker-compose down docker-compose up -d ``` 4. Unseal each node (using keys from Part 3): ```bash export VAULT_ADDR=http://localhost:8200 vault operator unseal export VAULT_ADDR=http://localhost:8201 vault operator unseal export VAULT_ADDR=http://localhost:8202 vault operator unseal ``` 5. Verify a secret: ```bash export VAULT_ADDR=http://localhost:8200 vault kv get secret/my-app ``` This manual process underscores the need for automation. --- ## Transit-Based Auto-Unseal with Dedicated Vault Manual unsealing is tedious, time-consuming, and prone to errors, especially in a production environment where downtime matters. Auto-unseal offers a better way by automating the process, making Vault startup faster and more reliable. Now, we’ll introduce a completely separate vault-transit container to host the transit secrets engine, enabling auto-unseal for vault1, vault2, and vault3. This standalone vault-transit instance ensures isolation from the cluster. We’ll set up vault-transit first, then reconfigure the cluster to use auto-unseal, eliminating manual key entry. ### Diagram ```mermaid graph TD A[Client: Browser/CLI] --> B[Docker Compose] B --> C[Vault Node 1: Port 8200
Transit Auto-Unseal] B --> D[Vault Node 2: Port 8201
Transit Auto-Unseal] B --> E[Vault Node 3: Port 8202
Transit Auto-Unseal] B --> F[Vault-Transit: Port 8203
Transit Engine] C --> G[Vault1 Data] D --> H[Vault2 Data] E --> I[Vault3 Data] F --> J[Vault-Transit Data] C --> F[Transit Auto-Unseal] D --> F[Transit Auto-Unseal] E --> F[Transit Auto-Unseal] ``` ### Step 2: Update Docker Compose Update `docker-compose.yml` to add `vault-transit`, keeping separate volumes: ```yaml services: vault1: image: hashicorp/vault:latest container_name: vault1 ports: - "8200:8200" environment: - VAULT_ADDR=http://0.0.0.0:8200 cap_add: - IPC_LOCK volumes: - vault1-data:/vault/file - ./vault-config/vault1:/vault/config command: vault server -config=/vault/config/vault.hcl networks: - vault-net vault2: image: hashicorp/vault:latest container_name: vault2 ports: - "8201:8200" environment: - VAULT_ADDR=http://0.0.0.0:8200 cap_add: - IPC_LOCK volumes: - vault2-data:/vault/file - ./vault-config/vault2:/vault/config command: vault server -config=/vault/config/vault.hcl networks: - vault-net vault3: image: hashicorp/vault:latest container_name: vault3 ports: - "8202:8200" environment: - VAULT_ADDR=http://0.0.0.0:8200 cap_add: - IPC_LOCK volumes: - vault3-data:/vault/file - ./vault-config/vault3:/vault/config command: vault server -config=/vault/config/vault.hcl networks: - vault-net vault-transit: image: hashicorp/vault:latest container_name: vault-transit ports: - "8203:8200" environment: - VAULT_ADDR=http://0.0.0.0:8200 cap_add: - IPC_LOCK volumes: - vault-transit-data:/vault/file - ./vault-config/vault-transit:/vault/config command: vault server -config=/vault/config/vault.hcl networks: - vault-net volumes: vault1-data: vault2-data: vault3-data: vault-transit-data: networks: vault-net: driver: bridge ``` **Key Notes**: * Retains Part 3’s separate volumes (`vault1-data`, `vault2-data`, `vault3-data`) for Raft. * Adds `vault-transit-data` for `vault-transit`. * `vault-net` ensures communication. ### Step 3: Configure Transit Vault 1. Create a `vault-transit` subdirectory in `vault-config` and add `vault.hcl`: **vault-config/vault-transit/vault.hcl**: ```bash ui = true api_addr = "http://vault-transit:8200" storage "file" { path = "/vault/file" } listener "tcp" { address = "0.0.0.0:8200" tls_disable = 1 } ``` 2. Start containers: ```bash docker-compose up -d ``` 3. Initialize `vault-transit`: ```bash export VAULT_ADDR=http://localhost:8203 vault operator init -key-shares=1 -key-threshold=1 ``` **Sample Response**: ```bash Unseal Key 1: Initial Root Token: s. Vault initialized with 1 key shares and a key threshold of 1. Please securely distribute the key shares printed above. ``` 4. Unseal `vault-transit`: ```bash vault operator unseal ``` 5. Log in: ```bash export VAULT_TOKEN= vault login ``` 6. Enable transit engine: ```bash vault secrets enable -path=transit transit ``` **Sample Response**: ```bash Success! Enabled the transit secrets engine at: transit/ ``` 7. Create an unseal key: ```bash vault write -f transit/keys/unseal_key ``` **Sample Response**: ```bash Success! Data written to: transit/keys/unseal_key ``` 8. Create a policy: ```bash vault policy write transit-unseal - < transit-token.txt ``` **Sample Response**: ```bash { "auth": { "client_token": "", "policies": ["default", "transit-unseal"], "lease_duration": 2764800, ... } } ``` ### Step 4: Update Cluster for Auto-Unseal 1. Update `vault1/vault.hcl`, `vault2/vault.hcl`, and `vault3/vault.hcl` to add the `seal` stanza with the transit token from `transit-token.txt`: **vault-config/vault1/vault.hcl**: ```bash ui = true api_addr = "http://vault1:8200" cluster_addr = "http://vault1:8201" storage "raft" { path = "/vault/file" node_id = "vault1" retry_join { leader_api_addr = "http://vault2:8200" leader_api_addr = "http://vault3:8200" } } listener "tcp" { address = "0.0.0.0:8200" tls_disable = 1 } seal "transit" { address = "http://vault-transit:8200" token = "" mount_path = "transit/" } ``` **vault-config/vault2/vault.hcl**: ```bash ui = true api_addr = "http://vault2:8200" cluster_addr = "http://vault2:8201" storage "raft" { path = "/vault/file" node_id = "vault2" retry_join { leader_api_addr = "http://vault1:8200" leader_api_addr = "http://vault3:8200" } } listener "tcp" { address = "0.0.0.0:8200" tls_disable = 1 } seal "transit" { address = "http://vault-transit:8200" token = "" mount_path = "transit/" } ``` **vault-config/vault3/vault.hcl**: ```bash ui = true api_addr = "http://vault3:8200" cluster_addr = "http://vault3:8201" storage "raft" { path = "/vault/file" node_id = "vault3" retry_join { leader_api_addr = "http://vault1:8200" leader_api_addr = "http://vault2:8200" } } listener "tcp" { address = "0.0.0.0:8200" tls_disable = 1 } seal "transit" { address = "http://vault-transit:8200" token = "" mount_path = "transit/" } ``` 2. Restart cluster nodes: ```bash docker-compose restart vault1 vault2 vault3 ``` 3. Initialize cluster nodes: ```bash export VAULT_ADDR=http://localhost:8200 vault operator init -key-shares=1 -key-threshold=1 ``` **Sample Response**: ```bash Success! Vault is initialized Initial Root Token: Vault initialized with 1 key shares and a key threshold of 1. Please securely distribute the key shares printed above. ``` *If you’re continuing from Part 3, you might wonder why reinitialization is needed. Switching from manual unsealing to transit-based auto-unseal requires a new keyring compatible with the transit secrets engine, as the existing Raft data is tied to manual unseal keys. Reinitializing applies the new seal configuration while preserving your data in the Raft storage volumes (vault1-data, vault2-data, vault3-data). Initializing one node (e.g., vault1) is sufficient, as Raft replicates the state to vault2 and vault3 via retry\_join. All nodes auto-unseal automatically using vault-transit, so no unseal keys or additional initialization is needed.* ### Step 5: Test the Cluster 1. Log in to `vault1`: ```bash export VAULT_ADDR=http://localhost:8200 export VAULT_TOKEN= vault login ``` 2. Store a secret: ```bash vault kv put secret/my-app db_password=supersecret ``` 3. Retrieve from `vault2`: ```bash export VAULT_ADDR=http://localhost:8201 vault kv get secret/my-app ``` 4. Check cluster status: ```bash vault operator raft list-peers ``` **Sample Response**: ```bash Node Address State Voter ---- ------- ----- ----- vault1 vault1:8201 leader true vault2 vault2:8201 follower true vault3 vault3:8201 follower true ``` ### Step 6: Access Vault * **Web UI**: Open `http://localhost:8200`, `8201`, or `8202` for the cluster; `8203` for `vault-transit`. Use the cluster or transit root token, respectively. * **CLI**: Use any cluster node’s address (e.g., `VAULT_ADDR=http://localhost:8200`). * Secrets persist via separate Raft storage volumes. ### Step 7: Stop the Cluster ```bash docker-compose down ``` On restart, unseal `vault-transit` manually; cluster nodes auto-unseal. --- ## Best Practices * **Use KMS or HSM in Production**: Transit is for learning. Use a KMS (e.g., AWS KMS, Google Cloud KMS) or HSM for auto-unseal. See [Vault Auto-Unseal Docs](https://www.vaultproject.io/docs/configuration/seal). * **Secure Tokens**: Store transit and root tokens in a password manager or HSM. Rotate transit tokens: ```bash vault token revoke vault token create -policy=transit-unseal ``` * **Multiple Keys in Production**: Use multiple unseal keys (e.g., three out of five) for `vault-transit`. * **Enable TLS**: Set `tls_disable = 0` and use certificates. * **Monitor Raft and Retry Join**: Ensure Raft storage is backed up and `retry_join` targets are stable. * **Backup Storage**: Back up `vault1-data`, `vault2-data`, `vault3-data`, and `vault-transit-data` volumes. * **Isolate Transit Vault**: The separate `vault-transit` instance enhances security. --- ## Troubleshooting * **Retry Join Fails**: Verify target nodes (`curl http://vault1:8200/v1/sys/health`). Check `vault-net` (`docker network inspect vault-net`). * **Auto-Unseal Fails**: Ensure `vault-transit` is unsealed and the transit token is valid. Check logs (`docker logs vault1`). * **Raft Issues**: Verify separate volumes (`docker inspect vault1`). Check logs (`docker logs vault1`). * **Token Errors**: Confirm the transit token matches `transit-token.txt` and has the `transit-unseal` policy. --- ## What’s Next? You’ve transitioned your Vault cluster to transit-based auto-unseal! To extend your learning: * Adapt MySQL dynamic secrets from [Part 2](hashicorp-vault-docker-compose-part2). * Test leader failover by stopping `vault1` and checking `vault operator raft list-peers`. * Explore KMS-based auto-unseal using [Vault Auto-Unseal Docs](https://www.vaultproject.io/docs/configuration/seal). Share your progress in the comments or join the [HashiCorp Community Forum](https://discuss.hashicorp.com)! --- ## Resources * **HashiCorp Vault Documentation**: [vaultproject.io](https://www.vaultproject.io) * **Vault Auto-Unseal Docs**: [vaultproject.io/docs/configuration/seal](https://www.vaultproject.io/docs/configuration/seal) * **Vault HA Documentation**: [vaultproject.io/docs/concepts/ha](https://www.vaultproject.io/docs/concepts/ha) * **Docker Compose Documentation**: [docs.docker.com/compose](https://docs.docker.com/compose) *Note*: For questions or setup help, comment below or check the Vault documentation.