• Vault Enterprise bills by client identity, not by token count
• Entity-backed authentication (LDAP, OIDC, userpass, AppRole with aliases) produces stable, predictable billing
• Non-entity tokens (direct token creation, orphan tokens, token auth) are billed by unique policy set and can grow unexpectedly
• Migrating to entity-backed auth is the only reliable way to control client counts

Current best practices to control client counts

1. Prefer entity-aware auth methods whenever possible
2. Use entity-aware auth for CI/CD and bootstrap (AppRole, Kubernetes auth, cloud IAM, JWT/OIDC)
3. For lifecycle independence, re-authenticate via entity-aware method instead of using orphan tokens
4. Use batch tokens (non-orphan) for short-lived credentials
5. Audit regularly with sys/internal/counters/activity and audit logs
6. Monitor for policy name changes and refactors (they create new client identities)

If you run Vault Enterprise in production, you will eventually encounter a moment where the client count doesn't match your expectations.

Nothing changed in your workloads. Nothing new was deployed. And yet the number moved—sometimes sharply.

This post exists to explain why.

Not based on assumptions. Not based on sales slides. But based on controlled, repeatable testing against Vault Enterprise.

The short version is this:

Vault Enterprise does not count tokens. It counts clients. And client identity is derived in two fundamentally different ways, depending on how a token was created.

Understanding that distinction explains most billing surprises—and helps you avoid creating them.

Scope and methodology

All findings below are based on:

• Vault Enterprise 1.21.1
• Raft storage
• Tokens were actually exercised against Vault (unused tokens do not count as clients)
• Client counts observed via sys/internal/counters/activity

Where Vault behavior is documented, I say so. Where behavior is empirical, I am explicit about that.

The two client identity models

Vault Enterprise uses two distinct client identity models.

1. Entity-backed clients (stable, predictable)

When authentication happens via an entity-aware auth method, Vault assigns an entity_id.

Examples:

• userpass
• LDAP
• OIDC / JWT
• AppRole

In this model:

• All tokens associated with the same entity_id count as one client
• Token policies do not affect client count
• Child tokens inherit the entity automatically

This behavior is documented and stable.

Note: This deduplication can be dramatic. A single entity-backed user with sudo capability on auth/token/create can mint tokens with any policy—not just subsets of their own. This bypasses the normal "child policies must be subset of parent" rule (see token create documentation). The result: one user creating hundreds of unique policy set combinations, all collapsing to one client. This is efficient for billing, but means client count no longer reflects actual token or policy diversity.

Effectively, client identity behaves as:

client ~ namespace + entity_id

2. Non-entity clients (policy-based, unstable)

When a token is created without entity context, Vault assigns no entity_id.

In this case, client identity is derived from the policy name combination attached to the token.

Empirically, client identity behaves as if derived from:

client ~ namespace + sorted(unique_policy_names)

This behavior is not fully documented, but it is consistent, repeatable, and observable across versions. And it is the source of most billing surprises.

Important: Since Vault 1.9, non-entity tokens with identical namespace + policy sets are deduplicated into a single client (via a constructed entity). Pre-1.9 behavior was much worse—often counting one client per token. The combinatorial explosion described below still applies, but only unique policy sets count.

The policy-set explosion

For non-entity tokens, each unique set of attached policy names creates a separate billed client. Order is irrelevant—[p1, p2] and [p2, p1] are the same client.

Example:

[p1]        -> client A
[p2]        -> client B
[p1, p2]    -> client C
[p1, p3]    -> client D

Four unique policy sets. Four billable clients.

With N policies, the worst case is:

2^N - 1 clients

This is combinatorial growth, not theoretical—and it appears quickly in real systems.

In a real environment with 10 policies, modest variation in policy sets can already produce hundreds of clients where entity-backed authentication would produce one.

How non-entity tokens are created

Non-entity tokens are not edge cases. They are easy to create—often accidentally.

Confirmed creation paths

Direct token creation

vault token create

Common during bootstrap, automation, or CI/CD.

Token chains from a non-entity parent

If the parent token has no entity, children inherit none. The entire chain remains non-entity forever.

Token auth method

auth/token/create

Tokens created via the token auth method have no entity by design.

Orphan tokens

vault token create -orphan

Even when created by an entity-backed user, orphan tokens explicitly drop entity context.

Once a token is non-entity, policy-set-based identity rules apply and client counting becomes unstable.

These paths aren't rare edge cases—they're everyday automation patterns that sneak in during bootstrap, scaling, or CI/CD integration.

Why orphan tokens exist

Orphan tokens solve a real operational problem: lifecycle independence.

They are commonly used for:

• Long-running batch jobs
• CI/CD pipelines
• Service bootstrap tokens
• Cross-team delegation
• Break-glass scenarios

In all cases, the requirement is the same:

"This token must survive the revocation or expiration of the token that created it."

That is exactly what orphan tokens do. But there is a cost.

The critical orphan token rule

Orphan tokens never inherit entity_id. Not sometimes. Not conditionally. Never.

This was tested via:

• Orphan creation from root
• Orphan creation from entity-backed users with sudo
• Orphan token child chains

In all cases:

• entity_id = n/a
• Token is treated as non-entity
• Policy-based client identity applies

Child vs orphan tokens

There is no native Vault mechanism that provides both:

• lifecycle independence and
• entity-anchored client identity

If you need both, you must re-authenticate via an entity-aware auth method.

Children of orphan tokens

Orphan tokens do not inherit entity context. Neither do their children.

Entity user (entity_id: abc-123)
    |
    +-- [creates orphan] --> orphan token (entity_id: n/a)
                                  |
                                  +-- child A (entity_id: n/a)
                                  |       |
                                  |       +-- grandchild (entity_id: n/a)
                                  |
                                  +-- child B (entity_id: n/a)

Once a token becomes non-entity, the entire chain remains non-entity forever. There is no mechanism to recover entity context downstream—the only option is to discard the chain and re-authenticate.

Batch tokens: the safe alternative

Batch tokens behave differently—and this matters.

Batch tokens inherit entity_id like service tokens, but can't be renewed or create children—making them safer for short-lived automation where you don't want token sprawl.

vault token create -type=batch -policy=p1

Result:

• entity_id inherited
• No new client created

However:

vault token create -type=batch -orphan -policy=p1

Result:

• entity_id lost
• New non-entity client (based on policy set)

Batch vs service tokens

Batch tokens are appropriate when you need:

• Short-lived credentials
• No renewal
• No child tokens
• Entity-anchored billing

From a billing perspective, batch tokens behave like entity-anchored child tokens—unless created with the -orphan flag.

Remediation options

If you're currently using non-entity tokens and want to reduce client count:

The common theme: re-authenticate via an entity-aware method rather than creating tokens from tokens.

What does not affect client identity

Explicitly tested and not used in client counting:

• Token accessor
• Token creation time
• Parent token
• Periodic vs non-periodic
• Policy content

Policy name matters. Renaming a policy creates a new client. Changing what it allows does not.

Namespaces matter

Client identity is namespace-scoped, which means the same policy set used in different namespaces counts as different clients. This is expected behavior, but it's easy to overlook in multi-namespace deployments where teams might unknowingly duplicate policy structures across namespaces.

Auditing your current exposure

vault read sys/internal/counters/activity/monthly

Warning signs

Look for these patterns in your activity data:

• Non-entity clients significantly outnumber entity clients
• Client count grows after policy refactors or renames
• Many clients with similar or overlapping policy sets
• Client spikes without corresponding workload changes

If you see these patterns, you likely have policy-set-based client identity at work—and it may be worth investigating migration options.

Documentation and stability

HashiCorp explicitly warns that token behavior may change across versions (see the token documentation). That alone is reason to be cautious about relying on undocumented heuristics for billing-sensitive behavior. The patterns described in this post have been stable since Vault 1.9, but always verify after major upgrades.

Why this matters

This behavior enables both:

Over-counting Policy set sprawl leads to combinatorial client growth, which leads to billing surprises.

Under-counting Many workloads anchored under a single entity can collapse into fewer clients than expected—which might seem like a win until you realize your billing doesn't reflect actual usage patterns.

Real-world example: dozens of microservices as 1 client

A client with a lot of microservices each using distinct policy sets but Vault bills them as a single client.

How it works:

1. Entity-backed auth (userpass or AppRole) → gets entity_id
2. sudo on auth/token/create → bypasses "child must be subset of parent" rule
3. Creates tokens with any arbitrary policy combination
4. All tokens inherit same entity_id → 1 client

This is not a bug. It's how entity deduplication works when combined with sudo capability on token creation (see token create documentation).

Trade-off: Billing is efficient, but client count no longer reflects workload diversity. To maintain visibility, consider using audit logs with client ID correlation, or split into multiple entities where usage diversity matters for reporting.

This behavior isn't misuse or misconfiguration. It's simply how Vault works today. The mechanics are consistent and predictable once you understand them, but most teams learn the hard way—staring at an unexpected invoice wondering what changed.

This post exists so you don't have to go through that.

Tested on Vault Enterprise 1.21.1 (Jan 2026). Behavior stable since 1.9, but always verify after upgrades as token mechanics may evolve.

Cover Image: A digital padlock on a circuit board, symbolizing secure secrets management. (Source: Unsplash)

Prerequisites

Before starting, ensure you have:

• Docker (version 20.10 or later). Install from Docker’s guide.

• Docker Compose (version 2.0 or later). See Docker Compose installation.

• Vault CLI for CLI access. Download from releases.hashicorp.com/vault or use a package manager (e.g., brew install vault on macOS).

• A text editor (e.g., VS Code).

• Familiarity with Parts 1 and 2 (development mode, persistent storage, unsealing).

• Commands are tested on Linux/macOS; Windows users may need to use set instead of export for environment variables.

Setting Up a Three-Node Vault Cluster

A Vault cluster consists of multiple nodes sharing the same storage backend, with one node acting as the leader and others as followers. If the leader fails, a follower takes over. Below is a diagram of the setup:

graph TD
    A[Client: Browser/CLI] --> B[Docker Compose]
    B --> C[Vault Node 1: Port 8200]
    B --> D[Vault Node 2: Port 8201]
    B --> E[Vault Node 3: Port 8202]
    C --> F[Shared Storage]
    D --> F
    E --> F

Step 1: Create the Docker Compose File

Create a file named docker-compose.yml in a new directory (e.g., vault-cluster):

services:
  vault1:
    image: hashicorp/vault:latest
    container_name: vault1
    ports:
      - "8200:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault1-data:/vault/file
      - ./vault-config/vault1:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault2:
    image: hashicorp/vault:latest
    container_name: vault2
    ports:
      - "8201:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault2-data:/vault/file
      - ./vault-config/vault2:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

  vault3:
    image: hashicorp/vault:latest
    container_name: vault3
    ports:
      - "8202:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault3-data:/vault/file
      - ./vault-config/vault3:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    networks:
      - vault-net

volumes:
  vault1-data:
  vault2-data:
  vault3-data:

networks:
  vault-net:
    driver: bridge

Key Notes:

• Separate volumes (vault1-data, vault2-data, vault3-data) ensure Raft storage is independent per node.

• Each node maps a unique config directory (vault-config/vault1, etc.) and exposes a different host port (8200, 8201, 8202).

• vault-net: A custom network ensures nodes can communicate.

• Inside containers, Vault listens on port 8200, mapped to unique host ports.

Step 2: Create Vault Configuration Files

To simplify cluster setup, we could manually join nodes using vault operator raft join, but this approach is tedious and error-prone, requiring careful coordination for each node. Instead, we’ll use retry_join to automate node discovery, making the process faster, more reliable, and less boring. This configuration allows each node to automatically find and join the cluster by contacting other nodes, streamlining setup and improving resilience.

Moving to Raft storage

In Part 2, we used file storage with a shared volume across containers for simplicity, enabling a single-node Vault to persist data easily. In Part 3, we switch to Raft storage, which is integrated into Vault and designed for high availability clustering. Raft ensures each node maintains its own consistent state and supports leader election and replication, making it ideal for a resilient, multi-node cluster. Additionally, unlike Part 2’s shared volume, Part 3 uses separate storage volumes for each node (vault1-data, vault2-data, vault3-data), eliminating shared storage dependencies and aligning with Raft’s requirement for independent node data.

Create a vault-config directory with subdirectories: vault1, vault2, vault3. Add vault.hcl files following the specified pattern:

vault-config/vault1/vault.hcl:

ui = true
api_addr = "http://vault1:8200"
cluster_addr = "http://vault1:8201"

storage "raft" {
  path = "/vault/file"
  node_id = "vault1"

  retry_join {
    leader_api_addr = "http://vault2:8200"
    leader_api_addr = "http://vault3:8200"
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

The retry_join configuration automates cluster formation by enabling each Vault node to attempt joining other nodes’ API addresses (e.g., vault1 tries vault2 and vault3) until successful. This resilient mechanism retries on failure and excludes the current node to avoid self-joining, ensuring robust clustering. By specifying multiple leader_api_addr entries, retry_join simplifies setup and enhances reliability compared to manual joining.

vault-config/vault2/vault.hcl:

ui = true
api_addr = "http://vault2:8200"
cluster_addr = "http://vault2:8201"

storage "raft" {
  path = "/vault/file"
  node_id = "vault2"

  retry_join {
    leader_api_addr = "http://vault1:8200"
    leader_api_addr = "http://vault3:8200"
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

vault-config/vault3/vault.hcl:

ui = true
api_addr = "http://vault3:8200"
cluster_addr = "http://vault3:8201"

storage "raft" {
  path = "/vault/file"
  node_id = "vault3"

  retry_join {
    leader_api_addr = "http://vault1:8200"
    leader_api_addr = "http://vault2:8200"
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

Key Notes:

• storage "raft": Uses Raft for HA, with a unique node_id per node.

• retry_join: Excludes the current node (e.g., vault1 joins vault2 and vault3), improving resilience.

• tls_disable = 1: Disabled for simplicity; production requires TLS.

• Production Warning: Use multiple unseal keys and TLS in production.

• api_addr: Specifies the node’s API address for client communication.

• cluster_addr: Defines the address for cluster communication (port 8201 to avoid conflicts with 8200).

Step 3: Start and Initialize the Cluster

1. Navigate to vault-cluster and start the containers:

    docker-compose up -d
   

2. Verify the containers are running:

    docker ps
   

   You should see vault1, vault2, and vault3.

3. Initialize the leader node (vault1):

    export VAULT_ADDR=http://localhost:8200
    vault operator init -key-shares=1 -key-threshold=1
   
    Unseal Key 1: <unseal-key>
    Initial Root Token: <root-token>
   
    Vault initialized with 1 key shares and a key threshold of 1.
    Please securely distribute the key shares printed above.
   

4. This outputs one unseal key and a root token. Save the unseal key and root token securely (e.g., in a password manager). As in Part 2, we’re using a single key for simplicity, but production clusters typically use multiple keys (e.g., three out of five) for security. Do not use a single key in production.

5. Unseal vault1:

    vault operator unseal <unseal-key>
   

6. Log in with the root token:

    export VAULT_TOKEN=<root-token>
    vault login
   

7. Join vault2 to the cluster:

    export VAULT_ADDR=http://localhost:8201
    vault operator init -key-shares=1 -key-threshold=1
    vault operator unseal <vault2-unseal-key>
   

8. Join vault3 to the cluster:

    export VAULT_ADDR=http://localhost:8202
    vault operator init -key-shares=1 -key-threshold=1
    vault operator unseal <vault3-unseal-key>
   

Step 4: Test the Cluster

1. Store a secret on vault1:

    export VAULT_ADDR=http://localhost:8200
    vault kv put secret/my-app db_password=supersecret
   

2. Retrieve it from vault2:

    export VAULT_ADDR=http://localhost:8201
    vault kv get secret/my-app
   

3. Check cluster status:

    vault operator raft list-peers
   
    Node      Address          State       Voter
    ----      -------          -----       -----
    vault1    vault1:8201      leader      true
    vault2    vault2:8201      follower    true
    vault3    vault3:8201      follower    true
   

   This shows all three nodes, with vault1 as the leader.

Step 5: Access Vault

• Web UI: Open http://localhost:8200, http://localhost:8201, or http://localhost:8202. Log in with the root token. Navigate to “Secrets” to view or create secrets.

• CLI: Use any node’s address to interact with the cluster (e.g., VAULT_ADDR=http://localhost:8200).

• Secrets persist via separate Raft storage volumes.

Step 6: Stop the Cluster

To stop the containers:

docker-compose down

On restart, unseal each node using the same unseal key.

Best Practices

• Use Multiple Keys in Production: Configure multiple unseal keys (e.g., three out of five) for security. Avoid a single key.

• Monitor Retry Join: Ensure retry_join targets are stable. Use cloud metadata or multiple addresses in production.

• Backup Raft Storage: Regularly back up vault1-data, vault2-data, and vault3-data volumes.

• Secure Unseal Key and Root Token: Store the unseal key and root token in a secure location (e.g., password manager or HSM). In production, use multiple keys (e.g., three out of five) per node.

• Ensure Network Reliability: Nodes must communicate via cluster_addr. Use a stable network (e.g., vault-net) and monitor connectivity.

• Monitor Leader Election: Check vault operator raft list-peers to ensure a leader is active. If the leader fails, a follower takes over automatically.

• Enable TLS: In production, set tls_disable = 0 in vault.hcl and use certificates for secure communication.

• Backup Storage: Regularly back up the vault-data volume to prevent data loss.

Troubleshooting

• Node Fails to Join: Verify vault1 is unsealed and accessible (curl http://vault1:8200/v1/sys/health). Check Docker network (vault-net) with docker network inspect vault-net.

• Raft Issues: Ensure separate volumes are mounted (docker inspect vault1). Check logs (docker logs vault1).

• Unseal Fails: Ensure the unseal key is correct. Re-run vault operator init -key-shares=1 -key-threshold=1 on vault1 if lost (this resets the cluster).

• Cluster Not Responding: Check logs (docker logs vault1) for errors. Ensure ports 8200-8202 are free (lsof -i :8200-8202).

What’s Next?

You’ve built a high-availability Vault cluster with Raft and retry_join! In Part 4, we’ll eliminate manual unsealing by implementing transit-based auto-unseal with a dedicated Vault instance. Try:

• Stop vault1 and verify a new leader is elected (vault operator raft list-peers).

• Adapting MySQL secrets from Part 2.

• Store a secret on one node and retrieve it from another.

For dynamic secrets (e.g., MySQL), revisit Part 2 and adapt the configuration for this cluster. Share your progress in the comments or join the HashiCorp Community Forum!

Resources

• HashiCorp Vault Documentation: vaultproject.io

• Vault HA Documentation: vaultproject.io/docs/concepts/ha

• Raft Storage Docs: vaultproject.io/docs/configuration/storage/raft

• Docker Compose Documentation: docs.docker.com/compose

Note: For questions or setup help, comment below or check the Vault documentation.

⚠️ In dev mode, all secrets are lost when the container stops.

In this second part of the series, we’ll take a small step toward a more realistic setup:

• Enable persistent storage so secrets survive restarts,

• Use manual unsealing (a core Vault security feature),

• Connect Vault to a MySQL database to issue dynamic secrets - temporary credentials that expire automatically.

This guide is still beginner-friendly and assumes no prior production Vault experience. You’ll build confidence by running everything locally with Docker, just like in Part 1, but with added layers of realism.

🧠 Why dynamic secrets?

Unlike static secrets, dynamic secrets are generated on demand. When an app or user needs access to a database, Vault can create a temporary username and password with limited permissions that self-destruct after an hour or so. This removes the need to manage and rotate credentials manually.

🔁 Quick Recap of Part 1

Previously, we:

• Ran Vault in development mode using Docker Compose.

• Stored secrets in memory (they were lost after each restart).

• Explored the UI and created static secrets manually.

Now, we're building a persistent, more production-like setup with real secret automation

Prerequisites

Before starting, ensure you have:

• Docker (version 20.10 or later). Install from Docker’s guide.

• Docker Compose (version 2.0 or later). See Docker Compose installation.

• Vault CLI for CLI access. Download from releases.hashicorp.com/vault or use a package manager (e.g., brew install vault on macOS).

• A text editor (e.g., VS Code).

• Familiarity with Part 1’s setup.

🪟 Windows users: Use set instead of export for environment variables.

Setting Up Vault with Persistent Storage

Step 1: Create the Docker Compose File

Create a file named docker-compose.yml in a new directory (e.g., vault-docker):

services:
  vault:
    image: hashicorp/vault:latest
    ports:
      - "8200:8200"
    environment:
      - VAULT_ADDR=http://0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    volumes:
      - vault-data:/vault/data
      - ./vault-config:/vault/config
    command: vault server -config=/vault/config/vault.hcl
    depends_on:
      - mysql
  mysql:
    image: mysql:8
    container_name: mysql
    environment:
      MYSQL_ROOT_PASSWORD: rootpass
    ports:
      - "3306:3306"

volumes:
  vault-data:

Key Notes:

• vault-data: A Docker volume for persistent storage, keeping secrets between restarts.

• vault-config: Maps a local directory for the Vault configuration file.

• mysql: A MySQL container for dynamic secrets, with a root password (rootpass).

• depends_on: Ensures MySQL starts before Vault for dynamic secrets setup.

Step 2: Create the Vault Configuration

Unlike dev mode, we now provide an actual configuration file (vault.hcl) that Vault reads when starting in server mode.

Create a vault-config directory in vault-docker and add a file named vault.hcl.

storage "file" {
  path = "/vault/data"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

ui = true

Key Notes:

• storage "file": Stores secrets in /vault/data, mapped to the vault-data volume.

• tls_disable = 1: Disables TLS for simplicity; in production, enable TLS with certificates.

• ui = true: Enables the Vault web UI.

Step 3: Start and Initialize Vault

1. Navigate to vault-docker folder and start the containers:

    docker-compose up -d
   

2. Verify the containers are running:

    docker ps
   

   You should see vault and mysql containers.

3. Initialize Vault (run once):

    export VAULT_ADDR=http://localhost:8200
    vault operator init -key-shares=1 -key-threshold=1
   

   This outputs one unseal key and a root token. Save them securely (e.g., in a password manager). Normally, Vault generates multiple keys (e.g., five, with a threshold of three) for added security, but we’re using a single key for simplicity. Do not use a single key in production, as it reduces security.

4. Unseal Vault:

    vault operator unseal <unseal-key>
   

   Use the single unseal key from the init output. Vault is now unsealed and ready.

5. Log in with the root token:

    export VAULT_TOKEN=<root-token>
    vault login
   

Step 4: Set Up MySQL Dynamic Secrets

Vault can generate temporary MySQL credentials that expire after a set time (e.g., 1 hour). Let’s configure this:

# Enable the database secrets engine
vault secrets enable database

# Configure MySQL connection so Vault can communicate with MySQL
vault write database/config/my-mysql \
    plugin_name=mysql-database-plugin \
    connection_url="root:rootpass@tcp(mysql:3306)/" \
    allowed_roles="my-role"

# Create a role that controls how dynamic users are created
vault write database/roles/my-role \
    db_name=my-mysql \
    creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}'; GRANT SELECT ON *.* TO '{{name}}'@'%';" \
    default_ttl="1h" \
    max_ttl="24h"

# Test: generate credentials on the fly!
vault read database/creds/my-role

Key Notes:

• This creates temporary MySQL credentials that expire after 1 hour (default_ttl="1h").

• The credentials grant SELECT permissions on all databases.

• Run vault read database/creds/my-role again to generate new credentials.

Step 5: Access Vault

• Web UI: Open http://localhost:8200 (or http://127.0.0.1:8200 if localhost is blocked). Log in with the root token. Navigate to “Secrets” to view or create secrets.

• CLI: Check Vault status or store a secret:

    vault status
    vault kv put secret/my-app db_password=supersecret
    vault kv get secret/my-app
  

• Secrets persist across container restarts due to the vault-data volume.

Step 6: Stop Vault

To stop the containers:

docker-compose down

On restart, you’ll need to unseal Vault again using the single unseal key.

Best Practices

• Secure Unseal Key and Root Token: Store the unseal key and root token in a secure location (e.g., password manager or HSM). Never store them in plain text or version control. In production, use multiple keys (e.g., three out of five) for better security.

• Use Multiple Unseal Keys: In production, use e.g., 3-of-5 shares.

• Limit Root Token Use: Use the root token only for setup. Create non-root users (see Part 1 experiments) for regular access.

• Backup Storage: The vault-data volume persists data, but back up the storage directory regularly in production.

• Enable TLS: In production, set tls_disable = 0 in vault.hcl and provide TLS certificates.

• Dynamic Secrets Are Short-Lived: Vault-created DB credentials only live for the TTL you set. This reduces risk if credentials leak or are forgotten.

Troubleshooting

• Vault Not Accessible: Ensure containers are running (docker ps) and port 8200 is free (lsof -i :8200). Check firewall settings.

• Unseal Fails: Verify the unseal key is correct. If lost, re-run vault operator init -key-shares=1 -key-threshold=1 (this resets Vault).

• MySQL Connection Error: Confirm MySQL is running (docker logs mysql) and the connection_url matches the MySQL container’s credentials.

What’s Next?

You’ve set up Vault with persistent storage and dynamic MySQL credentials! This setup is closer to production but still runs on a single node. In Part 3, we’ll create a three-node Vault cluster for high availability. Try these experiments:

• Store a secret in the web UI and retrieve it via CLI.

• Generate new MySQL credentials and test them with a MySQL client.

• Explore token time-to-live (TTL) behavior for dynamic secrets.

Share your progress in the comments or join the HashiCorp Community Forum!

Resources

• HashiCorp Vault Documentation: vaultproject.io

• Docker Compose Documentation: docs.docker.com/compose

• HashiCorp Learn: Database Secrets: learn.hashicorp.com/tutorials/vault/database-secrets

Note: For questions or setup help, reach out, comment below or check the Vault documentation.

What is HashiCorp Vault?

Vault securely stores and manages sensitive data, called secrets, and controls access to them. Key features include:

• Secrets storage: Safely store API keys, passwords, or certificates.

• Access control: Define who or what can access secrets using policies.

• Dynamic secrets: Generate temporary credentials (e.g., for databases), which we’ll explore in later parts.

Running Vault with Docker Compose is perfect for beginners, as it simplifies setup and lets you focus on learning Vault’s basics.

Prerequisites

Before starting, ensure you have:

• Docker (version 20.10 or later). Install from Docker’s guide.

• Docker Compose (version 2.0 or later). See Docker Compose installation.

• Vault CLI (optional, for CLI access). Download from releases.hashicorp.com/vault or use a package manager (e.g., brew install vault on macOS).

• A text editor (e.g., VS Code).

• Commands are tested on Linux/macOS; Windows users may need to use set instead of export for environment variables (see below).

⚠️ Important: Dev Mode Warning

Warning: Never use Vault dev mode for production workloads. All secrets are lost on restart, and security is minimal. Dev mode is only for local testing and learning.

Setting Up Vault with Docker Compose

Let’s create a simple Vault setup using Docker Compose. This configuration runs Vault in development mode, which uses in-memory storage and auto-unseals (no manual key entry). It’s ideal for testing but not suitable for production, as secrets are lost on restart.

Step 1: Create the Docker Compose File

Create a file named docker-compose.yml in a new directory (e.g., vault-docker):

services:
  vault:
    image: hashicorp/vault:latest
    container_name: vault
    ports:
      - "8200:8200"
    environment:
      - VAULT_DEV_ROOT_TOKEN_ID=myroot
      - VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    command: vault server -dev

Key Notes:

• VAULT_DEV_ROOT_TOKEN_ID=myroot: Sets a root token for testing. Change it for security.

• cap_add: [IPC_LOCK]: Prevents Vault from swapping sensitive data to disk.

• Port 8200 is exposed for Vault’s web UI and API.

Step 2: Start Vault

1. Open a terminal, navigate to the vault-docker directory, and run:

docker compose up -d

The -d flag runs the container in the background.

2. Verify the container is running:

docker ps

You should see a vault container on port 8200.

Step 3: Access Vault

• Web UI:
  Open http://localhost:8200/ui (or http://127.0.0.1:8200/ui if localhost is blocked) in your browser. Log in with the token myroot. You can create, view, and manage secrets visually.

• CLI:
  You can authenticate to Vault using one of two methods:

  Option 1: Use vault login (recommended for CLI use)

    vault login myroot
  

  This command authenticates you and securely stores your token in a local helper file (usually ~/.vault-token). You won’t need to provide the token again for future CLI commands.

  Option 2: Set the VAULT_TOKEN environment variable (useful for scripts or temporary sessions)

  On Linux/macOS:

    export VAULT_ADDR=http://localhost:8200
    export VAULT_TOKEN=myroot
    vault status
  

  On Windows (Command Prompt):

    set VAULT_ADDR=http://localhost:8200
    set VAULT_TOKEN=myroot
    vault status
  

  This method makes the token available only for your current shell session.

  What’s the difference?

  • vault login is best for daily CLI use: it stores your token securely and you don’t need to set it again.

  • Setting VAULT_TOKEN is great for scripts or temporary sessions, but less secure for long-term use.

• Store a secret:

    vault kv put secret/my-secret password=supersecret
  

• Retrieve it:

    vault kv get secret/my-secret
  

Step 4: Stop Vault

To stop the container:

docker compose down

To remove all containers, networks, and volumes for a clean slate:

docker compose down --volumes

Note: In development mode, secrets are lost when the container stops.

Troubleshooting

• Vault Not Accessible: Ensure the container is running (docker ps) and port 8200 is free (lsof -i :8200). Check your firewall settings.

• Login Fails: Verify the token is myroot and VAULT_ADDR is http://localhost:8200.

• Browser Blocks Localhost: Use http://127.0.0.1:8200/ui.

What’s Next?

You’ve set up a basic Vault instance and stored a secret! This development mode is great for testing, but it’s not secure for production. In Part 2, we’ll add persistent storage, manual unsealing, and a MySQL database for dynamic secrets. Future parts will cover clustering, auto-unseal, and load balancing.

Try these quick experiments:

• Store another secret in the web UI and retrieve it via CLI.

• Explore the Vault UI’s “Policies” tab to see how access control works.

Share your progress in the comments or join the HashiCorp Community Forum!

Resources

• HashiCorp Vault Documentation: vaultproject.io

• Docker Compose Documentation: docs.docker.com/compose

• HashiCorp Learn: Free Vault tutorials (learn.hashicorp.com)

Note: For questions or setup help, reach out, comment below or check the Vault documentation.

Why Use JWT for Vault Authentication?

Using JWT for authentication allows GitLab runners to securely authenticate to Vault without embedding static credentials. The benefits include:

• Short-lived tokens: Reduces risk exposure.

• Automatic token rotation: No need to manually manage credentials.

• Granular access control: Vault policies can define fine-grained permissions based on GitLab project, branch, and environment.

Here’s a step-by-step guide to set up the KV Secrets Engine with GitLab:

What Are Vault Secrets Engines?

Vault Secrets Engines are components of HashiCorp Vault that manage secrets and sensitive data. Each engine is mounted at a specific path in Vault and provides an API to interact with secrets. The KV Secrets Engine is widely used to store static key-value pairs, such as API tokens or passwords, making it ideal for GitLab integration. In this setup, GitLab Pipelines can fetch these secrets securely during runtime.

GitLab’s Integration with the KV Secrets Engine

GitLab Pipelines integrates with HashiCorp Vault to retrieve secrets from the KV Secrets Engine using JSON Web Tokens (JWT) for authentication. Here’s the process:

• Authentication: GitLab generates a JWT for each pipeline job, which Vault verifies using GitLab’s JWKS endpoint (https://gitlab.com/-/jwks). If valid, Vault provides a temporary token.

• Secrets Access: The pipeline uses this token to fetch secrets from the KV Secrets Engine. This ensures secrets remain secure and are never exposed in your repository or CI variables.

Scenario

• GitLab: https://gitlab.example.com (self-hosted)

• Vault: https://vault.example.com

• Goal: Fetch an api_key secret for project ID 42, main branch

Step 1: Setting Up Vault

Vault must be configured to trust GitLab’s JWTs and grant access based on pipeline metadata.

1. Enable JWT Authentication

vault auth enable jwt

2. Configure the JWT Backend

Link Vault to your GitLab instance:

vault write auth/jwt/config \
    jwks_url="https://gitlab.example.com/-/jwks" \
    bound_issuer="https://gitlab.example.com" \
    default_role="project-42-role"

• jwks_url – GitLab’s public key endpoint for JWT verification.

• bound_issuer – Ensures JWTs come from your GitLab instance.

• default_role – Optional fallback role (override in pipeline if needed).

Note: For GitLab.com, use jwks_url="https://gitlab.com/-/jwks" and bound_issuer="https://gitlab.com".

3. Create a Role

Define a role to map GitLab pipeline metadata to Vault policies:

vault write auth/jwt/role/project-42-role \
    role_type="jwt" \
    policies="project-42-policy" \
    token_ttl="5m" \
    token_max_ttl="10m" \
    bound_claims.project_id="42" \
    bound_claims.ref="main" \
    bound_claims.ref_type="branch" \
    bound_claims.ref_protected="true"

bound_claims – Restricts access to project 42, main branch, and protected refs.

4. Create a Policy

Grant read access to specific secrets:

vault policy write project-42-policy - <<EOF
path "secret/data/project-42/*" {
    capabilities = ["read"]
}
EOF

5. Enable Secret Engine and Store a Secret

Enable the KV Secrets Engine in Vault:

vault secrets enable -path=secret kv-v2

version 2 is recommended for features like versioning

Add the secret to Vault:

vault kv put secret/project-42/api api_key="s3cr3t-k3y"

Note: Uses KV v2 (secret/data/) for versioning support.

6. Enable Audit Logging

Track secret access:

vault audit enable file file_path=/var/log/vault_audit.log

Step 2: GitLab Pipeline Configuration

GitLab offers multiple ways to authenticate with Vault using JWTs, depending on your version and needs.

Option 1: Manual Way with CI_JOB_JWT (GitLab 13.4-13.12)

For older GitLab versions, use CI_JOB_JWT manually:

deploy_job:
  image: alpine:latest
  variables:
    VAULT_ADDR: "https://vault.example.com"
    VAULT_CACERT: "/etc/ssl/certs/vault-ca.crt"
  before_script:
    - apk add --no-cache vault
    - if [ -z "$CI_JOB_JWT" ]; then echo "JWT missing!" && exit 1; fi
  script:
    - export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role=project-42-role jwt=$CI_JOB_JWT)
    - export API_KEY=$(vault kv get -field=api_key secret/project-42/api)
    - echo "API Key retrieved successfully"
    - ./deploy.sh --key "$API_KEY"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true"'
  environment:
    name: production

How It Works: CI_JOB_JWT (introduced in 13.4) is sent to Vault to obtain a token, then used to fetch the secret.

• CI_JOB_JWT is a predefined variable in GitLab 13.4+ containing a signed JWT with pipeline metadata (e.g., project_id, ref).

• The script sends it to Vault’s auth/jwt/login endpoint, gets a Vault token, and uses it to fetch the secret.

• No secrets or id_tokens - all logic is manual.

Option 2: Manual Way with id_tokens (GitLab 14.0+)

Use id_tokens for a custom JWT ($VAULT_ID_TOKEN), handling Vault calls manually:

deploy_job:
  image: alpine:latest
  variables:
    VAULT_ADDR: "https://vault.example.com"
    VAULT_CACERT: "/etc/ssl/certs/vault-ca.crt"
  id_tokens:
    VAULT_ID_TOKEN:
      aud: "https://vault.example.com"
  before_script:
    - apk add --no-cache vault
    - if [ -z "$VAULT_ID_TOKEN" ]; then echo "ID token missing!" && exit 1; fi
  script:
    - export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role=project-42-role jwt=$VAULT_ID_TOKEN)
    - export API_KEY=$(vault kv get -field=api_key secret/project-42/api)
    - echo "API Key retrieved successfully"
    - ./deploy.sh --key "$API_KEY"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true"'
  environment:
    name: production

How It Works:

• ID_TOKENS defines VAULT_ID_TOKEN as a JWT with a custom aud claim (https://vault.example.com).

• The script manually sends this JWT to Vault’s auth/jwt/login endpoint to get a token, then fetches the secret.

• No secrets keyword—full control is in the script.

Option 3: Modern Way with secrets (GitLab 14.0+)

The streamlined approach using secrets:

deploy_job:
  image: alpine:latest
  variables:
    VAULT_ADDR: "https://vault.example.com"
    VAULT_CACERT: "/etc/ssl/certs/vault-ca.crt"
  id_tokens:
    VAULT_ID_TOKEN:
      aud: "https://vault.example.com"
  secrets:
    API_KEY:
      vault: secret/project-42/api/api_key
  script:
    - if [ -z "$API_KEY" ]; then echo "Secret not injected!" && exit 1; fi
    - echo "API Key retrieved successfully"
    - ./deploy.sh --key "$API_KEY"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true"'
  environment:
    name: production

How It Works:

• id_tokens generates a JWT tailored for Vault (with a custom aud claim).

• secrets handles authentication and secret injection automatically.

Sidenote: Why id_tokens and secrets Are Preferred Over CI_JOB_JWT

• Security Improvements: id_tokens allow for specifying an audience (aud), reducing the risk of token misuse.

• Flexibility: id_tokens provide a cleaner integration with external authentication providers, allowing better access control.

• Modern Best Practices: The secrets method automates secret injection, reducing the need for manual script handling.

• Deprecation of CI_JOB_JWT: GitLab is phasing out CI_JOB_JWT in favor of id_tokens. Update your pipeline configurations accordingly!

Best Practices

Secure Variables

• Store VAULT_ADDR and VAULT_CACERT as protected, masked GitLab CI/CD variables in Settings > CI/CD > Variables.

• Mark them as protected to limit access to protected branches (e.g., main).

• Use masked variables for sensitive data to prevent accidental logging.

• Always use verified images

Restrict Pipeline Execution

• Use rules with $CI_COMMIT_REF_PROTECTED to limit secrets are only accessed in protected branches or tags.

• Avoid running on untrusted branches (e.g., feature branches) to prevent JWT misuse.

Environment Tracking

• Define an environment for visibility in GitLab’s Environments dashboard.

Error Handling

• Validate JWTs and fail fast if missing or Vault calls fail.

• Suppress errors (2>/dev/null) to avoid leaking sensitive info.

Manual CI_JOB_JWT Best Practices

Minimize Dependencies

• Use a lightweight image (e.g., alpine) and cache vault in a custom runner image:

• Cache dependencies in a custom runner image to speed up pipelines:

FROM alpine:latest
RUN apk add --no-cache vault

Secure JWT handling

• Never log (or echo) $CI_JOB_JWT - it’s sensitive and could be exploited if exposed.

• Pass it directly to Vault without intermediate storage.

• If avoiding the Vault CLI, use curl for portability

export VAULT_TOKEN=$(curl -s --request POST --data "{\"jwt\": \"$CI_JOB_JWT\", \"role\": \"project-42-role\"}" $VAULT_ADDR/v1/auth/jwt/login | jq -r .auth.client_token)

Manual id_tokens (without secrets) Best Practices

Audience Specificity & JWT Validation

• Set aud in id_tokens to match Vault’s expected audience.

• Set the aud in id_tokens to match your Vault server’s URL exactly, ensuring the JWT is purpose-specific.

• Check $VAULT_ID_TOKEN presence in before_script.

Secrets

• Use fully qualified paths (e.g., secret/project-42/api/api_key) in secrets to avoid ambiguity with Vault mounts.

• Only define secrets you need in the job—don’t pull unnecessary ones.

Manual Flexibility

Use for custom logic (e.g., dynamic roles):

export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role="${CI_PROJECT_ID}-role" jwt=$VAULT_ID_TOKEN)

Modern secrets Best Practices

• Use exact secret paths (e.g., secret/project-42/api/api_key).

• Fetch only required secrets per job.

Don't forget also Vault Best Practices

Use Short-Lived Tokens

• Set token_ttl (e.g., 5m) and token_max_ttl (e.g., 10m) to limit exposure.

• Grant minimal capabilities (e.g., read) to specific paths.

• Enable audit logs in Vault to monitor access.

Why These Practices Matter

• Security: Short TTLs, bound claims, and protected variables minimize risks of token or secret exposure.

• Reliability: Error handling and validation prevent silent failures.

• Maintainability: Audit logs and environment tracking simplify debugging and compliance.

Comparison of Methods

Testing and Validation

• Pipeline Logs: Look for “API Key retrieved successfully” and ensure no sensitive data (e.g., $VAULT_TOKEN) leaks.

• Vault Audit: Verify in /var/log/vault_audit.log that only authorized roles access secrets.

• Debugging:

  • Manual methods: Add vault token lookup to inspect token capabilities.

  • Modern method: Check $API_KEY presence.

• Dry Run: Test on a non-protected branch to confirm rules block execution.

Troubleshooting

• "invalid role": Ensure role name matches (project-42-role) and bound_claims align with pipeline metadata.

• "x509: certificate signed by unknown authority": Set VAULT_CACERT or use VAULT_SKIP_VERIFY=true (testing only).

• "Secret not found": Verify path (secret/project-42/api vs. secret/data/project-42/api) and KV version.

Conclusion

Whether you’re stuck on GitLab 13.4 with CI_JOB_JWT, need manual control with id_tokens, or prefer the simplicity of secrets, this guide provides a secure, practical way to integrate GitLab pipelines with Vault. Apply the best practices to minimize risks and streamline your workflow. For self-signed certificates or custom setups, adjust the TLS handling as needed.

Need Expert Help with HashiCorp Vault?

🚀 Struggling with Vault integration, security, or scaling? Whether you're using open-source Vault, Vault Enterprise, or Vault Cloud, I can help optimize your setup, enhance security, and streamline your GitLab CI/CD workflows.

Services I offer:

✅ Secure and scalable Vault deployments ✅ CI/CD pipeline optimization with Vault integration ✅ Vault Enterprise and multi-cloud implementations ✅ Security best practices and compliance

📩 Let’s connect! Reach out on LinkedIn or schedule a free consultation and take your Vault setup to the next level.

Happy deploying! 🚀