# GitLab Pipeline JWT Authentication with HashiCorp Vault: A Comprehensive Guide

This guide explores how to integrate GitLab CI/CD pipelines with HashiCorp Vault using JWT (JSON Web Token) authentication to securely fetch secrets. We’ll cover three approaches—manual `CI_JOB_JWT` (older GitLab versions), manual `id_tokens` without secrets, and the modern `secrets` method—along with Vault setup, best practices, and testing tips. Whether you’re on GitLab 13.4 or the latest version, this guide has you covered.

### **Why Use JWT for Vault Authentication?**

Using JWT for authentication allows GitLab runners to securely authenticate to Vault without embedding static credentials. The benefits include:

* **Short-lived tokens:** Reduces risk exposure.
    
* **Automatic token rotation:** No need to manually manage credentials.
    
* **Granular access control:** Vault policies can define fine-grained permissions based on GitLab project, branch, and environment.
    

---

### **Here’s a step-by-step guide to set up the KV Secrets Engine with GitLab:**

#### **What Are Vault Secrets Engines?**

Vault Secrets Engines are components of HashiCorp Vault that manage secrets and sensitive data. Each engine is mounted at a specific path in Vault and provides an API to interact with secrets. The KV Secrets Engine is widely used to store static key-value pairs, such as API tokens or passwords, making it ideal for GitLab integration. In this setup, GitLab Pipelines can fetch these secrets securely during runtime.

#### **GitLab’s Integration with the KV Secrets Engine**

GitLab Pipelines integrates with HashiCorp Vault to retrieve secrets from the KV Secrets Engine using JSON Web Tokens (JWT) for authentication. Here’s the process:

* **Authentication:** GitLab generates a JWT for each pipeline job, which Vault verifies using GitLab’s JWKS endpoint ([https://gitlab.com/-/jwks](https://gitlab.com/-/jwks)). If valid, Vault provides a temporary token.
    
* **Secrets Access:** The pipeline uses this token to fetch secrets from the KV Secrets Engine. This ensures secrets remain secure and are never exposed in your repository or CI variables.
    

---

### **Scenario**

* **GitLab:** `https://gitlab.example.com` (self-hosted)
    
* **Vault:** `https://vault.example.com`
    
* **Goal:** Fetch an `api_key` secret for project ID `42`, `main` branch
    

---

### **Step 1: Setting Up Vault**

Vault must be configured to trust GitLab’s JWTs and grant access based on pipeline metadata.

#### **1\. Enable JWT Authentication**

```bash
vault auth enable jwt
```

#### **2\. Configure the JWT Backend**

Link Vault to your GitLab instance:

```bash
vault write auth/jwt/config \
    jwks_url="https://gitlab.example.com/-/jwks" \
    bound_issuer="https://gitlab.example.com" \
    default_role="project-42-role"
```

* `jwks_url` – GitLab’s public key endpoint for JWT verification.
    
* `bound_issuer` – Ensures JWTs come from your GitLab instance.
    
* `default_role` – Optional fallback role (override in pipeline if needed).
    

> **Note:** For GitLab.com, use `jwks_url="https://gitlab.com/-/jwks"` and `bound_issuer="https://gitlab.com"`.

#### **3\. Create a Role**

Define a role to map GitLab pipeline metadata to Vault policies:

```bash
vault write auth/jwt/role/project-42-role \
    role_type="jwt" \
    policies="project-42-policy" \
    token_ttl="5m" \
    token_max_ttl="10m" \
    bound_claims.project_id="42" \
    bound_claims.ref="main" \
    bound_claims.ref_type="branch" \
    bound_claims.ref_protected="true"
```

> `bound_claims` – Restricts access to project `42`, `main` branch, and protected refs.

#### **4\. Create a Policy**

Grant read access to specific secrets:

```bash
vault policy write project-42-policy - <<EOF
path "secret/data/project-42/*" {
    capabilities = ["read"]
}
EOF
```

#### **5\. Enable Secret Engine and Store a Secret**

Enable the KV Secrets Engine in Vault:

```bash
vault secrets enable -path=secret kv-v2
```

> version 2 is recommended for features like versioning

Add the secret to Vault:

```bash
vault kv put secret/project-42/api api_key="s3cr3t-k3y"
```

> **Note:** Uses KV v2 (`secret/data/`) for versioning support.

#### **6\. Enable Audit Logging**

Track secret access:

```bash
vault audit enable file file_path=/var/log/vault_audit.log
```

---

## **Step 2: GitLab Pipeline Configuration**

GitLab offers multiple ways to authenticate with Vault using JWTs, depending on your version and needs.

### **Option 1: Manual Way with** `CI_JOB_JWT` (GitLab 13.4-13.12)

For older GitLab versions, use CI\_JOB\_JWT manually:

```yaml
deploy_job:
  image: alpine:latest
  variables:
    VAULT_ADDR: "https://vault.example.com"
    VAULT_CACERT: "/etc/ssl/certs/vault-ca.crt"
  before_script:
    - apk add --no-cache vault
    - if [ -z "$CI_JOB_JWT" ]; then echo "JWT missing!" && exit 1; fi
  script:
    - export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role=project-42-role jwt=$CI_JOB_JWT)
    - export API_KEY=$(vault kv get -field=api_key secret/project-42/api)
    - echo "API Key retrieved successfully"
    - ./deploy.sh --key "$API_KEY"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true"'
  environment:
    name: production
```

> **How It Works:** *CI\_JOB\_JWT* (introduced in 13.4) is sent to Vault to obtain a token, then used to fetch the secret.
> 
> * CI\_JOB\_JWT is a predefined variable in GitLab 13.4+ containing a signed JWT with pipeline metadata (e.g., project\_id, ref).
>     
> * The script sends it to Vault’s auth/jwt/login endpoint, gets a Vault token, and uses it to fetch the secret.
>     
> * No secrets or id\_tokens - all logic is manual.
>     

### **Option 2: Manual Way with** `id_tokens` (GitLab 14.0+)

Use id\_tokens for a custom JWT ($VAULT\_ID\_TOKEN), handling Vault calls manually:

```yaml
deploy_job:
  image: alpine:latest
  variables:
    VAULT_ADDR: "https://vault.example.com"
    VAULT_CACERT: "/etc/ssl/certs/vault-ca.crt"
  id_tokens:
    VAULT_ID_TOKEN:
      aud: "https://vault.example.com"
  before_script:
    - apk add --no-cache vault
    - if [ -z "$VAULT_ID_TOKEN" ]; then echo "ID token missing!" && exit 1; fi
  script:
    - export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role=project-42-role jwt=$VAULT_ID_TOKEN)
    - export API_KEY=$(vault kv get -field=api_key secret/project-42/api)
    - echo "API Key retrieved successfully"
    - ./deploy.sh --key "$API_KEY"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true"'
  environment:
    name: production
```

> **How It Works:**

* **ID\_TOKENS** defines *VAULT\_ID\_TOKEN* as a JWT with a custom aud claim ([https://vault.example.com](https://vault.example.com/)).
    
* The script manually sends this JWT to Vault’s auth/jwt/login endpoint to get a token, then fetches the secret.
    
* No secrets keyword—full control is in the script.
    

### **Option 3: Modern Way with** `secrets` (GitLab 14.0+)

The streamlined approach using secrets:

```yaml
deploy_job:
  image: alpine:latest
  variables:
    VAULT_ADDR: "https://vault.example.com"
    VAULT_CACERT: "/etc/ssl/certs/vault-ca.crt"
  id_tokens:
    VAULT_ID_TOKEN:
      aud: "https://vault.example.com"
  secrets:
    API_KEY:
      vault: secret/project-42/api/api_key
  script:
    - if [ -z "$API_KEY" ]; then echo "Secret not injected!" && exit 1; fi
    - echo "API Key retrieved successfully"
    - ./deploy.sh --key "$API_KEY"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main" && $CI_COMMIT_REF_PROTECTED == "true"'
  environment:
    name: production
```

> **How It Works:**

* **id\_tokens** generates a JWT tailored for Vault (with a custom aud claim).
    
* **secrets** handles authentication and secret injection automatically.
    

#### **Sidenote: Why id\_tokens and secrets Are Preferred Over CI\_JOB\_JWT**

* **Security Improvements:** id\_tokens allow for specifying an audience (aud), reducing the risk of token misuse.
    
* **Flexibility:** id\_tokens provide a cleaner integration with external authentication providers, allowing better access control.
    
* **Modern Best Practices:** The secrets method automates secret injection, reducing the need for manual script handling.
    
* **Deprecation of CI\_JOB\_JWT:** GitLab is phasing out CI\_JOB\_JWT in favor of id\_tokens. Update your pipeline configurations accordingly!
    

---

### **Best Practices**

#### **Secure Variables**

* Store *VAULT\_ADDR* and *VAULT\_CACERT* as protected, masked GitLab CI/CD variables in Settings &gt; CI/CD &gt; Variables.
    
* Mark them as protected to limit access to protected branches (e.g., main).
    
* Use masked variables for sensitive data to prevent accidental logging.
    
* Always use verified images
    

#### **Restrict Pipeline Execution**

* Use rules with *$CI\_COMMIT\_REF\_PROTECTED* to limit secrets are only accessed in protected branches or tags.
    
* Avoid running on untrusted branches (e.g., feature branches) to prevent JWT misuse.
    

#### **Environment Tracking**

* Define an environment for visibility in GitLab’s Environments dashboard.
    

#### **Error Handling**

* Validate JWTs and fail fast if missing or Vault calls fail.
    
* Suppress errors (2&gt;/dev/null) to avoid leaking sensitive info.
    

### **Manual *CI\_JOB\_JWT* Best Practices**

#### **Minimize Dependencies**

* Use a lightweight image (e.g., alpine) and cache vault in a custom runner image:
    
* Cache dependencies in a custom runner image to speed up pipelines:
    

```dockerfile

FROM alpine:latest
RUN apk add --no-cache vault
```

#### **Secure JWT handling**

* Never log (or echo) *$CI\_JOB\_JWT* - it’s sensitive and could be exploited if exposed.
    
* Pass it directly to Vault without intermediate storage.
    
* If avoiding the Vault CLI, use curl for portability
    

```bash
export VAULT_TOKEN=$(curl -s --request POST --data "{\"jwt\": \"$CI_JOB_JWT\", \"role\": \"project-42-role\"}" $VAULT_ADDR/v1/auth/jwt/login | jq -r .auth.client_token)
```

### **Manual *id\_tokens* (without *secrets*) Best Practices**

#### **Audience Specificity & JWT Validation**

* Set aud in id\_tokens to match Vault’s expected audience.
    
* Set the aud in id\_tokens to match your Vault server’s URL exactly, ensuring the JWT is purpose-specific.
    
* Check *$VAULT\_ID\_TOKEN* presence in before\_script.
    

#### **Secrets**

* Use fully qualified paths (e.g., *secret/project-42/api/api\_key*) in secrets to avoid ambiguity with Vault mounts.
    
* Only define secrets you need in the job—don’t pull unnecessary ones.
    

#### **Manual Flexibility**

Use for custom logic (e.g., dynamic roles):

```bash

export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role="${CI_PROJECT_ID}-role" jwt=$VAULT_ID_TOKEN)
```

### **Modern secrets Best Practices**

* Use exact secret paths (e.g., *secret/project-42/api/api\_key*).
    
* Fetch only required secrets per job.
    

### **Don't forget also Vault Best Practices**

#### **Use Short-Lived Tokens**

* Set token\_ttl (e.g., 5m) and token\_max\_ttl (e.g., 10m) to limit exposure.
    
* Grant minimal capabilities (e.g., read) to specific paths.
    
* Enable audit logs in Vault to monitor access.
    

### **Why These Practices Matter**

* **Security:** Short TTLs, bound claims, and protected variables minimize risks of token or secret exposure.
    
* **Reliability:** Error handling and validation prevent silent failures.
    
* **Maintainability:** Audit logs and environment tracking simplify debugging and compliance.
    

---

## **Comparison of Methods**

| **Method** | **GitLab Version** | **JWT Variable** | **Pros** | **Cons** | **Use case** |
| --- | --- | --- | --- | --- | --- |
| Manual `CI_JOB_JWT` | 13.4-13.12 | `CI_JOB_JWT` | Works on older versions | No `aud` customization | Legacy GitLab instances |
| Manual `id_tokens` | 14.0+ | `VAULT_ID_TOKEN` | Custom `aud`, flexible logic | Requires manual scripting | Custom authentication setups |
| Modern `secrets` | 14.0+ | `VAULT_ID_TOKEN` | Simple, automatic | Less control over process | Best for most CI/CD workflows |

---

## **Testing and Validation**

* **Pipeline Logs:** Look for “API Key retrieved successfully” and ensure no sensitive data (e.g., $VAULT\_TOKEN) leaks.
    
* **Vault Audit:** Verify in /var/log/vault\_audit.log that only authorized roles access secrets.
    
* **Debugging:**
    
    * **Manual methods:** Add vault token lookup to inspect token capabilities.
        
    * **Modern method:** Check $API\_KEY presence.
        
* **Dry Run:** Test on a non-protected branch to confirm rules block execution.
    

---

## **Troubleshooting**

* **"invalid role":** Ensure role name matches (*project-42-role*) and bound\_claims align with pipeline metadata.
    
* **"x509: certificate signed by unknown authority":** Set *VAULT\_CACERT* or use *VAULT\_SKIP\_VERIFY=true* (testing only).
    
* **"Secret not found"**: Verify path (*secret/project-42/api* vs. *secret/data/project-42/api*) and KV version.
    

---

## **Conclusion**

Whether you’re stuck on GitLab 13.4 with CI\_JOB\_JWT, need manual control with id\_tokens, or prefer the simplicity of secrets, this guide provides a secure, practical way to integrate GitLab pipelines with Vault. Apply the best practices to minimize risks and streamline your workflow. For self-signed certificates or custom setups, adjust the TLS handling as needed.

---

## **Need Expert Help with HashiCorp Vault?**

🚀 Struggling with **Vault integration, security, or scaling**? Whether you're using **open-source Vault, Vault Enterprise, or Vault Cloud**, I can help **optimize your setup, enhance security, and streamline your GitLab CI/CD workflows.**

**Services I offer:**

✅ Secure and scalable Vault deployments ✅ CI/CD pipeline optimization with Vault integration ✅ Vault Enterprise and multi-cloud implementations ✅ Security best practices and compliance

📩 **Let’s connect!** Reach out on [**LinkedIn**](https://www.linkedin.com/in/miroslavmilak/) or schedule a [**free consultation**](https://calendly.com/myros/vault-consultation) and take your Vault setup to the next level.

---

Happy deploying! 🚀