Secrets Management: Vault, AWS Secrets Manager, or SOPS?

Introduction

Every application needs secrets—database passwords, API keys, TLS certificates, encryption keys. How you manage these secrets can be the difference between a secure system and a catastrophic data breach.

Hardcoded secrets in code repositories are leaked constantly. Environment variables can be exposed through logs or error messages. Configuration files stored in version control are a security nightmare. Yet teams continue using these anti-patterns because proper secrets management seems complex.

In this comprehensive guide, we'll explore three leading secrets management solutions—HashiCorp Vault, AWS Secrets Manager, and SOPS—helping you choose the right approach for your security requirements. Proper secrets management is a critical component of container security and should be integrated into your CI/CD pipeline.

Why Secrets Management Matters

The Cost of Leaked Secrets

Real incidents:
- Uber: $148M fine (credentials in GitHub)
- Capital One: 100M records (misconfigured IAM)
- Codecov: Supply chain attack (exposed secrets)
- Travis CI: Exposed environment variables

Average cost of data breach: $4.35M (IBM 2023)

Common Anti-Patterns

Hardcoded in Code:

# NEVER do this
AWS_ACCESS_KEY = "AKIAIOSFODNN7EXAMPLE"
AWS_SECRET_KEY = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
DB_PASSWORD = "supersecret123"

Committed to Git:

# This is searchable on GitHub
cat .env
DATABASE_URL=postgresql://user:password@localhost/db
API_KEY=sk_live_51H...
STRIPE_SECRET=whsec_...

Environment Variables Exposed:

# Error messages often dump environment
import os
print(os.environ)  # All secrets exposed in logs

# Container inspect shows env vars
docker inspect <container> | grep -i password

Unencrypted ConfigMaps/Secrets:

# Kubernetes Secrets are only base64 encoded, not encrypted
kubectl get secret db-password -o yaml
# Anyone with cluster access can decode

Solution Comparison Overview

Feature	HashiCorp Vault	AWS Secrets Manager	SOPS
Type	Centralized vault	Managed service	File encryption
Cost	Self-hosted: $0 Enterprise: $$$	$0.40/secret/month $0.05/10K API calls	Free
Complexity	High	Low	Medium
Dynamic Secrets	✅ Yes	⚠️ Limited	❌ No
Secret Rotation	✅ Automatic	✅ Automatic	❌ Manual
Audit Logging	✅ Detailed	✅ CloudTrail	❌ Limited
Multi-Cloud	✅ Yes	❌ AWS only	✅ Yes
GitOps Friendly	⚠️ External	⚠️ External	✅ Yes
Encryption at Rest	✅ Yes	✅ Yes	✅ Yes
Access Control	✅ Fine-grained	✅ IAM-based	⚠️ KMS-based

HashiCorp Vault

Architecture

Application → Vault Agent → Vault Server → Storage Backend
                              ↓
                        (Encrypted)
                              ↓
                      Consul/etcd/S3

Core Concepts

Secrets Engines: Different types of secret storage and generation

# Key-Value secrets (static)
vault kv put secret/database/config \
  username="dbuser" \
  password="supersecret"

# Dynamic secrets (generated on-demand)
vault read database/creds/readonly
# Returns temporary credentials that auto-expire

Authentication Methods: How clients prove identity

# Kubernetes authentication
vault write auth/kubernetes/role/myapp \
  bound_service_account_names=myapp \
  bound_service_account_namespaces=production \
  policies=myapp-policy \
  ttl=1h

Policies: Fine-grained access control

# myapp-policy.hcl
path "secret/data/database/config" {
  capabilities = ["read"]
}

path "database/creds/readonly" {
  capabilities = ["read"]
}

path "secret/data/api-keys/*" {
  capabilities = ["read", "list"]
}

Installation and Setup

# Vault on Kubernetes with Helm
helm repo add hashicorp https://helm.releases.hashicorp.com

helm install vault hashicorp/vault \
  --set server.ha.enabled=true \
  --set server.ha.replicas=3 \
  --set ui.enabled=true \
  --set server.dataStorage.size=10Gi

# Initialize Vault
kubectl exec vault-0 -- vault operator init
# Save unseal keys and root token securely!

# Unseal Vault (repeat on all pods)
kubectl exec vault-0 -- vault operator unseal <unseal-key-1>
kubectl exec vault-0 -- vault operator unseal <unseal-key-2>
kubectl exec vault-0 -- vault operator unseal <unseal-key-3>

Using Vault with Applications

Method 1: Vault Agent Sidecar

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "myapp"
        vault.hashicorp.com/agent-inject-secret-database: "secret/data/database/config"
        vault.hashicorp.com/agent-inject-template-database: |
          {{- with secret "secret/data/database/config" -}}
          DATABASE_URL=postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/mydb
          {{- end }}
    spec:
      serviceAccountName: myapp
      containers:
      - name: app
        image: myapp:v1.0
        # Secret automatically written to /vault/secrets/database

Method 2: Vault SDK in Application

import hvac

# Authenticate with Kubernetes
client = hvac.Client(url='http://vault:8200')

with open('/var/run/secrets/kubernetes.io/serviceaccount/token') as f:
    jwt = f.read()

client.auth.kubernetes.login(
    role='myapp',
    jwt=jwt
)

# Read secrets
secret = client.secrets.kv.v2.read_secret_version(
    path='database/config'
)

db_user = secret['data']['data']['username']
db_pass = secret['data']['data']['password']

Dynamic Secrets

Vault generates short-lived credentials on-demand:

# Configure database secrets engine
vault secrets enable database

vault write database/config/postgresql \
  plugin_name=postgresql-database-plugin \
  allowed_roles="readonly" \
  connection_url="postgresql://{{username}}:{{password}}@postgres:5432/mydb" \
  username="vault" \
  password="vaultpass"

# Create role for read-only access
vault write database/roles/readonly \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Application requests credentials
vault read database/creds/readonly
# Returns:
# Key                Value
# lease_id           database/creds/readonly/abc123
# lease_duration     1h
# username           v-readonly-abc123
# password           A1a-generated-password

# Credentials automatically revoked after 1h

Secret Rotation

# Automatic rotation for supported systems
vault write -f database/rotate-root/postgresql
# Vault rotates its own database credentials

# For static secrets, create rotation policy
vault write sys/rotate-root/config \
  rotation_period="720h"  # 30 days

Strengths

Dynamic Secrets: Generate short-lived credentials

Secret Rotation: Automatic credential rotation

Fine-Grained Access: Policies per path/operation

Encryption as a Service: Use Vault to encrypt/decrypt data

# Encrypt data without storing it
vault write transit/encrypt/orders \
  plaintext=$(echo "sensitive data" | base64)

vault write transit/decrypt/orders \
  ciphertext="vault:v1:..."

Multi-Cloud: Works anywhere

Audit Logging: Detailed audit trail

Weaknesses

Operational Complexity: Requires HA setup, unsealing, backups

Learning Curve: Many concepts to understand

Unsealing Requirement: Manual unsealing after restarts

# After pod restart, must unseal
kubectl exec vault-0 -- vault status
# Sealed: true

# Must provide unseal keys
kubectl exec vault-0 -- vault operator unseal <key-1>
kubectl exec vault-0 -- vault operator unseal <key-2>
kubectl exec vault-0 -- vault operator unseal <key-3>

Cost: Enterprise features (DR, namespaces) require license

When to Use Vault

✓ Multi-cloud or hybrid infrastructure ✓ Need dynamic secrets ✓ Require automatic secret rotation ✓ Compliance requirements (detailed audit logs) ✓ Have dedicated platform/security team ✓ Large number of secrets (>100) ✓ Need encryption as a service

Cost

Open Source (self-hosted):
- Infrastructure: $200-500/month (HA cluster)
- Operations: 0.25 FTE = $3,000-5,000/month
Total: $3,200-5,500/month

Enterprise:
- License: $15,000-50,000/year
- Infrastructure: $200-500/month
- Operations: 0.25 FTE
Total: $4,500-9,500/month

AWS Secrets Manager

Architecture

Application → AWS SDK → Secrets Manager → KMS
                              ↓
                        (Encrypted storage)

Creating Secrets

# Create secret
aws secretsmanager create-secret \
  --name production/database/password \
  --description "Production database password" \
  --secret-string "supersecret123"

# Create with JSON structure
aws secretsmanager create-secret \
  --name production/database/config \
  --secret-string '{
    "username": "dbuser",
    "password": "supersecret123",
    "host": "db.example.com",
    "port": 5432
  }'

Using Secrets in Applications

import boto3
import json

def get_secret():
    client = boto3.client('secretsmanager', region_name='us-east-1')
    
    response = client.get_secret_value(
        SecretId='production/database/config'
    )
    
    secret = json.loads(response['SecretString'])
    return secret

# Use in application
secret = get_secret()
db_url = f"postgresql://{secret['username']}:{secret['password']}@{secret['host']}:{secret['port']}/mydb"

Kubernetes Integration

External Secrets Operator:

# Install External Secrets Operator
helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets-system \
  --create-namespace

# Create SecretStore
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

---
# Create ExternalSecret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-config
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: database-config
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: production/database/config
      property: username
  - secretKey: password
    remoteRef:
      key: production/database/config
      property: password

Automatic Rotation

# Lambda function for rotation
import boto3
import pymysql

def lambda_handler(event, context):
    service_client = boto3.client('secretsmanager')
    
    # Get current secret
    current_secret = service_client.get_secret_value(
        SecretId=event['SecretId']
    )
    
    # Generate new password
    new_password = generate_random_password()
    
    # Update database
    connection = pymysql.connect(
        host=current_secret['host'],
        user=current_secret['username'],
        password=current_secret['password']
    )
    
    with connection.cursor() as cursor:
        cursor.execute(
            f"ALTER USER '{current_secret['username']}' IDENTIFIED BY '{new_password}'"
        )
    connection.commit()
    
    # Update secret
    service_client.put_secret_value(
        SecretId=event['SecretId'],
        SecretString=json.dumps({
            'username': current_secret['username'],
            'password': new_password
        })
    )

# Enable automatic rotation
aws secretsmanager rotate-secret \
  --secret-id production/database/password \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:rotate-secret \
  --rotation-rules AutomaticallyAfterDays=30

Cross-Region Replication

aws secretsmanager replicate-secret-to-regions \
  --secret-id production/database/password \
  --add-replica-regions Region=eu-west-1 \
  --add-replica-regions Region=ap-southeast-1

Strengths

Fully Managed: Zero operational overhead

AWS Integration: Native IAM, CloudTrail, VPC endpoints

Automatic Rotation: Built-in rotation for RDS, Redshift, DocumentDB

Cross-Region Replication: Automatic failover

Compliance: SOC, PCI, HIPAA certified

Weaknesses

AWS Only: Can't use with other clouds

Cost: Expensive at scale ($0.40/secret/month + API calls)

No Dynamic Secrets: Can't generate temporary credentials

Limited Rotation: Only supports specific AWS services

When to Use Secrets Manager

✓ AWS-only infrastructure ✓ Want zero operational overhead ✓ Need automatic rotation for RDS/Redshift ✓ Require cross-region replication ✓ Small to medium number of secrets (<1000) ✓ Compliance requirements (AWS certified)

Cost

100 secrets, 1M API calls/month:

- Secrets: 100 × $0.40 = $40/month
- API calls: 1M × $0.05/10K = $5/month
Total: $45/month

1,000 secrets, 10M API calls/month:
- Secrets: 1,000 × $0.40 = $400/month
- API calls: 10M × $0.05/10K = $50/month
Total: $450/month

SOPS (Secrets OPerationS)

Architecture

Developer → SOPS → KMS/PGP → Encrypted File → Git
                ↓
         (Encrypt/Decrypt)
                ↓
           Application

Core Concept

SOPS encrypts files while keeping structure readable:

# Original secrets.yaml
api:
  key: sk_live_51H...
  secret: whsec_...
database:
  password: supersecret123
  host: db.example.com

# Encrypted with SOPS
api:
  key: ENC[AES256_GCM,data:abc123...,iv:xyz...,tag:def...]
  secret: ENC[AES256_GCM,data:uvw456...,iv:rst...,tag:ghi...]
database:
  password: ENC[AES256_GCM,data:mno789...,iv:jkl...,tag:pqr...]
  host: db.example.com  # Not encrypted (no sensitive data)
sops:
  kms:
  - arn: arn:aws:kms:us-east-1:123456789:key/abc-123
    created_at: "2024-01-15T10:00:00Z"
  pgp:
  - fingerprint: ABC123...

Installation and Configuration

# Install SOPS
brew install sops

# Or download binary
curl -LO https://github.com/mozilla/sops/releases/download/v3.8.1/sops-v3.8.1.linux.amd64
chmod +x sops-v3.8.1.linux.amd64
sudo mv sops-v3.8.1.linux.amd64 /usr/local/bin/sops

Configuration (.sops.yaml):

creation_rules:
  # Production secrets (AWS KMS)
  - path_regex: production/.*\.yaml$
    kms: arn:aws:kms:us-east-1:123456789:key/production-key
    encrypted_regex: ^(data|stringData|password|secret|key)$
  
  # Staging secrets (different KMS key)
  - path_regex: staging/.*\.yaml$
    kms: arn:aws:kms:us-east-1:123456789:key/staging-key
    encrypted_regex: ^(data|stringData|password|secret|key)$
  
  # Development (PGP)
  - path_regex: development/.*\.yaml$
    pgp: >-
      ABC123DEF456,
      GHI789JKL012
    encrypted_regex: ^(data|stringData|password|secret|key)$

Using SOPS

# Encrypt file
sops --encrypt secrets.yaml > secrets.enc.yaml

# Edit encrypted file (decrypts, opens editor, re-encrypts on save)
sops secrets.enc.yaml

# Decrypt file
sops --decrypt secrets.enc.yaml

# Decrypt and pipe to kubectl
sops --decrypt secrets.enc.yaml | kubectl apply -f -

GitOps with SOPS and Flux

# Flux Kustomization with SOPS decryption
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  # Decrypt SOPS-encrypted files
  decryption:
    provider: sops
    secretRef:
      name: sops-kms
---
# KMS credentials for decryption
apiVersion: v1
kind: Secret
metadata:
  name: sops-kms
  namespace: flux-system
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: AKIAIOSFODNN7EXAMPLE
  AWS_SECRET_ACCESS_KEY: wJalrXUtn...

ArgoCD with SOPS

# Install SOPS plugin
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  kustomize.buildOptions: --enable-alpha-plugins --enable-helm
  configManagementPlugins: |
    - name: sops
      generate:
        command: ["sh", "-c"]
        args: ["sops -d secrets.yaml | kubectl apply -f -"]

Strengths

GitOps Friendly: Secrets version-controlled alongside code

Simple: Just a binary, no infrastructure

Multi-Cloud: Works with AWS KMS, GCP KMS, Azure Key Vault, PGP

Free: Open source, no licensing costs

Selective Encryption: Encrypt only sensitive fields

Auditable: Git history shows who changed what

Weaknesses

No UI: Command-line only

No Dynamic Secrets: Static secrets only

No Automatic Rotation: Manual rotation required

Key Management: Must manage KMS keys/PGP keys

Limited Audit: Only Git history, no detailed access logs

When to Use SOPS

✓ GitOps workflow ✓ Small team ✓ Want secrets in version control (encrypted) ✓ Budget-conscious (free) ✓ Simple use case (static secrets) ✓ Multi-cloud (using different KMS per cloud) ✓ Don't need dynamic secrets or rotation

Cost

SOPS:
- Software: Free
- KMS usage: ~$1/month (per key)
- Operations: Minimal
Total: ~$1-5/month

Detailed Comparison

Security Posture

Vault:
- Encryption at rest ✓
- Encryption in transit ✓
- Detailed audit logs ✓
- Fine-grained access ✓
- Secret rotation ✓
- Dynamic secrets ✓
Score: 10/10

Secrets Manager:
- Encryption at rest ✓
- Encryption in transit ✓
- CloudTrail audit ✓
- IAM access control ✓
- Automatic rotation ✓ (limited)
- Dynamic secrets ✗
Score: 8/10

SOPS:
- Encryption at rest ✓
- Encryption in transit ⚠️ (Git over HTTPS)
- Git audit logs ⚠️
- KMS access control ✓
- Secret rotation ✗
- Dynamic secrets ✗
Score: 5/10

Operational Overhead

Vault: HIGH
- Setup HA cluster
- Configure storage backend
- Implement unsealing strategy
- Backup and recovery
- Monitoring and alerting
- Regular updates
Time: 40 hours initial + 20 hours/month

Secrets Manager: NONE
- Fully managed
- No infrastructure
- Automatic updates
Time: 2 hours initial + 1 hour/month

SOPS: LOW
- Install binary
- Configure .sops.yaml
- Manage KMS keys
Time: 4 hours initial + 2 hours/month

Hybrid Approaches

SOPS + Vault

Use SOPS for GitOps, Vault for dynamic secrets:

# SOPS-encrypted Kubernetes Secret
apiVersion: v1
kind: Secret
metadata:
  name: vault-config
type: Opaque
stringData:
  vault-token: ENC[AES256_GCM,data:abc123...]
  vault-addr: https://vault.example.com

# Application uses Vault for dynamic DB credentials
import hvac

# Vault token from SOPS-encrypted Kubernetes Secret
client = hvac.Client(
    url=os.getenv('VAULT_ADDR'),
    token=os.getenv('VAULT_TOKEN')
)

# Get dynamic database credentials
db_creds = client.secrets.database.generate_credentials(
    name='readonly'
)

Secrets Manager + Parameter Store

Use Secrets Manager for sensitive secrets, Parameter Store for config:

# Sensitive: Secrets Manager
aws secretsmanager create-secret \
  --name production/database/password \
  --secret-string "supersecret"

# Non-sensitive: Parameter Store (cheaper)
aws ssm put-parameter \
  --name /production/database/host \
  --value "db.example.com" \
  --type String

Best Practices

Principle of Least Privilege

# Vault: Minimal policy
path "secret/data/myapp/*" {
  capabilities = ["read"]
}

path "database/creds/readonly" {
  capabilities = ["read"]
}

# Deny everything else (implicit)

// AWS IAM: Minimal policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789:secret:production/myapp/*"
    }
  ]
}

Secret Rotation

Rotation schedule:
- API keys: 90 days
- Passwords: 30-60 days
- Certificates: 90 days (Let's Encrypt)
- Encryption keys: Yearly

Audit and Monitor

# Vault: Enable audit logging
vault audit enable file file_path=/var/log/vault/audit.log

# AWS: Enable CloudTrail for Secrets Manager
aws cloudtrail create-trail \
  --name secrets-audit \
  --s3-bucket-name audit-logs

# Alert on secret access
aws cloudwatch put-metric-alarm \
  --alarm-name high-secret-access \
  --metric-name GetSecretValue \
  --threshold 1000 \
  --comparison-operator GreaterThanThreshold

Never Log Secrets

# Bad
logger.info(f"Connecting with password: {password}")

# Good
logger.info("Connecting to database")
# Don't log secrets

Choosing the Right Solution

Decision Tree

Need dynamic secrets or automatic rotation?
├─ Yes → Vault or Secrets Manager
│  ├─ AWS-only?
│  │  ├─ Yes → Secrets Manager
│  │  └─ No → Vault
│  └─ Have ops team?
│     ├─ Yes → Vault
│     └─ No → Secrets Manager
└─ No → SOPS or Secrets Manager
   ├─ Using GitOps?
   │  ├─ Yes → SOPS
   │  └─ No → Secrets Manager
   └─ Budget?
      ├─ Limited → SOPS
      └─ Flexible → Secrets Manager

Recommendations by Team Size

Small Team (<10 engineers): → SOPS or Secrets Manager

• Simple to use
• Low/no operations
• Cost-effective

Medium Team (10-50 engineers): → Secrets Manager or Vault

• Secrets Manager if AWS-only
• Vault if multi-cloud
• Need audit and compliance

Large Team (>50 engineers): → Vault

• Dynamic secrets essential
• Fine-grained access control
• Dedicated platform team

Conclusion

There's no one-size-fits-all secrets management solution:

HashiCorp Vault: Most powerful, but requires operational expertise. Choose when you need dynamic secrets, automatic rotation, and have a platform team.

AWS Secrets Manager: Fully managed, AWS-native. Choose when you're AWS-only and want zero operational overhead.

SOPS: Simple file encryption. Choose when you use GitOps, have a small team, and don't need dynamic secrets.

Remember: Any secrets management solution is better than hardcoded secrets. Start with the simplest solution that meets your security requirements, then evolve as your needs grow.

Related Articles

• Container Security Best Practices: A Complete Guide from Dev to Production
• CI/CD Best Practices for Modern DevOps Teams