Sealed Secrets

Purpose: Encrypt secrets for safe storage in Git repositories

Version: 2.18.4

Namespace: kube-system

Description

Sealed Secrets provides one-way encryption of Kubernetes Secrets, allowing them to be safely stored in Git repositories. Only the controller running in the cluster can decrypt them.

This enables true GitOps workflows where secrets can be version-controlled without exposing sensitive data.

How It Works

Developer creates a standard Kubernetes Secret
kubeseal CLI encrypts it into a SealedSecret custom resource
SealedSecret is committed to Git (safe, only controller can decrypt)
ArgoCD syncs the SealedSecret to the cluster
Controller decrypts and creates the actual Kubernetes Secret

Installation

Installed via ArgoCD from Helm chart: bitnami/sealed-secrets

Configuration

Key settings:

Installed in kube-system namespace (not a dedicated namespace)
Default key rotation: 30 days
Private key backed up for disaster recovery

Using Sealed Secrets

Step 1: Install kubeseal CLI

# macOS
brew install kubeseal

# Linux
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.28.0/kubeseal-0.28.0-linux-amd64.tar.gz
tar -xzf kubeseal-0.28.0-linux-amd64.tar.gz
sudo mv kubeseal /usr/local/bin/

# Verify installation
kubeseal --version

Step 2: Get the Public Certificate

# Download the public cert from the cluster
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d > sealed-secrets-cert.pem

Step 3: Create a Secret

Create a standard Kubernetes Secret YAML:

# my-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: default
type: Opaque
stringData:
  username: admin
  password: supersecret123
  api-key: my-api-key-12345

Step 4: Encrypt the Secret

# Encrypt using the public cert
kubeseal --cert sealed-secrets-cert.pem \
  --format yaml \
  < my-secret.yaml \
  > my-sealedsecret.yaml

# Or encrypt using the cluster directly (requires kubectl access)
kubeseal --format yaml \
  --controller-namespace kube-system \
  < my-secret.yaml \
  > my-sealedsecret.yaml

Step 5: Commit to Git

The resulting my-sealedsecret.yaml looks like:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: my-secret
  namespace: default
spec:
  encryptedData:
    username: AgBxyz...
    password: AgC123...
    api-key: AgD456...
  template:
    metadata:
      name: my-secret
      namespace: default
    type: Opaque

This file is safe to commit to Git! Only the controller in your cluster can decrypt it.

Step 6: Deploy via ArgoCD

Add the SealedSecret to your application's Helm chart:

# charts/my-app/templates/sealed-secret.yaml
{{- if .Values.sealedSecret.enabled }}
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: {{ include "my-app.fullname" . }}
  namespace: {{ .Release.Namespace }}
spec:
  encryptedData:
    {{- toYaml .Values.sealedSecret.encryptedData | nindent 4 }}
  template:
    metadata:
      name: {{ include "my-app.fullname" . }}
      namespace: {{ .Release.Namespace }}
    type: Opaque
{{- end }}

Then reference it in your deployment:

# charts/my-app/templates/deployment.yaml
env:
  - name: USERNAME
    valueFrom:
      secretKeyRef:
        name: {{ include "my-app.fullname" . }}
        key: username
  - name: PASSWORD
    valueFrom:
      secretKeyRef:
        name: {{ include "my-app.fullname" . }}
        key: password

Secret Types

Opaque Secrets

Generic key-value secrets:

kubectl create secret generic my-secret \
  --from-literal=username=admin \
  --from-literal=password=supersecret123 \
  --dry-run=client -o yaml | kubeseal --format yaml > sealed.yaml

TLS Secrets

For certificates and private keys:

kubectl create secret tls my-tls-secret \
  --cert=path/to/tls.crt \
  --key=path/to/tls.key \
  --dry-run=client -o yaml | kubeseal --format yaml > sealed-tls.yaml

Docker Registry Secrets

For private container registries:

kubectl create secret docker-registry regcred \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=myuser \
  --docker-password=mypassword \
  --dry-run=client -o yaml | kubeseal --format yaml > sealed-reg.yaml

Scopes

Sealed Secrets supports three scopes:

Strict (Default)

Secret can only be decrypted in the exact namespace with the exact name:

metadata:
  name: my-secret
  namespace: default

Namespace-Wide

Secret can be decrypted in the namespace with any name:

metadata:
  name: my-secret
  namespace: default
  annotations:
    sealedsecrets.bitnami.com/namespace-wide: "true"

Cluster-Wide

Secret can be decrypted anywhere in the cluster:

metadata:
  name: my-secret
  namespace: default
  annotations:
    sealedsecrets.bitnami.com/cluster-wide: "true"

Key Management

Backup Private Key

CRITICAL: Back up the private key or you'll lose access to all sealed secrets!

# Get the private key (KEEP THIS SAFE!)
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o jsonpath='{.items[0].data.tls\.key}' | base64 -d > sealed-secrets-key.pem

# Store in a secure location (e.g., vault, encrypted USB)
chmod 600 sealed-secrets-key.pem

Restore from Backup

If you need to restore the controller (e.g., cluster rebuild):

# Delete existing key
kubectl delete secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

# Create secret from backup key
kubectl create secret tls sealed-secrets-key \
  --cert=sealed-secrets-cert.pem \
  --key=sealed-secrets-key.pem \
  -n kube-system

# Label it
kubectl label secret sealed-secrets-key \
  sealedsecrets.bitnami.com/sealed-secrets-key=active \
  -n kube-system

# Restart controller to pick up key
kubectl rollout restart deployment -n kube-system sealed-secrets

Key Rotation

Keys are automatically rotated every 30 days by default. Old keys are kept for decryption, but new secrets use the latest key.

To manually trigger rotation:

# Delete the current key secret (controller will generate a new one)
kubectl delete secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

# Wait for controller to generate new key
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key

Updating Sealed Secrets

To update a secret:

# 1. Create new secret YAML
cat > updated-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: default
type: Opaque
stringData:
  password: newpassword456
EOF

# 2. Re-encrypt
kubeseal --format yaml < updated-secret.yaml > my-sealedsecret.yaml

# 3. Commit and deploy
git add my-sealedsecret.yaml
git commit -m "Update my-secret"
git push

Important: You cannot edit sealed secrets in-place. You must re-encrypt the entire secret.

Troubleshooting

Check Controller Status

# Check sealed-secrets controller
kubectl get deployment -n kube-system sealed-secrets
kubectl get pods -n kube-system -l app.kubernetes.io/name=sealed-secrets

Check SealedSecret Status

# Check if SealedSecret was decrypted
kubectl get sealedsecret my-secret -n default
kubectl describe sealedsecret my-secret -n default

View Events

# Check events in the namespace
kubectl get events -n default --field-selector reason=Unsealed

Controller Logs

# View controller logs
kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets

Common Issues

1. Secret Not Created

Symptom: SealedSecret exists but no corresponding Secret.

Check:

kubectl describe sealedsecret my-secret -n default
kubectl logs -n kube-system -l app.kubernetes.io/name=sealed-secrets

Possible causes:

Controller not running
Invalid encrypted data (copy-paste error)
Wrong namespace or name (strict scope)

2. Cannot Decrypt After Cluster Rebuild

Cause: Lost private key during cluster rebuild.

Solution: You must re-encrypt all secrets with the new cluster's public certificate. This is why backing up the private key is critical!

3. kubeseal Command Fails

Error: cannot get cert from controller

Solution:

# Ensure kubectl is configured
kubectl config current-context

# Try with explicit cert file
kubeseal --cert sealed-secrets-cert.pem --format yaml < secret.yaml > sealed.yaml

4. Wrong Scope

Symptom: Moving secret to different namespace fails.

Solution: Add cluster-wide or namespace-wide annotation before encrypting:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
  namespace: default
  annotations:
    sealedsecrets.bitnami.com/cluster-wide: "true"

Security Best Practices

Backup private key - Store in secure vault, encrypted USB, or password manager
Rotate keys regularly - Default 30-day rotation is good
Use strict scope - Limit blast radius with namespace/name scoping
Don't commit unencrypted secrets - Review PRs carefully
Use CI/CD to encrypt - Automate sealing in CI pipeline
Audit sealed secrets - Track changes via Git history

CI/CD Integration

GitHub Actions Example

name: Seal Secret
on:
  workflow_dispatch:
    inputs:
      secret_name:
        description: 'Secret name'
        required: true

jobs:
  seal:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install kubeseal
        run: |
          wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.28.0/kubeseal-0.28.0-linux-amd64.tar.gz
          tar -xzf kubeseal-0.28.0-linux-amd64.tar.gz
          sudo mv kubeseal /usr/local/bin/

      - name: Get certificate
        run: |
          echo "${{ secrets.SEALED_SECRETS_CERT }}" > cert.pem

      - name: Seal secret
        run: |
          cat > secret.yaml <<EOF
          apiVersion: v1
          kind: Secret
          metadata:
            name: ${{ inputs.secret_name }}
            namespace: default
          type: Opaque
          stringData:
            password: ${{ secrets.MY_PASSWORD }}
          EOF

          kubeseal --cert cert.pem --format yaml < secret.yaml > sealed.yaml

      - name: Commit sealed secret
        run: |
          git add sealed.yaml
          git commit -m "Seal ${{ inputs.secret_name }}"
          git push

Integration with Other Components

With ArgoCD

SealedSecret works seamlessly with ArgoCD:

Developer → kubeseal → SealedSecret YAML → Git → ArgoCD → Controller → Secret

With cert-manager

Combine with cert-manager for TLS secrets:

# Certificate from cert-manager
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-cert
  namespace: default
spec:
  secretName: my-tls-secret  # Created by cert-manager
  dnsNames:
    - myapp.ssdk8s.xyz
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

With Istio

Sealed secrets for Istio credentials:

# SealedSecret for Istio Gateway TLS
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: istio-gateway-tls
  namespace: istio-system
spec:
  encryptedData:
    tls.crt: AgB...
    tls.key: AgC...
  template:
    metadata:
      name: istio-gateway-tls
      namespace: istio-system
    type: kubernetes.io/tls

Description​

How It Works​

Installation​

Configuration​

Using Sealed Secrets​

Step 1: Install kubeseal CLI​

Step 2: Get the Public Certificate​

Step 3: Create a Secret​

Step 4: Encrypt the Secret​

Step 5: Commit to Git​

Step 6: Deploy via ArgoCD​

Secret Types​

Opaque Secrets​

TLS Secrets​

Docker Registry Secrets​

Scopes​

Strict (Default)​

Namespace-Wide​

Cluster-Wide​

Key Management​

Backup Private Key​

Restore from Backup​

Key Rotation​

Updating Sealed Secrets​

Troubleshooting​

Check Controller Status​

Check SealedSecret Status​

View Events​

Controller Logs​

Common Issues​

1. Secret Not Created​

2. Cannot Decrypt After Cluster Rebuild​

3. kubeseal Command Fails​

4. Wrong Scope​

Security Best Practices​

CI/CD Integration​

GitHub Actions Example​

Integration with Other Components​

With ArgoCD​

With cert-manager​

With Istio​

References​