OPA Gatekeeper

Purpose: Policy engine for Kubernetes admission control

Version: 3.22.0

Namespace: gatekeeper-system

Description

OPA Gatekeeper is a validating and mutating webhook that enforces CRD-based policies executed by Open Policy Agent (OPA). It provides admission control, audit, and policy enforcement for your Kubernetes cluster.

This implementation uses GitOps with ArgoCD to manage Gatekeeper policies as code.

Architecture

Installation

Gatekeeper is installed via multi-source Helm pattern:

sources:
  - chart: gatekeeper
    repoURL: https://open-policy-agent.github.io/gatekeeper/charts
    targetRevision: 3.22.0
    helm:
      valueFiles:
        - $values/config/dev/applications/gatekeeper-values.yaml
  - repoURL: https://github.com/simran2491/k8s-playground-cluster
    targetRevision: main
    ref: values

Configuration

Gatekeeper Settings

File: config/dev/applications/gatekeeper-values.yaml

gatekeeperNamespace: gatekeeper-system

# Audit controller configuration
audit:
  replicas: 1
  logLevel: INFO

# Webhook configuration
webhook:
  replicas: 2
  logLevel: INFO

# Enable mutation webhook
enableMutation: true

# Resource constraints
resources:
  audit:
    limits:
      cpu: 1000m
      memory: 512Mi
    requests:
      cpu: 50m
      memory: 64Mi
  webhook:
    limits:
      cpu: 1000m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 128Mi

Policy Structure

Policies are organized in a dedicated Helm chart:

charts/gatekeeper-policies/
├── Chart.yaml
├── values.yaml
└── templates/
    ├── constraint-templates/
    │   ├── k8srequiredlabels.yaml
    │   ├── k8srequiredresources.yaml
    │   ├── k8sdisallowedprivileges.yaml
    │   └── k8smaxresources.yaml
    └── constraints/
        ├── required-labels.yaml
        ├── required-resources.yaml
        ├── no-privileged-containers.yaml
        └── max-resources-per-namespace.yaml

Policy Types

1. Required Labels (K8sRequiredLabels)

Enforces that all workloads have specific labels.

ConstraintTemplate:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

Constraint:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: workload-required-labels
spec:
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment", "StatefulSet", "DaemonSet"]
      - apiGroups: [""]
        kinds: ["Pod"]
    excludedNamespaces:
      - kube-system
      - gatekeeper-system
      - argocd
      # ... system namespaces
  parameters:
    labels: ["app", "team"]

2. Required Resources (K8sRequiredResources)

Ensures all containers have CPU and memory limits/requests defined.

Parameters: None required - just checks existence

Enforcement: Denies pods without:

• resources.limits.cpu
• resources.limits.memory
• resources.requests.cpu
• resources.requests.memory

3. Disallowed Privileges (K8sDisallowedPrivileges)

Blocks privileged containers and privilege escalation.

Checks:

• securityContext.privileged == true
• securityContext.allowPrivilegeEscalation == true

Excluded Namespaces: System namespaces where privileged containers may be required.

4. Maximum Resources (K8sMaxResources)

Enforces per-namespace maximum resource limits.

Configuration:

maxResources:
  namespaces:
    - name: testing
      maxCpu: "500m"
      maxMemory: "512Mi"
    - name: development
      maxCpu: "1000m"
      maxMemory: "1Gi"
    - name: default
      maxCpu: "1000m"
      maxMemory: "2Gi"

Excluded Namespaces

The following namespaces are excluded from policies:

Namespace	Reason
kube-system	Kubernetes system components
gatekeeper-system	Gatekeeper itself
istio-system	Service mesh components
cert-manager	Certificate management
monitoring	Prometheus, Grafana
loki	Logging stack
pihole	Pi-hole DNS
argocd	ArgoCD components
metallb-system	Load balancer
ingress-nginx	Ingress controller
external-dns	DNS management

ArgoCD Integration

Sync Waves

Policies use ArgoCD sync waves for proper ordering:

# ConstraintTemplates - Wave 1
metadata:
  annotations:
    argocd.argopro.io/sync-wave: "1"

# Constraints - Wave 2
metadata:
  annotations:
    argocd.argopro.io/sync-wave: "2"
    argocd.argopro.io/sync-options: SkipDryRunOnMissingResource=true

Ignore Differences

Gatekeeper updates status fields and ArgoCD should ignore them:

ignoreDifferences:
  - group: templates.gatekeeper.sh
    kind: ConstraintTemplate
    jsonPointers:
      - /status
  - group: constraints.gatekeeper.sh
    kind: K8sRequiredLabels
    jsonPointers:
      - /status
      - /metadata/labels
      - /metadata/annotations

Monitoring Violations

Check All Constraints

kubectl get k8srequiredlabels,k8srequiredresources,k8sdisallowedprivileges,k8smaxresources \
  -o custom-columns=POLICY:.metadata.name,VIOLATIONS:.status.totalViolations

Check Specific Violations

# Label violations
kubectl get k8srequiredlabels workload-required-labels -o json | \
  jq '.status.violations[] | "\(.namespace)/\(.name) (\(.kind))"'

# Resource limit violations
kubectl get k8srequiredresources workload-required-resources -o json | \
  jq '.status.violations[] | "\(.namespace)/\(.name) (\(.kind))"'

# Max resources by namespace
kubectl get k8smaxresources max-resources-testing -o json | \
  jq '.status.violations[]'

Audit Logs

# Gatekeeper audit pod
kubectl logs -n gatekeeper-system -l control-plane=gatekeeper-audit --tail=200

# Controller manager
kubectl logs -n gatekeeper-system -l control-plane=gatekeeper-controller-manager --tail=200

Troubleshooting

ConstraintTemplate Not Created

Error: failed to discover server resources for group version constraints.gatekeeper.sh/v1beta1

Solution: Ensure Gatekeeper is installed and running before applying policies. Check sync waves are configured correctly.

Strict Decoding Error

Error: unknown field "metadata.app.kubernetes.io/instance"

Cause: Gatekeeper CRDs have strict schemas that reject arbitrary Helm labels.

Solution: Remove helm.sh/* and app.kubernetes.io/* labels from Constraint and ConstraintTemplate manifests. Only keep ArgoCD annotations.

Violations Not Clearing

1. Check if namespace is excluded
2. Verify the resource has the required fields
3. Wait for audit cycle (up to 60 seconds)
4. Check constraint pod statuses:

   kubectl get constraintpodstatuses.status.gatekeeper.sh -n gatekeeper-system

Best Practices

Policy Development

1. Test in dev first - Validate policies in non-production
2. Use audit mode - Set enforcementAction: dryrun initially
3. Start with exclusions - Exclude system namespaces from day one
4. Version policies - Keep policies in Git with application code

Performance

1. Limit scope - Use specific match.kinds instead of wildcards
2. Exclude namespaces - Don't audit system namespaces
3. Tune audit - Adjust audit interval based on cluster size

Security

1. Deny by default - Use enforcementAction: deny for production
2. Block privileges - Always enforce no-privileged-containers
3. Resource limits - Prevent resource exhaustion attacks
4. Label requirements - Enable better cost allocation and ownership

Adding New Policies

1. Create ConstraintTemplate in templates/constraint-templates/
2. Create Constraint in templates/constraints/
3. Add sync-wave annotations (CT=1, Constraint=2)
4. Add SkipDryRunOnMissingResource=true to Constraints
5. Update values.yaml with configuration
6. Test with helm template before committing
7. Commit and create release tag

References

• OPA Gatekeeper Documentation
• Gatekeeper Policy Library
• Rego Language Reference