Istio Service Mesh

Purpose: Service mesh for traffic management, security, and observability

Version: 1.24.0

Namespace: istio-system

Description

Istio is a service mesh that provides traffic management, security, and observability features for microservices. It uses sidecar proxies (Envoy) or ambient mode to intercept and control all service-to-service communication.

This cluster uses Istio in sidecar mode for fine-grained traffic control and mTLS security.

Architecture

Components

Control Plane

Component	Deployment	Purpose
istiod	`istiod`	Unified control plane (Pilot + Citadel + Galley)
Ingress Gateway	`istio-ingressgateway`	Edge load balancer for incoming traffic

Data Plane

Component	Type	Purpose
Envoy Sidecar	Init Container + Container	Intercepts all pod network traffic
istio-init	Init Container	Sets up iptables rules
istio-proxy	Sidecar Container	Envoy proxy for traffic management

Installation

Istio is installed via multi-source Helm pattern:

sources:
  - chart: istio-base
    repoURL: https://istio-release.storage.googleapis.com/charts
    targetRevision: 1.24.0
  - repoURL: https://github.com/simran2491/k8s-playground-cluster
    targetRevision: main
    ref: values

Installation Order

istio-base - Installs CRDs (VirtualService, DestinationRule, Gateway, etc.)
istiod - Control plane components
istio-ingressgateway - Ingress gateway (optional, if using Istio ingress)

Configuration

Sidecar Injection

Istio automatically injects sidecars into pods with the istio-injection=enabled label:

kubectl label namespace default istio-injection=enabled

Pod annotation to control injection:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    sidecar.istio.io/inject: "true"  # or "false" to disable

Resource Limits

Sidecar proxy resources:

proxy:
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 256Mi

Pilot (istiod) Resources

pilot:
  resources:
    requests:
      cpu: 500m
      memory: 2Gi
    limits:
      cpu: 1000m
      memory: 4Gi

Traffic Management

VirtualService

Defines how requests are routed to services:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: my-app
  namespace: default
spec:
  hosts:
    - my-app
  http:
    - match:
        - uri:
            prefix: /api
      route:
        - destination:
            host: my-app
            port:
              number: 8080
      retries:
        attempts: 3
        perTryTimeout: 2s

DestinationRule

Configures traffic policy for service destinations:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-app
  namespace: default
spec:
  host: my-app
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: UPGRADE
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000
    loadBalancer:
      simple: ROUND_ROBIN
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s

Gateway

Defines edge load balancer for incoming traffic:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: my-app-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: my-app-tls
      hosts:
        - myapp.ssdk8s.xyz

Security Features

mTLS (Mutual TLS)

Istio provides automatic mTLS between services:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT  # or PERMISSIVE for migration

Authorization Policies

Control access between services:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: require-jwt
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-api
  action: ALLOW
  rules:
    - from:
        - source:
            requestPrincipals: ["*"]

JWT Authentication

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: jwt-auth
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-api
  jwtRules:
    - issuer: "https://auth.ssdk8s.xyz"
      jwksUri: "https://auth.ssdk8s.xyz/.well-known/jwks.json"

Observability

Distributed Tracing

Istio automatically generates tracing spans for all service calls. Compatible with:

Jaeger
Zipkin
OpenTelemetry

Metrics

Istio generates standard metrics:

istio_requests_total - Request count
istio_request_duration_milliseconds - Request latency
istio_request_bytes - Request size
istio_response_bytes - Response size

Scraped by Prometheus automatically via ServiceMonitor.

Access Logs

Envoy sidecars generate access logs:

meshConfig:
  accessLogFile: /dev/stdout
  accessLogFormat: |
    [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" 
    %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% 
    "%UPSTREAM_HOST%" "%USER_AGENT%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%"

Integration with Other Components

With NGINX Ingress

Internet → NGINX Ingress → Istio Gateway → VirtualService → Service → Pod+Sidecar

With cert-manager

Istio Gateway uses TLS secrets created by cert-manager:

# Certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-app-tls
  namespace: default
spec:
  secretName: my-app-tls  # Referenced by Istio Gateway
  dnsNames:
    - myapp.ssdk8s.xyz
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

With Prometheus

Istio exposes metrics at :15090/metrics - Prometheus scrapes automatically via:

Pod annotations
ServiceMonitor (if configured)

With Loki

Istio access logs can be shipped to Loki via:

Fluent Bit sidecar
Grafana Alloy (recommended)

Monitoring

Check Istio Health

# Control plane status
istioctl analyze

# Proxy status for a pod
istioctl proxy-status <pod-name>.<namespace>

# Check sidecar configuration
istioctl proxy-config <pod-name>.<namespace>

Metrics Dashboard

Access via Grafana:

Istio Mesh Dashboard - Overall mesh health
Istio Service Dashboard - Per-service metrics
Istio Workload Dashboard - Per-workload metrics
Istio Performance Dashboard - Latency, throughput

Tracing

Access via Jaeger/Tempo:

View request traces across services
Identify latency bottlenecks
Debug service-to-service calls

Common Issues

Sidecar Not Injecting

Check namespace label:

kubectl get namespace default -o jsonpath='{.metadata.labels}'
# Should show: istio-injection=enabled

Check injection webhook:

kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml

503 Service Unavailable

Cause: Destination rule misconfiguration or no healthy endpoints.

Debug:

istioctl proxy-config endpoints <pod-name>.<namespace>
istioctl analyze

mTLS Issues

Check mTLS mode:

istioctl analyze --namespace default

Fix:

# Set to PERMISSIVE for debugging
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: PERMISSIVE
EOF

Best Practices

Traffic Management

Use retries - Configure retries for transient failures
Circuit breakers - Prevent cascade failures
Timeouts - Set reasonable timeouts on all routes
Canary deployments - Use VirtualService weights for gradual rollouts

Security

STRICT mTLS - Use STRICT mode in production
Authorization policies - Default deny, explicit allow
JWT validation - Validate tokens at ingress
Regular rotation - Rotate certificates regularly

Performance

Resource limits - Set appropriate sidecar resources
Connection pooling - Configure connection pools
Outlier detection - Eject unhealthy endpoints
Load balancing - Choose appropriate LB algorithm

Observability

Distributed tracing - Enable tracing for all services
Access logs - Standardize log format
Metrics - Monitor key Istio metrics
Dashboards - Create custom Grafana dashboards

Commands Reference

# Install istioctl CLI
curl -L https://istio.io/downloadIstio | sh -
export PATH=$PWD/bin:$PATH

# Analyze cluster
istioctl analyze

# Check proxy status
istioctl proxy-status

# View proxy config
istioctl proxy-config <pod-name>.<namespace>

# View endpoints
istioctl proxy-config endpoints <pod-name>.<namespace>

# View listeners
istioctl proxy-config listeners <pod-name>.<namespace>

# View routes
istioctl proxy-config route <pod-name>.<namespace>

# Check certificate
istioctl proxy-config secret <pod-name>.<namespace>

# Describe pod
istioctl describe pod <pod-name>.<namespace>

# Check injection
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].name}'
# Should show: istio-proxy container

Description​

Architecture​

Components​

Control Plane​

Data Plane​

Installation​

Installation Order​

Configuration​

Sidecar Injection​

Resource Limits​

Pilot (istiod) Resources​

Traffic Management​

VirtualService​

DestinationRule​

Gateway​

Security Features​

mTLS (Mutual TLS)​

Authorization Policies​

JWT Authentication​

Observability​

Distributed Tracing​

Metrics​

Access Logs​

Integration with Other Components​

With NGINX Ingress​

With cert-manager​

With Prometheus​

With Loki​

Monitoring​

Check Istio Health​

Metrics Dashboard​

Tracing​

Common Issues​

Sidecar Not Injecting​

503 Service Unavailable​

mTLS Issues​

Best Practices​

Traffic Management​

Security​

Performance​

Observability​

Commands Reference​

References​

Description

Architecture

Components

Control Plane

Data Plane

Installation

Installation Order

Configuration

Sidecar Injection

Resource Limits

Pilot (istiod) Resources

Traffic Management

VirtualService

DestinationRule

Gateway

Security Features

mTLS (Mutual TLS)

Authorization Policies

JWT Authentication

Observability

Distributed Tracing

Metrics

Access Logs

Integration with Other Components

With NGINX Ingress

With cert-manager

With Prometheus

With Loki

Monitoring

Check Istio Health

Metrics Dashboard

Tracing

Common Issues

Sidecar Not Injecting

503 Service Unavailable

mTLS Issues

Best Practices

Traffic Management

Security

Performance

Observability

Commands Reference

References