Skip to main content

kube-prometheus-stack

Purpose: Complete monitoring stack with Prometheus, Grafana, and Alertmanager

Version: 68.0.0

Namespace: monitoring

Description​

Comprehensive monitoring solution based on Prometheus Operator. Includes:

  • Prometheus for metrics collection and storage
  • Grafana for visualization and dashboards
  • Alertmanager for alert routing and notification
  • node-exporter for node-level metrics
  • kube-state-metrics for Kubernetes object metrics
  • Grafana Alloy for unified observability collection (logs, metrics, traces)

Installation​

Installed via ArgoCD from Helm chart: prometheus-community/kube-prometheus-stack

Configuration​

Configuration file: config/dev/applications/kube-prometheus-stack-values.yaml

Key settings:

  • 30s scrape interval
  • 15-day retention
  • 20Gi persistent storage (local-path)
  • ServiceMonitor-based discovery
  • Grafana Alloy integration for metrics collection

Access​

ServiceURLPort
Prometheushttps://prometheus.ssdk8s.xyz9090
Grafanahttps://grafana.ssdk8s.xyz3000
Alertmanagerhttps://alertmanager.ssdk8s.xyz9093

Credentials​

Grafana Username: admin

Get Grafana Password:

kubectl get secret kube-prometheus-stack-grafana \
  -n monitoring \
  -o jsonpath='{.data.admin-password}' | base64 -d

Dashboards​

Pre-installed dashboards:

  • Kubernetes / Compute Resources / Cluster
  • Kubernetes / Compute Resources / Namespace (pods)
  • Kubernetes / Compute Resources / Workloads
  • ArgoCD (Dashboard ID: 19993)
tip

Import Grafana dashboard ID 19993 for comprehensive ArgoCD monitoring.

Prometheus Configuration​

ServiceMonitor Selectors​

Prometheus scrapes targets with label: release: kube-prometheus-stack

Scraped Targets​

  • kube-state-metrics
  • node-exporter
  • Prometheus itself
  • Alertmanager
  • Ingress NGINX
  • ArgoCD components (server, repo-server, controller, applicationset)
  • Istio components (istiod, gateway, sidecars via annotations)
  • OPA Gatekeeper (audit and controller metrics)
  • MetalLB
  • cert-manager
  • External-DNS
  • Grafana Alloy (self-monitoring)

Grafana Alloy Integration​

Grafana Alloy collects and forwards metrics to Prometheus:

  • Discovers targets via Kubernetes service discovery
  • Supports metrics relabeling
  • Forwards to Prometheus via remote_write
  • Also collects logs and traces (see logging.md)

See logging.md for complete Grafana Alloy documentation.

Storage​

  • Storage Class: local-path
  • Size: 20Gi
  • Retention: 15 days

Alertmanager​

Adding Custom Metrics​

Create a ServiceMonitor​

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  namespace: default
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s

Expose Metrics in Your App​

apiVersion: v1
kind: Service
metadata:
  name: my-app
  namespace: default
  labels:
    app: my-app
spec:
  ports:
    - name: metrics
      port: 8080
      targetPort: metrics
  selector:
    app: my-app

Troubleshooting​

Check Prometheus Targets​

# Check ServiceMonitors
kubectl get servicemonitor -A

# Check Prometheus status
kubectl get prometheus -n monitoring

# Check Prometheus pods
kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus

# View Prometheus targets (via UI)
# Open https://prometheus.ssdk8s.xyz/targets

Check Grafana Dashboards​

# Check Grafana pods
kubectl get pods -n monitoring -l app.kubernetes.io/name=grafana

# Get Grafana admin password
kubectl get secret kube-prometheus-stack-grafana \
  -n monitoring \
  -o jsonpath='{.data.admin-password}' | base64 -d

Check Storage​

kubectl get pvc -n monitoring
kubectl describe pvc prometheus-kube-prometheus-prometheus-0 -n monitoring

Check Grafana Alloy Metrics Collection​

# Check Alloy pods
kubectl get pods -n monitoring -l app.kubernetes.io/name=grafana-alloy

# Check Alloy metrics
kubectl logs -n monitoring -l app.kubernetes.io/name=grafana-alloy | grep -i metric

Common Issues​

  1. No data in Grafana - Check ServiceMonitor labels match Prometheus selector (release: kube-prometheus-stack)
  2. PVC pending - Ensure local-path-provisioner is running
  3. High memory usage - Increase retention or reduce scrape targets
  4. Missing Istio metrics - Ensure pods have Istio sidecar and Prometheus annotations
  5. Alloy not scraping - Check Alloy configuration and Kubernetes service discovery