Logging (Loki + Grafana Alloy)

Purpose: Centralized log aggregation and visualization

Versions: Loki 6.55.0, Grafana Alloy 0.10.0

Namespace: monitoring

Description

Logging stack based on Grafana Loki:

• Loki: Horizontally-scalable log aggregation system
• Grafana Alloy: Unified observability collector for logs, metrics, and traces

Loki is designed to be cost-effective and integrates tightly with Grafana. Alloy is the successor to Promtail, providing a unified observability collector with better Kubernetes integration.

Installation

Installed via ArgoCD from Helm charts:

• grafana/loki - Loki stack (simple scalable mode)
• grafana/alloy - Log collector (DaemonSet mode)

Configuration

Loki Configuration

Key settings:

• Simple Scalable deployment mode
• TSDB storage
• 24h index period
• Structured metadata enabled
• Ingestion rate limit: 16 MB/sec (supports cluster-wide log volume)

Grafana Alloy Configuration

Key settings:

• Runs as DaemonSet on every node
• Uses loki.source.kubernetes for K8s-native log collection
• Discovers pods via Kubernetes API
• Labels logs with: namespace, pod, container, job, app
• Ships logs to http://loki-gateway.monitoring/loki/api/v1/push

Access

Loki is accessed via Grafana using the Loki data source.

Add Loki as Data Source in Grafana

1. Go to https://grafana.ssdk8s.xyz
2. Navigate to Configuration → Data Sources
3. Click Add data source
4. Select Loki
5. Configure:
   • URL: http://loki-gateway.monitoring.svc.cluster.local:80
   • Access: Server (default)
6. Click Save & Test

Query Logs in Grafana

1. Go to Explore in Grafana
2. Select Loki as data source
3. Use LogQL to query logs

LogQL Examples

Find logs by namespace:

{namespace="default"}

Find logs by pod label:

{app="nginx-ingress-ingress-nginx-controller"}

Find error logs:

{namespace="default"} |= "error"

Parse JSON logs:

{namespace="default"} | json | status_code >= 500

Count logs over time:

count_over_time({namespace="default"}[5m])

How It Works

Log Flow

1. Grafana Alloy runs as a DaemonSet on every node
2. Discovers pods via Kubernetes API (discovery.kubernetes)
3. Reads container logs through K8s API (loki.source.kubernetes)
4. Labels each log line with metadata (namespace, pod, container, job, app)
5. Ships logs to Loki via HTTP push API
6. Loki stores and indexes logs by labels
7. Grafana queries and visualizes logs

Alloy Configuration (River)

Alloy uses River configuration language. The key components:

// Discover Kubernetes pods on the same node
discovery.kubernetes "pod" {
  role = "pod"
  selectors {
    role = "pod"
    field = "spec.nodeName=" + coalesce(sys.env("HOSTNAME"), constants.hostname)
  }
}

// Apply relabeling to create proper labels
discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pod.targets

  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }

  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }

  rule {
    source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_container_name"]
    target_label  = "job"
    separator   = "/"
  }
}

// Tail logs from Kubernetes containers
loki.source.kubernetes "pod_logs" {
  targets    = discovery.relabel.pod_logs.output
  forward_to = [loki.write.default.receiver]
}

// Write logs to Loki
loki.write "default" {
  endpoint {
    url = "http://loki-gateway.monitoring/loki/api/v1/push"
  }
}

Storage

• Storage Class: local-path
• Backend: Filesystem (TSDB)
• Retention: Configured based on PVC size

Alloy vs Promtail

Feature	Grafana Alloy	Promtail
Status	Active development	Maintenance mode
Collection	Logs, metrics, traces	Logs only
Configuration	River (HCL-based)	YAML
Kubernetes integration	Native (loki.source.kubernetes)	File-based
Unified pipeline	Yes	Limited

Note: Grafana Alloy replaces Promtail in this cluster. Promtail is deprecated but may still be present for backward compatibility.

Troubleshooting

Check Alloy Status

kubectl get daemonset -n monitoring grafana-alloy
kubectl get pods -n monitoring -l app.kubernetes.io/name=grafana-alloy

Check Alloy Logs

kubectl logs -n monitoring -l app.kubernetes.io/name=grafana-alloy

Check Loki Status

kubectl get pods -n monitoring -l app.kubernetes.io/name=loki
kubectl logs -n monitoring -l app.kubernetes.io/name=loki

Verify Log Collection

1. Check if Alloy is discovering pods:

kubectl logs -n monitoring -l app.kubernetes.io/name=grafana-alloy | grep -i "discovered"

2. Check if logs are reaching Loki:

kubectl logs -n monitoring -l app.kubernetes.io/name=loki | grep -i "ingest"

Common Issues

1. No logs appearing - Verify Alloy pods are running on all nodes
2. Missing labels - Check relabeling configuration in Alloy
3. Ingestion errors - Check Loki rate limits and network connectivity
4. High memory usage - Adjust Alloy scrape intervals and batch sizes