Calico

Purpose: Network policy enforcement and CNI plugin

Version: Latest (via Helm)

Namespace: calico-system

Description

Calico provides networking and network policy enforcement for Kubernetes clusters. It replaces Flannel as the CNI plugin for better security and policy control.

Key features:

• Pod Networking: Layer 3 networking with IP-per-pod
• Network Policies: Full Kubernetes NetworkPolicy API enforcement
• Security: Pod-to-pod encryption support, policy isolation
• Observability: Flow logs and network metrics
• Cross-node: VXLAN encapsulation for multi-node clusters

Installation

Installed via ArgoCD from Helm charts:

• projectcalico/tigera-operator - Operator for lifecycle management
• projectcalico/calico - Calico components

Configuration file: cluster/dev/config.yaml

Architecture

Configuration

IP Pool

Setting	Value
Pod CIDR	10.244.0.0/16
Encapsulation	VXLAN (cross-node)
IP Mode	Always (first found)
NAT Outgoing	Enabled
Node-to-Node Mesh	Enabled

Policy Enforcement

• Default: Allow all traffic (unless NetworkPolicy exists)
• With NetworkPolicy: Strict enforcement (default deny)

Features

Network Policy Types

1. Ingress Policies - Control incoming traffic
2. Egress Policies - Control outgoing traffic
3. Default Deny - Block all traffic by default
4. Allow Specific - Whitelist specific sources/destinations

Advanced Features

• GlobalNetworkPolicy - Cluster-wide policies
• NetworkSet - Group of external IPs/CIDRs
• HostEndpoint - Policies for node interfaces
• Tier - Policy ordering and priority

Usage

Default Deny All Traffic

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: default
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Allow Ingress from Specific Pods

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api
      ports:
        - protocol: TCP
          port: 80

Allow Egress to DNS

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: default
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53

Allow Egress to Internet (External)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-internet
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: external-service
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 172.16.0.0/12
              - 192.168.0.0/16
      ports:
        - protocol: TCP
          port: 443
        - protocol: TCP
          port: 80

Allow Traffic from Namespace

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-monitoring
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: my-app
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - protocol: TCP
          port: 9090

Components

Component	Deployment	Namespace	Purpose
tigera-operator	tigera-operator	tigera-operator	Operator lifecycle management
calico-node	DaemonSet	calico-system	Runs on each node, handles networking
calico-kube-controllers	Deployment	calico-system	Syncs K8s objects to Calico
calico-apiserver	Deployment	calico-apiserver	Calico API server (optional)

Troubleshooting

Check Calico Pods

# Check all Calico pods
kubectl get pods -n calico-system

# Check calico-node on specific node
kubectl get pods -n calico-system -o wide | grep <node-name>

# Check tigera-operator
kubectl get pods -n tigera-operator

Check Network Policies

# List all network policies
kubectl get networkpolicy -A

# Describe specific policy
kubectl describe networkpolicy <name> -n <namespace>

# Check policy enforcement
kubectl get networkpolicy default-deny-all -n default -o yaml

Check Calico Status

# Check Felix configuration
kubectl get felixconfigurations.crd.projectcalico.org default -o yaml

# Check IP pools
kubectl get ippools.crd.projectcalico.org

# Check BGP peers (if using BGP)
kubectl get bgppeers.crd.projectcalico.org

Test Network Connectivity

# Test connectivity between pods
kubectl run test-pod --rm -it --image=busybox --restart=Never -- nslookup kubernetes

# Test from one pod to another
kubectl exec -it <pod-name> -- ping <target-pod-ip>

# Test port connectivity
kubectl exec -it <pod-name> -- nc -zv <target-ip> <port>

Check Calico Logs

# calico-node logs
kubectl logs -n calico-system -l k8s-app=calico-node --tail=50

# kube-controllers logs
kubectl logs -n calico-system -l k8s-app=calico-kube-controllers --tail=50

# tigera-operator logs
kubectl logs -n tigera-operator -l name=tigera-operator --tail=50

Common Issues

1. Pod networking broken

   • Check calico-node pods are running on all nodes
   • Verify CNI config: ls -la /etc/cni/net.d/
   • Restart calico-node: kubectl rollout restart daemonset/calico-node -n calico-system

2. Policies not enforced

   • Verify br_netfilter module is loaded: lsmod | grep br_netfilter
   • Check sysctl: sysctl net.bridge.bridge-nf-call-iptables (should be 1)
   • Ensure kube-proxy is running

3. Cross-node communication fails

   • Check VXLAN port (4789) is open between nodes
   • Verify firewall rules allow UDP 4789
   • Check node-to-node mesh: kubectl get felixconfigurations default -o yaml

4. DNS resolution broken after policy

   • Ensure DNS egress is allowed (see example above)
   • Check CoreDNS pods are accessible: kubectl get pods -n kube-system -l k8s-app=kube-dns

5. IP address exhaustion

   • Check IP pool usage: kubectl get ippools.crd.projectcalico.org -o yaml
   • Expand IP pool if needed (requires careful planning)

Debug Commands

# Get detailed pod network info
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.podIP}'

# Check node labels for Calico
kubectl get nodes --show-labels | grep calico

# Verify CNI plugin
kubectl get pods -n kube-system -l k8s-app=calico-node -o wide

# Check iptables rules (on node)
sudo iptables-save | grep calico

# Check VXLAN interface (on node)
ip -d link show vxlan.calico

Best Practices

Policy Design

1. Start with default deny - Begin with deny-all, then allow specific traffic
2. Use labels consistently - Match pods using app, tier, environment labels
3. Namespace isolation - Use namespace selectors for cross-namespace policies
4. Document policies - Add comments explaining why each policy exists

Security

1. Least privilege - Only allow required traffic
2. Egress filtering - Don't forget outbound traffic
3. DNS access - Always allow DNS egress
4. Monitoring access - Allow Prometheus/scraping access

Performance

1. Policy ordering - More specific policies first
2. Label selectors - Use efficient label selectors
3. Avoid broad CIDRs - Be specific with IP blocks
4. Monitor Felix CPU - High CPU may indicate policy complexity

References

• Calico Documentation
• Kubernetes Network Policies
• Calico GitHub