Skip to main content

Adding Nodes

This cluster uses kubeadm on AlmaLinux with Kubernetes v1.29.

On Master Node​

Generate a join token:

kubeadm token create --print-join-command

Copy the output - it will look like:

kubeadm join 10.0.0.219:6443 --token xxxxxx --discovery-token-ca-cert-hash sha256:xxxxxx
note

The token expires after 24 hours by default. Use kubeadm token create to generate a new one if needed.

On New Node​

Run these commands as root:

# 1. Set hostname
hostnamectl set-hostname node05

# 2. Stop firewall
systemctl stop firewalld
systemctl stop iptables

# 3. Disable SELinux
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

# 4. Disable swap
swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab

# 5. Load kernel modules
modprobe overlay
modprobe br_netfilter

# 6. Configure sysctl
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system

# 7. Add k8s repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.29/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

# 8. Add Docker CE repo and install containerd
dnf install -y dnf-plugins-core
dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf install -y containerd.io
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl enable --now containerd

# 9. Install k8s components
dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

# 10. Enable kubelet
systemctl enable --now kubelet

# 11. Join cluster (USE TOKEN FROM MASTER)
kubeadm join 10.0.0.219:6443 --token <TOKEN> --discovery-token-ca-cert-hash sha256:<HASH>

On Master Node Again​

Approve the CSR if needed:

kubectl get nodes
kubectl get csr
# If there are pending CSRs:
kubectl certificate approve <csr-name>

Verify:

kubectl get nodes -o wide

Post-Join Verification​

  1. Check node status:

    kubectl get nodes
    # Should show Ready status

  2. Check system pods:

    kubectl get pods -n kube-system -o wide
    # Verify calico-node, kube-proxy are scheduled on new node

  3. Check MetalLB speaker:

    kubectl get pods -n metallb-system -o wide
    # Verify speaker pod is running on new node

  4. Check Promtail:

    kubectl get pods -n monitoring -l app.kubernetes.io/name=promtail -o wide
    # Verify Promtail is collecting logs from new node

Troubleshooting​

Node Not Ready​

# Check kubelet status
systemctl status kubelet
journalctl -u kubelet -f

# Check certificates
ls -la /etc/kubernetes/pki/

# Rejoin if needed
kubeadm reset
# Then run join command again

Pod Not Scheduled​

# Check node taints
kubectl describe node <node-name> | grep Taint

# Check node labels
kubectl get node <node-name> --show-labels

Network Issues​

# Check calico-node pod
kubectl get pods -n calico-system -o wide

# Check BGP peering
kubectl exec -n calico-system <calico-node-pod> -- calicoctl node status