DaemonSets

In the previous tutorial, we ran stateful applications with StatefulSets — databases, message queues, the heavy hitters. Now let's talk about a totally different kind of workload.

Sometimes you need a pod running on every single node in your cluster. Not 3 replicas, not 5 — one on every node. When nodes are added, the pod automatically shows up. When nodes are removed, the pod cleans itself up.

That's a DaemonSet. Think of it like installing a security camera on every floor of a building. New floor added? Camera goes up automatically.

Use Cases

DaemonSets are the workhorses of cluster infrastructure:

Log collectors: Fluentd, Filebeat, Logstash (gotta collect those logs from every node)
Monitoring agents: Prometheus Node Exporter, Datadog agent (gotta watch every node)
Network plugins: Calico, Weave, Cilium (networking infrastructure)
Storage daemons: Ceph, GlusterFS
Security agents: Falco, Sysdig (gotta secure every node)

DaemonSet vs Deployment

"Why not just use a Deployment with enough replicas?"

Because DaemonSets guarantee exactly one pod per node, and they scale automatically as nodes are added/removed:

Feature	Deployment	DaemonSet
Pods per node	Any number	Exactly one
Scheduling	Kubernetes scheduler	DaemonSet controller
Scaling	Manual replicas	Automatic (1 per node)
Use case	Application workloads	Node-level services

Create a DaemonSet

Let's deploy a log collector on every node — the classic DaemonSet use case:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.16
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: containers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: containers
        hostPath:
          path: /var/lib/docker/containers
      terminationGracePeriodSeconds: 30

Apply:

kubectl apply -f fluentd-daemonset.yaml

Check pods:

kubectl get daemonset fluentd

NAME      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluentd   3         3         3       3            3           <none>          30s

kubectl get pods -l app=fluentd -o wide

NAME            READY   STATUS    NODE
fluentd-abc12   1/1     Running   node-1
fluentd-def34   1/1     Running   node-2
fluentd-ghi56   1/1     Running   node-3

One pod per node, automatically. You didn't tell it how many replicas — it just figured out how many nodes there are and deployed accordingly. Beautiful.

Node Selector

"What if I only want the DaemonSet on certain nodes?"

Use a node selector:

spec:
  template:
    spec:
      nodeSelector:
        disk: ssd

Only nodes with label disk=ssd get the pod.

Label nodes:

kubectl label nodes node-1 disk=ssd
kubectl label nodes node-2 disk=ssd

Node Affinity

For more flexible node selection (when nodeSelector isn't enough):

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
              - key: node-type
                operator: NotIn
                values:
                - virtual

This runs on Linux nodes that aren't virtual. Because sometimes you need to be picky.

Tolerations

"My DaemonSet isn't running on the control plane node. What gives?"

Control plane nodes have "taints" that repel regular pods (like a force field). DaemonSets need tolerations to schedule there:

spec:
  template:
    spec:
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule

Now the DaemonSet runs on control plane nodes too.

Tolerate All Taints

For critical system daemons that absolutely MUST run everywhere, no exceptions:

spec:
  template:
    spec:
      tolerations:
      - operator: Exists

This tolerates all taints on all nodes. The nuclear option. Use it for things like log collectors and monitoring agents that need to be everywhere.

Update Strategy

RollingUpdate (Default)

Update pods one at a time across nodes:

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1

maxUnavailable: How many nodes can be without the daemon during update
Can be a number or percentage

OnDelete

Only update when a pod is manually deleted — for when you want full control:

spec:
  updateStrategy:
    type: OnDelete

Useful for critical daemons where you want to update nodes one at a time during maintenance windows. No surprises.

Real-World Example: Node Exporter

Let's deploy something real — Prometheus Node Exporter for collecting system metrics:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.7.0
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root
        - --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+)($|/)
        ports:
        - containerPort: 9100
          hostPort: 9100
        resources:
          limits:
            cpu: 250m
            memory: 180Mi
          requests:
            cpu: 100m
            memory: 128Mi
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
        - name: root
          mountPath: /host/root
          readOnly: true
          mountPropagation: HostToContainer
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: root
        hostPath:
          path: /
      tolerations:
      - operator: Exists
---
apiVersion: v1
kind: Service
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    app: node-exporter
  ports:
  - port: 9100
  clusterIP: None  # Headless for discovery

Key configurations (and why each matters):

hostNetwork: true — uses node's network namespace (sees real network traffic)
hostPID: true — sees host processes (needed for process metrics)
hostPort: 9100 — exposes on node's IP directly
Volume mounts to /host/proc, /host/sys — reads system information
Tolerates all taints — runs on every node, including control plane

This is basically installing htop on every node, but with Prometheus-compatible output.

Real-World Example: Log Shipper

Another classic — Filebeat shipping logs to Elasticsearch:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      serviceAccountName: filebeat
      containers:
      - name: filebeat
        image: elastic/filebeat:8.11.0
        args:
        - -c
        - /etc/filebeat/filebeat.yml
        - -e
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch.logging.svc.cluster.local
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: containers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: filebeat-config
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
      - name: varlog
        hostPath:
          path: /var/log
      - name: containers
        hostPath:
          path: /var/lib/docker/containers
      tolerations:
      - operator: Exists
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: logging
data:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - /var/lib/docker/containers/*/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/lib/docker/containers/"
    output.elasticsearch:
      hosts: ["${ELASTICSEARCH_HOST}:9200"]

Communicating with DaemonSet Pods

"How do other pods talk to DaemonSet pods?"

Good question! There are a few patterns:

Service with Endpoints

Create a headless service for DNS discovery:

apiVersion: v1
kind: Service
metadata:
  name: fluentd
spec:
  clusterIP: None
  selector:
    app: fluentd
  ports:
  - port: 24224

Now other pods can discover all fluentd instances via DNS. Handy.

NodeLocal Access

If DaemonSet uses hostPort, you can access it directly via the node's IP:

curl http://<node-ip>:9100/metrics

Using Environment Variables

Pods can find the daemon on their node:

env:
- name: NODE_IP
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP

Then connect to $(NODE_IP):9100. Your pod talks to the daemon running on its own node. No cross-node traffic needed.

Priority and Preemption

Ensure critical daemons aren't evicted when the node runs low on resources:

spec:
  template:
    spec:
      priorityClassName: system-node-critical

Built-in priority classes:

system-node-critical — highest priority ("do NOT evict me, I am essential")
system-cluster-critical — high priority ("please don't evict me, I'm important")

Viewing DaemonSet Status

kubectl get daemonset fluentd

NAME      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
fluentd   3         3         3       3            3           <none>          5m

Columns explained:

DESIRED: Number of nodes that should run the daemon
CURRENT: Pods currently scheduled
READY: Pods that are ready to serve
UP-TO-DATE: Pods running the latest template
AVAILABLE: Pods available for service

If DESIRED doesn't match READY, something's wrong. Time to investigate.

Detailed status:

kubectl describe daemonset fluentd

Troubleshooting

Pods Not Scheduled on Some Nodes

Most likely a taint issue. Check node taints:

kubectl describe node <node-name> | grep Taints

Add tolerations to the DaemonSet or remove taints from nodes. Nine times out of ten, it's a taint issue.

DaemonSet Pods Evicted

Pods may be evicted if the node is under resource pressure. That's bad for system daemons. Solutions:

Set appropriate resource requests/limits
Use priorityClassName: system-node-critical (the "don't touch me" flag)
Configure a Pod Disruption Budget (covered in the last tutorial!)

Pods Stuck in Pending

kubectl describe pod <daemonset-pod>

Check for:

Node affinity/selector not matching
Insufficient resources on node
PVC binding issues

Clean Up

kubectl delete daemonset fluentd
kubectl delete daemonset node-exporter -n monitoring
kubectl delete daemonset filebeat -n logging

What's Next?

Nice work! You now know how to run node-level services across your entire cluster automatically. Log collection, monitoring, security scanning — DaemonSets have you covered.

But sometimes a single pod needs to do more than just run one container. What if you need a setup task before the main app starts? Or a helper container running alongside it? In the next tutorial, we'll dive into Init Containers and Sidecars — patterns for building smarter, more capable pods. Let's go!