Kubernetes, Deep Dive

Before you change anything, find out where you are:

kubectl config current-context
kubectl get nodes -o wide

kind-k8s-lab

NAME                    STATUS   ROLES           AGE   VERSION   INTERNAL-IP   OS-IMAGE             CONTAINER-RUNTIME
k8s-lab-control-plane   Ready    control-plane   8m    v1.34.0   172.18.0.4    Debian GNU/Linux 12  containerd://2.1.4
k8s-lab-worker          Ready    <none>          7m    v1.34.0   172.18.0.2    Debian GNU/Linux 12  containerd://2.1.4
k8s-lab-worker2         Ready    <none>          7m    v1.34.0   172.18.0.3    Debian GNU/Linux 12  containerd://2.1.4

1.1 — Why do the workers say <none>?

Because "role" isn't a real field. It's just a label, and kubectl prints any label named node-role.kubernetes.io/* in that column:

kubectl get nodes --show-labels | tr ',' '\n' | grep node-role

node-role.kubernetes.io/control-plane=

Only one node has one. The workers aren't missing anything — nobody bothered to label them. Prove it costs nothing:

kubectl label node k8s-lab-worker node-role.kubernetes.io/worker=
kubectl label node k8s-lab-worker2 node-role.kubernetes.io/worker=
kubectl get nodes

NAME                    STATUS   ROLES           AGE   VERSION
k8s-lab-control-plane   Ready    control-plane   9m    v1.34.0
k8s-lab-worker          Ready    worker          8m    v1.34.0
k8s-lab-worker2         Ready    worker          8m    v1.34.0

1.2 — Why your apps will never land on the control plane

The control-plane node is a perfectly healthy machine with free capacity. Your Pods will still refuse to go there. Find out why:

kubectl describe node k8s-lab-control-plane | grep -i -A2 taints
kubectl describe node k8s-lab-worker | grep -i -A2 taints

Taints:   node-role.kubernetes.io/control-plane:NoSchedule
Taints:   <none>

1.3 — How much room is there, really?

The scheduler does arithmetic, so you should see the numbers it works with:

kubectl get nodes -o custom-columns='NODE:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory,PODS:.status.allocatable.pods'

NODE                    CPU   MEM        PODS
k8s-lab-control-plane   8     8123456Ki  110
k8s-lab-worker          8     8123456Ki  110
k8s-lab-worker2         8     8123456Ki  110

Allocatable is what's left for Pods after the kubelet and the OS take their cut. Because kind nodes are containers sharing your laptop, all three report your whole machine — they are not really separate hardware, and they will happily oversubscribe. On a real cluster these numbers differ per machine and matter enormously.

1.4 — Namespaces, and the machinery in kube-system

kubectl get namespaces
kubectl get pods            # your workspace: empty

"No resources found in default namespace" is correct, not an error. Kubernetes is being honest. Now look where the interesting things live:

kubectl get pods -n kube-system -o wide

NAME                                            READY   STATUS    NODE
coredns-66bc5c9577-8xm4d                        1/1     Running   k8s-lab-control-plane
coredns-66bc5c9577-qd4l2                        1/1     Running   k8s-lab-control-plane
etcd-k8s-lab-control-plane                      1/1     Running   k8s-lab-control-plane
kube-apiserver-k8s-lab-control-plane            1/1     Running   k8s-lab-control-plane
kube-controller-manager-k8s-lab-control-plane   1/1     Running   k8s-lab-control-plane
kube-scheduler-k8s-lab-control-plane            1/1     Running   k8s-lab-control-plane
kindnet-7x2mq                                   1/1     Running   k8s-lab-control-plane
kindnet-bl9vk                                   1/1     Running   k8s-lab-worker
kindnet-mn4rt                                   1/1     Running   k8s-lab-worker2
kube-proxy-fl9rz                                1/1     Running   k8s-lab-control-plane
kube-proxy-hc8dm                                1/1     Running   k8s-lab-worker
kube-proxy-vz6kp                                1/1     Running   k8s-lab-worker2

Read the NODE column and a pattern jumps out. Some things exist once, on the control plane. Others exist three times, once per node.

How does something run on every node automatically?

kubectl get daemonsets -n kube-system

NAME         DESIRED   CURRENT   READY   NODE SELECTOR   AGE
kindnet      3         3         3       <none>          11m
kube-proxy   3         3         3       kubernetes.io/os=linux   11m

Most people treat kubectl apply as magic. It isn't, and the day something breaks you will need to know which of six components let you down. So let's take one command apart.

2.1 — kubectl is an HTTP client. That's all.

kubectl config view --minify --raw=false | grep -E 'server|name:'
kubectl get nodes -v=6

    server: https://127.0.0.1:45231
    name: kind-k8s-lab

I1016 11:04:22.118   round_trippers.go:553] GET https://127.0.0.1:45231/api/v1/nodes?limit=500 200 OK in 11 milliseconds
NAME                    STATUS   ROLES    AGE   VERSION
...

You can skip the formatting entirely and talk to the API yourself:

kubectl get --raw '/api/v1/nodes?limit=1' | head -c 300; echo
kubectl get --raw '/readyz?verbose' | head -n 6

2.2 — Create a Pod with curl, no kubectl verbs at all

This is the Docker lab's "the command is only a messenger" moment, in Kubernetes form. kubectl proxy opens an unauthenticated local door to the API, handling the certificates for you.

Terminal 2:

kubectl proxy --port=8001

Starting to serve on 127.0.0.1:8001

Terminal 1 — create a Pod by posting JSON, exactly as kubectl would:

curl -s -X POST http://127.0.0.1:8001/api/v1/namespaces/default/pods \
  -H 'Content-Type: application/json' \
  -d '{
    "apiVersion": "v1",
    "kind": "Pod",
    "metadata": { "name": "hello-raw" },
    "spec": { "containers": [ { "name": "hello", "image": "nginx:stable-alpine" } ] }
  }' | head -c 200; echo

kubectl get pod hello-raw -o wide

NAME        READY   STATUS    RESTARTS   AGE   IP           NODE
hello-raw   1/1     Running   0          9s    10.244.1.3   k8s-lab-worker2

2.3 — The evidence, in order

kubectl get events --field-selector involvedObject.name=hello-raw --sort-by=.metadata.creationTimestamp

LAST SEEN   TYPE     REASON      OBJECT          MESSAGE
25s         Normal   Scheduled   pod/hello-raw   Successfully assigned default/hello-raw to k8s-lab-worker2
24s         Normal   Pulling     pod/hello-raw   Pulling image "nginx:stable-alpine"
21s         Normal   Pulled      pod/hello-raw   Successfully pulled image in 2.9s
21s         Normal   Created     pod/hello-raw   Created container: hello
20s         Normal   Started     pod/hello-raw   Started container hello

That list is the handover, written down by the components themselves. Scheduled was the scheduler. Pulling / Pulled / Created / Started were the kubelet on worker 2, driving containerd. Five lines, two different machines, nobody coordinating them except through the API server.

The whole path, start to finish

2.4 — Who are you, and what are you allowed to do?

kubectl auth whoami
kubectl auth can-i create deployments
kubectl auth can-i delete nodes --as=system:serviceaccount:default:default

ATTRIBUTE   VALUE
Username    kubernetes-admin
Groups      [kubernetes:masters system:authenticated]

yes
no

You are cluster admin, which is why nothing has stopped you. That third command asks the same question as somebody else — a plain service account — and gets no. That's RBAC, step 3 of the path above, and it's how real clusters stop one team deleting another's work.

Tidy up before moving on. Delete the Pod, then press Ctrl+C in Terminal 2 to stop the proxy — step 4 reuses that terminal for something else:

kubectl delete pod hello-raw

curl -s -N 'http://127.0.0.1:8001/api/v1/namespaces/default/pods?watch=true' | head -n 3

You're going to run nginx — the same web server from the Docker lab. In Docker that was docker run -d nginx on one machine. Here you write the wish down and let the cluster decide where it goes.

3.1 — Write the manifest

nano deployment.yaml

Type this in. YAML uses spaces, never tabs — two spaces per level. If your editor inserts tabs, fix that now or every command for the next hour will fail.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:stable
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: "50m"
              memory: "64Mi"
            limits:
              cpu: "200m"
              memory: "128Mi"

Save with Ctrl+O, Enter, then exit with Ctrl+X.

What did you just write?

3.2 — Check it before you apply it

Get into this habit now; it costs one second and catches every indentation mistake:

kubectl apply --dry-run=client -f deployment.yaml

deployment.apps/nginx created (dry run)

Then do it for real, and ask when it's ready instead of guessing:

kubectl apply -f deployment.yaml
kubectl wait --for=condition=Available deployment/nginx --timeout=120s

deployment.apps/nginx created
deployment.apps/nginx condition met

3.3 — Where did they go?

kubectl get deployments
kubectl get replicasets
kubectl get pods -o wide

NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   3/3     3            3           25s

NAME               DESIRED   CURRENT   READY   AGE
nginx-7c9f4d8b6d   3         3         3       25s

NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE
nginx-7c9f4d8b6d-k2xvp   1/1     Running   0          25s   10.244.1.4   k8s-lab-worker
nginx-7c9f4d8b6d-m8ztq   1/1     Running   0          25s   10.244.2.3   k8s-lab-worker2
nginx-7c9f4d8b6d-p4wnc   1/1     Running   0          25s   10.244.1.5   k8s-lab-worker

Count them per machine:

kubectl get pods -l app=nginx -o wide --no-headers | awk '{print $7}' | sort | uniq -c

      2 k8s-lab-worker
      1 k8s-lab-worker2

How the scheduler actually chose

For every unassigned Pod, kube-scheduler runs two phases:

1. Filter — throw out every node that cannot work. Does the node have an untolerated taint? (Goodbye, control plane.) Does it have enough unreserved CPU and memory for this Pod's requests? Does it match any nodeSelector or affinity rule? Is it Ready and unschedulable-free?
2. Score — rank the survivors out of 100 each. Favour the node with more free resources (LeastAllocated), the one that already has the image cached (ImageLocality), and the one that keeps Pods of the same app spread apart (PodTopologySpread).

Highest score wins; ties are broken at random. Then it binds the Pod to that node — which is nothing more than writing a nodeName field back through the API server. The kubelet on that node takes it from there.

Read the decision in writing:

kubectl describe pod -l app=nginx | grep -E 'Node:|Successfully assigned' | head -n 6

3.4 — Read the names: the whole story is in them

Deployment:   nginx
ReplicaSet:   nginx-7c9f4d8b6d
Pod:          nginx-7c9f4d8b6d-k2xvp

You created one object. You now have four layers, each stamping its name onto the next:

• Deployment — handles versions and upgrades. Step 6 is entirely about this.
• ReplicaSet — handles how many. Step 5 is entirely about this.
• Pod — the smallest unit the scheduler places. It owns the IP address and belongs to exactly one node.
• Container — the actual nginx process, started by containerd, exactly like the Docker lab.

3.5 — Every object keeps a diary

kubectl describe deployment nginx | tail -n 12

That Events block lists, in plain English, everything that has happened to this object. When something breaks later today, this is the first place you look — not Google.

kubectl delete replicaset -l app=nginx
kubectl get replicasets

kubectl patch deployment nginx --type merge -p \
  '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/hostname":"k8s-lab-worker2"}}}}}'
kubectl rollout status deployment/nginx
kubectl get pods -l app=nginx -o wide --no-headers | awk '{print $7}' | sort | uniq -c

kubectl patch deployment nginx --type json \
  -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
kubectl rollout status deployment/nginx
kubectl get pods -l app=nginx -o wide --no-headers | awk '{print $7}' | sort | uniq -c

4.1 — Why you need this

kubectl get pods -o wide

Note a Pod IP — something like 10.244.1.4. That address belongs to that one Pod, on that one node, and it vanishes the moment the Pod does — which you'll prove in step 5. You cannot build anything on an address that disappears. That is what a Service is for.

4.2 — The Service

nano service.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 80

Notice apiVersion: v1 here, versus apps/v1 in the Deployment. Services are part of the original core API; Deployments were added later, in the apps group. Copying the wrong one is a classic beginner error with a confusing message.

kubectl apply -f service.yaml
kubectl get services
kubectl get endpoints nginx

service/nginx created

NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP   31m
nginx        ClusterIP   10.96.142.88   <none>        80/TCP    4s

NAME    ENDPOINTS                                 AGE
nginx   10.244.1.4:80,10.244.1.5:80,10.244.2.3:80   4s

Endpoints are a legacy view. The object Kubernetes actually maintains now is an EndpointSlice, and it carries more — including which node each Pod is on:

kubectl get endpointslices -l kubernetes.io/service-name=nginx \
  -o custom-columns='SLICE:.metadata.name,ADDRS:.endpoints[*].addresses,NODES:.endpoints[*].nodeName'

SLICE         ADDRS                                 NODES
nginx-x7k2m   10.244.1.4,10.244.1.5,10.244.2.3     k8s-lab-worker,k8s-lab-worker,k8s-lab-worker2

The endpoints controller wrote that, by watching Pods and matching labels. Nobody maintains this list by hand, and nothing you did in service.yaml mentioned a Pod.

4.3 — Reach it from inside the cluster (DNS)

kubectl run tmp-shell --rm -it --image=curlimages/curl --restart=Never -- \
  sh -c 'curl -s http://nginx | head -n 4; echo; nslookup nginx 2>/dev/null | tail -n 4'

Plain http://nginx worked. That's coredns, one of the Pods from step 1: every Service automatically gets the DNS name <service>.<namespace>.svc.cluster.local, and inside the same namespace the short name is enough. This is how microservices find each other — no config files, no IP addresses, just names.

4.4 — Way one: port-forward (the developer's tunnel)

Terminal 2 — the one you just freed up. Start this and leave it running:

kubectl port-forward service/nginx 8080:80

Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

The command looks frozen. That's correct — it's holding a tunnel open until you press Ctrl+C. Open http://localhost:8080 in your browser and you get the "Welcome to nginx!" page. 🎉 Reload a few times and watch Terminal 2 log a Handling connection line each time.

Leave it up. You're about to get a second route to the same app, and comparing the two is the point of this step.

4.5 — Way two: a NodePort on a real port

nano nodeport.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-public
  labels:
    app: nginx
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
    - name: http
      protocol: TCP
      port: 80          # the Service's own port, inside the cluster
      targetPort: 80    # the port inside the container
      nodePort: 30080   # the port opened on EVERY node

kubectl apply -f nodeport.yaml
kubectl get services

NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes     ClusterIP   10.96.0.1       <none>        443/TCP        38m
nginx          ClusterIP   10.96.142.88    <none>        80/TCP         6m
nginx-public   NodePort    10.96.201.17    <none>        80:30080/TCP   3s

Now open your normal browser on Windows — Chrome or Edge, not anything inside Ubuntu:

http://localhost:30080

The three ports, which everybody mixes up

4.6 — Make the Pods tell you apart

Every nginx serves the same welcome page, so you cannot tell which one answered. Fix that: give each Pod a page with its own name on it.

for p in $(kubectl get pods -l app=nginx -o jsonpath='{.items[*].metadata.name}'); do
  node=$(kubectl get pod $p -o jsonpath='{.spec.nodeName}')
  kubectl exec $p -- sh -c "echo '$p on $node' > /usr/share/nginx/html/index.html"
done
curl -s http://localhost:30080

nginx-7c9f4d8b6d-m8ztq on k8s-lab-worker2

4.7 — Both doors at once, and they don't behave the same

You now have two live routes to the identical Service. Hit each one twelve times and count who answered:

echo "--- NodePort, localhost:30080 ---"
for i in $(seq 1 12); do curl -s http://localhost:30080; done | sort | uniq -c

echo "--- port-forward, localhost:8080 ---"
for i in $(seq 1 12); do curl -s http://localhost:8080; done | sort | uniq -c

--- NodePort, localhost:30080 ---
      5 nginx-7c9f4d8b6d-k2xvp on k8s-lab-worker
      3 nginx-7c9f4d8b6d-m8ztq on k8s-lab-worker2
      4 nginx-7c9f4d8b6d-p4wnc on k8s-lab-worker

--- port-forward, localhost:8080 ---
     12 nginx-7c9f4d8b6d-m8ztq on k8s-lab-worker2

Which one should you use?

kubectl patch service nginx-public -p '{"spec":{"externalTrafficPolicy":"Local"}}'
curl -s --max-time 5 http://localhost:30080 || echo ">>> no answer - no local Pods on that node"
kubectl patch service nginx-public -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'
curl -s http://localhost:30080

5.1 — Scale up

Terminal 1:

kubectl scale deployment nginx --replicas=5
kubectl get pods -l app=nginx -o wide --no-headers | awk '{print $7}' | sort | uniq -c
kubectl get endpoints nginx

deployment.apps/nginx scaled

      3 k8s-lab-worker
      2 k8s-lab-worker2

NAME    ENDPOINTS                                                        AGE
nginx   10.244.1.4:80,10.244.1.5:80,10.244.1.6:80 + 2 more...            14m

The existing three Pods were not restarted — Kubernetes created only what was missing, and the scheduler placed the new ones where there was room. And you never touched either Service: they match by label, so any new Pod wearing app=nginx joins the pool the moment it's ready.

Go back to three, so the next part is easier to read:

kubectl scale deployment nginx --replicas=3

5.2 — Delete a Pod on purpose

Terminal 3 — start watching and leave it:

kubectl get pods -o wide -w

Terminal 1 — copy a real Pod name from your output, not from this page:

kubectl get pods -o wide
kubectl delete pod <paste-one-of-your-nginx-pod-names>

nginx-7c9f4d8b6d-k2xvp   1/1   Terminating         0   18m   10.244.1.4   k8s-lab-worker
nginx-7c9f4d8b6d-z9rtb   0/1   Pending             0   0s    <none>       <none>
nginx-7c9f4d8b6d-z9rtb   0/1   ContainerCreating   0   0s    <none>       k8s-lab-worker2
nginx-7c9f4d8b6d-z9rtb   1/1   Running             0   2s    10.244.2.4   k8s-lab-worker2

A replacement appeared in about two seconds, with a new name, a new IP — and look at the NODE column: possibly a different machine entirely. Notice the Pending line with <none> for the node: that is the half-second window where the Pod object existed but the scheduler hadn't chosen yet. That's step 2's pipeline, caught in the act.

Press Ctrl+C in Terminal 3. Now, back in Terminal 1, ask who is answering:

for i in $(seq 1 10); do curl -s http://localhost:30080 | head -n 1; done | sort | uniq -c

      3 <!DOCTYPE html>
      4 nginx-7c9f4d8b6d-m8ztq on k8s-lab-worker2
      3 nginx-7c9f4d8b6d-p4wnc on k8s-lab-worker

Now glance at Terminal 2. If the Pod you deleted happened to be the one your port-forward latched onto in step 4, it has died:

E1016 12:14:03.882  portforward.go:409] an error occurred forwarding 8080 -> 80:
  lost connection to pod

Meanwhile http://localhost:30080 never flinched. That is the table from step 4.7 happening to you: the NodePort follows the Service, the tunnel was pinned to one Pod. Restart it and it picks a new Pod:

# Terminal 2
kubectl port-forward service/nginx 8080:80

What just happened, precisely

This is the reconciliation loop, running right now, every few seconds, forever:

1. Observe — the ReplicaSet controller counts Pods labelled app=nginx: two.
2. Compare — desired is three.
3. Act — create one more Pod object from the template, with no node set.
4. Hand off — the scheduler picks a node; the kubelet there starts it; the endpoints controller adds its IP to both Services.
5. Repeat.

The new Pod is a different Pod, not the old one revived. New name, new IP, possibly a new machine, no memory of its predecessor. This is what people mean by "cattle, not pets" — and exactly why you must never hard-code a Pod IP anywhere.

5.3 — Now take a whole machine out of service

This is the part you could not do on a one-node cluster, and it is the single most common real maintenance task in the world: "I need to reboot that server."

First, see where everything is:

kubectl get pods -o wide

Then evict everything from one worker, in one command:

kubectl drain k8s-lab-worker2 --ignore-daemonsets --delete-emptydir-data

node/k8s-lab-worker2 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/kindnet-mn4rt, kube-system/kube-proxy-vz6kp
evicting pod default/nginx-7c9f4d8b6d-z9rtb
pod/nginx-7c9f4d8b6d-z9rtb evicted
node/k8s-lab-worker2 drained

kubectl get nodes
kubectl get pods -o wide
curl -s http://localhost:30080 | head -n 1

NAME                    STATUS                     ROLES           AGE   VERSION
k8s-lab-control-plane   Ready                      control-plane   48m   v1.34.0
k8s-lab-worker          Ready                      worker          47m   v1.34.0
k8s-lab-worker2         Ready,SchedulingDisabled   worker          47m   v1.34.0

NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE
nginx-7c9f4d8b6d-p4wnc   1/1     Running   0          22m   10.244.1.5   k8s-lab-worker
nginx-7c9f4d8b6d-m8ztq   1/1     Running   0          22m   10.244.1.7   k8s-lab-worker
nginx-7c9f4d8b6d-t8kqn   1/1     Running   0          18s   10.244.1.8   k8s-lab-worker

nginx-7c9f4d8b6d-p4wnc on k8s-lab-worker

Now put the node back to work:

kubectl uncordon k8s-lab-worker2
kubectl get nodes
kubectl get pods -o wide

To spread out again, give the Pods a reason to be reborn:

kubectl rollout restart deployment/nginx
kubectl rollout status deployment/nginx
kubectl get pods -l app=nginx -o wide --no-headers | awk '{print $7}' | sort | uniq -c

rollout restart replaces every Pod one at a time, with no downtime — the same machinery as step 6's image update, just with the same image. It's the standard way to pick up a changed secret, re-balance after maintenance, or simply "turn it off and on again" safely.

kubectl create poddisruptionbudget nginx-pdb --selector=app=nginx --min-available=2
kubectl get pdb
kubectl drain k8s-lab-worker --ignore-daemonsets --delete-emptydir-data
kubectl get pods -o wide

kubectl uncordon k8s-lab-worker
kubectl delete pdb nginx-pdb
kubectl rollout restart deployment/nginx
kubectl rollout status deployment/nginx

kubectl scale deployment nginx --replicas=0
kubectl get pods
kubectl get deployment nginx
kubectl get endpoints nginx
curl -s --max-time 5 http://localhost:30080 || echo ">>> nothing left to serve it"

kubectl scale deployment nginx --replicas=3
kubectl rollout status deployment/nginx

You'll move from nginx:stable to nginx:stable-alpine — the same nginx release built on Alpine Linux, so a much smaller image. Both are official, maintained tags, so this works today and will still work when you repeat the lab next month.

6.1 — Write down what you have now

kubectl get replicasets
kubectl get deployment nginx -o jsonpath='{.spec.template.spec.containers[0].image}'; echo
kubectl get deployment nginx -o jsonpath='{.spec.strategy}'; echo

NAME               DESIRED   CURRENT   READY   AGE
nginx-7c9f4d8b6d   3         3         3       32m

nginx:stable

{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"}

6.2 — Measure the downtime while it rolls

Terminal 3 — start hammering the NodePort, and leave it running:

for i in $(seq 1 300); do
  curl -s -o /dev/null -w "%{http_code}\n" --max-time 2 http://localhost:30080
  sleep 0.1
done | sort | uniq -c

Terminal 1 — immediately roll out the new image:

kubectl set image deployment/nginx nginx=nginx:stable-alpine
kubectl rollout status deployment/nginx

Waiting for deployment "nginx" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx" rollout to finish: 1 old replicas are pending termination...
deployment "nginx" successfully rolled out

When Terminal 3 finishes, read its summary:

    300 200

6.3 — See what changed

kubectl get replicasets

NAME               DESIRED   CURRENT   READY   AGE
nginx-5d7b9c4f8a   3         3         3       40s
nginx-7c9f4d8b6d   0         0         0       35m

6.4 — Read the history

kubectl rollout history deployment/nginx

REVISION  CHANGE-CAUSE
1         <none>
2         <none>
3         <none>

CHANGE-CAUSE is empty because nobody wrote down why. On a real team you would:

kubectl annotate deployment/nginx kubernetes.io/change-cause="switch to alpine base image"
kubectl rollout history deployment/nginx --revision=3

That second command prints the full Pod template for one revision — image tag, resources, labels, everything.

6.5 — Undo it

kubectl rollout undo deployment/nginx
kubectl rollout status deployment/nginx
kubectl get deployment nginx -o jsonpath='{.spec.template.spec.containers[0].image}'; echo
kubectl get endpoints nginx

Back to nginx:stable, 3/3, three endpoints. Reload http://localhost:30080 once more.

Why that was instant

A rolling update is not an edit-in-place. Kubernetes:

1. Creates a new ReplicaSet for the new Pod template — hence the new hash.
2. Scales the new one up, within maxSurge, and waits for those Pods to be ready.
3. Scales the old one down, never breaching maxUnavailable.
4. Repeats until the old one is at zero.

rollout undo runs that in reverse. Nothing is rebuilt and nothing is re-downloaded, because the old ReplicaSet object was never deleted — which is why rollback takes two seconds instead of twenty minutes. And the information came from etcd, not from a file on your laptop: a classmate with cluster access could roll your release back from their machine.

kubectl patch deployment nginx -p \
  '{"spec":{"strategy":{"rollingUpdate":{"maxUnavailable":0,"maxSurge":1}}}}'

kubectl set image deployment/nginx nginx=nginx:stable-alpine
kubectl rollout pause deployment/nginx
kubectl get pods -l app=nginx -o custom-columns='POD:.metadata.name,IMAGE:.spec.containers[0].image'
# ... check your dashboards, your logs, your error rate ...
kubectl rollout resume deployment/nginx      # happy? finish the rollout
kubectl rollout undo deployment/nginx        # unhappy? back out, having risked one Pod

kubectl patch deployment nginx -p \
  '{"spec":{"strategy":{"rollingUpdate":{"maxUnavailable":"25%","maxSurge":"25%"}}}}'
kubectl rollout status deployment/nginx

kubectl rollout history deployment/nginx
kubectl rollout undo deployment/nginx --to-revision=1
kubectl get deployment nginx -o jsonpath='{.spec.template.spec.containers[0].image}'; echo

Step 6 was a new version: the image tag changed, so the Pod template changed, so Kubernetes had something to react to. Most real redeployments aren't like that. You pushed a rebuilt image under the same tag, or a config value changed, or a Pod is wedged and you just want it replaced. Nothing in your YAML is different — and that is exactly the problem.

7.1 — Apply the same file again and watch nothing happen

kubectl apply -f deployment.yaml

deployment.apps/nginx unchanged

7.2 — The same-tag trap

Say you rebuild and push myapp:latest with a bug fix. Your manifest still says image: myapp:latest, so apply does nothing. Fine — you force Pods to be recreated. They may still run the old code. Here's why:

kubectl get deployment nginx -o jsonpath='{.spec.template.spec.containers[0].imagePullPolicy}'; echo

IfNotPresent

7.3 — The right way to force a redeploy

Before you run anything, take a snapshot. A redeploy is only interesting if you can see what moved:

kubectl get pods -l app=nginx -o wide
kubectl get replicasets
kubectl get deployment nginx -o jsonpath='{.spec.template.metadata.annotations}'; echo "  <-- annotations"
kubectl get pods -l app=nginx -o name | sort > ~/k8s-lab/pods-before.txt

NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE
nginx-7c9f4d8b6d-m8ztq   1/1     Running   0          41m   10.244.2.3   k8s-lab-worker2
nginx-7c9f4d8b6d-p4wnc   1/1     Running   0          41m   10.244.1.5   k8s-lab-worker
nginx-7c9f4d8b6d-t8kqn   1/1     Running   0          23m   10.244.1.8   k8s-lab-worker

NAME               DESIRED   CURRENT   READY   AGE
nginx-7c9f4d8b6d   3         3         3       41m

  <-- annotations

One ReplicaSet, three Pods up to forty-one minutes old, and no annotations at all. Now redeploy:

kubectl rollout restart deployment/nginx
kubectl rollout status deployment/nginx

deployment.apps/nginx restarted
Waiting for deployment "nginx" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx" rollout to finish: 2 old replicas are pending termination...
deployment "nginx" successfully rolled out

And take the same three readings again:

kubectl get pods -l app=nginx -o wide
kubectl get replicasets
kubectl get deployment nginx -o jsonpath='{.spec.template.metadata.annotations}'; echo

NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE
nginx-8a3e1c7d25-6rzv2   1/1     Running   0          22s   10.244.1.11   k8s-lab-worker
nginx-8a3e1c7d25-hq4nd   1/1     Running   0          18s   10.244.2.7    k8s-lab-worker2
nginx-8a3e1c7d25-xp9ck   1/1     Running   0          14s   10.244.1.12   k8s-lab-worker

NAME               DESIRED   CURRENT   READY   AGE
nginx-8a3e1c7d25   3         3         3       22s
nginx-7c9f4d8b6d   0         0         0       42m

{"kubectl.kubernetes.io/restartedAt":"2026-10-16T12:33:10Z"}

Prove every single Pod was replaced, rather than squinting at the names:

kubectl get pods -l app=nginx -o name | sort > ~/k8s-lab/pods-after.txt
diff ~/k8s-lab/pods-before.txt ~/k8s-lab/pods-after.txt

1,3c1,3
< pod/nginx-7c9f4d8b6d-m8ztq
< pod/nginx-7c9f4d8b6d-p4wnc
< pod/nginx-7c9f4d8b6d-t8kqn
---
> pod/nginx-8a3e1c7d25-6rzv2
> pod/nginx-8a3e1c7d25-hq4nd
> pod/nginx-8a3e1c7d25-xp9ck

Three out, three in. Not one name survived.

What changed, and what didn't

Want to see the whole thing happen rather than its before and after? Run the watch in Terminal 3 first, then restart again from Terminal 1:

# Terminal 3
kubectl get pods -l app=nginx -w

nginx-8a3e1c7d25-6rzv2   1/1   Running             0     3m
nginx-4d2b8e6f03-k7t2w   0/1   Pending             0     0s     <-- new Pod created first
nginx-4d2b8e6f03-k7t2w   0/1   ContainerCreating   0     0s
nginx-4d2b8e6f03-k7t2w   1/1   Running             0     2s     <-- it is ready...
nginx-8a3e1c7d25-6rzv2   1/1   Terminating         0     3m     <-- ...only THEN is an old one removed
nginx-4d2b8e6f03-9wscl   0/1   Pending             0     0s     <-- and the next pair begins

Read the order of those lines. Ready-then-terminate, ready-then-terminate, one pair at a time. That ordering is the entire reason the next section's measurement comes out the way it does.

7.4 — The blunt way, and the price of it

Plenty of people "redeploy" by deleting Pods and letting the controller rebuild them. Let's measure what that costs.

Terminal 3 — start the probe loop:

for i in $(seq 1 120); do
  curl -s -o /dev/null -w "%{http_code}\n" --max-time 2 http://localhost:30080
  sleep 0.1
done | sort | uniq -c

Terminal 1 — delete every Pod at once:

kubectl delete pod -l app=nginx

    103 200
     17 000

Deleting one Pod is a different matter and perfectly reasonable: the other two keep serving while the replacement starts, exactly as in step 5. It's deleting the whole set that bites.

So which one do I actually run?

Put yourself back in a good state before moving on:

kubectl rollout status deployment/nginx
kubectl get pods -l app=nginx -o wide
kubectl get endpoints nginx

docker exec k8s-lab-worker crictl images | head -n 6

IMAGE                      TAG            SIZE
docker.io/library/nginx    stable         75.2MB
registry.k8s.io/kube-proxy v1.34.0        30.3MB
...

# push it to a registry the nodes can reach, the real-world answer
docker push myregistry.example.com/notice-board:v1

# or side-load it straight into every kind node, the lab answer
kind load docker-image notice-board:v1 --name k8s-lab

kubectl patch deployment nginx -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}'
kubectl rollout restart deployment/nginx
kubectl get pods -l app=nginx -w     # Ctrl+C after ~20 seconds

kubectl patch deployment nginx -p \
  '{"spec":{"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"}}}}'
kubectl rollout status deployment/nginx

So far, "the container started" has meant "send it traffic". For nginx that's nearly true. For a real application — one that loads a model, warms a cache, or opens thirty database connections — it is dangerously false. It can take twenty seconds to be useful, and for those twenty seconds Kubernetes would cheerfully route users to it and they would all get errors.

Probes are how you tell the kubelet what "ready" and "alive" actually mean for your app.

8.1 — Add probes to your manifest

nano deployment.yaml

Add these two blocks inside the container, at the same indentation as resources:

          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 20
            periodSeconds: 10
            failureThreshold: 3

8.2 — Apply it and watch the gap

Terminal 3 — watch:

kubectl get pods -l app=nginx -w

Terminal 1:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/nginx

nginx-6f8c2a91b4-w4ltp   0/1   Pending             0   0s
nginx-6f8c2a91b4-w4ltp   0/1   ContainerCreating   0   0s
nginx-6f8c2a91b4-w4ltp   0/1   Running             0   2s    <-- running, but NOT ready
nginx-6f8c2a91b4-w4ltp   1/1   Running             0   13s   <-- readiness probe passed

Catch it in the act — run this immediately after a restart:

kubectl rollout restart deployment/nginx
sleep 4
kubectl get pods -l app=nginx
kubectl get endpoints nginx

You'll see more Pods than endpoints. The unready ones exist, cost memory, and are invisible to users. Wait fifteen seconds, run it again, and the numbers match.

8.3 — Break the readiness probe

Point it at a port nothing is listening on:

kubectl patch deployment nginx --type json -p \
  '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":9999}]'
sleep 30
kubectl get pods -l app=nginx
kubectl get endpoints nginx
curl -s http://localhost:30080 | head -n 1

NAME                     READY   STATUS    RESTARTS   AGE
nginx-6f8c2a91b4-w4ltp   1/1     Running   0          4m
nginx-6f8c2a91b4-x2dqk   1/1     Running   0          4m
nginx-84b7d5e3f1-j9vmr   0/1     Running   0          32s     <-- never going to be ready

NAME    ENDPOINTS                     AGE
nginx   10.244.1.9:80,10.244.2.6:80   58m

<!DOCTYPE html>

Now read the reason in the Pod's own words:

kubectl describe pod -l app=nginx | grep -A3 'Readiness probe failed' | head -n 8

Warning  Unhealthy  12s (x5 over 40s)  kubelet  Readiness probe failed:
  Get "http://10.244.1.10:9999/": dial tcp 10.244.1.10:9999: connect: connection refused

Repair it the way you already know:

kubectl rollout undo deployment/nginx
kubectl rollout status deployment/nginx
kubectl get pods -l app=nginx
kubectl get endpoints nginx

kubectl patch deployment nginx --type json -p \
  '[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/port","value":9999}]'
kubectl get pods -l app=nginx -w     # watch for ~90 seconds, then Ctrl+C

nginx-9c1f72a84e-lq8sv   0/1   Running              1 (5s ago)    52s
nginx-9c1f72a84e-lq8sv   0/1   Running              2 (31s ago)   1m22s
nginx-9c1f72a84e-lq8sv   0/1   CrashLoopBackOff     2 (18s ago)   1m48s

kubectl rollout undo deployment/nginx
kubectl rollout status deployment/nginx

9.1 — Break it

kubectl set image deployment/nginx nginx=nginx:this-tag-does-not-exist

Wait about twenty seconds, then:

kubectl get pods -o wide
curl -s http://localhost:30080 | head -n 1

NAME                     READY   STATUS             RESTARTS   AGE   NODE
nginx-7c9f4d8b6d-m8ztq   1/1     Running            0          25m   k8s-lab-worker
nginx-7c9f4d8b6d-p4wnc   1/1     Running            0          25m   k8s-lab-worker2
nginx-6b4d9f2c1e-t5qmx   0/1     ImagePullBackOff   0          22s   k8s-lab-worker

<!DOCTYPE html>

9.2 — Diagnose it: the standard five

Run these in order, every time, for the rest of your career. Use your broken Pod's name:

kubectl get pods -o wide                              # 1. what's the status, and where?
kubectl describe pod <broken-pod-name>                # 2. why is THIS one unhappy?
kubectl logs <broken-pod-name>                        # 3. what does the app itself say?
kubectl get events --sort-by=.metadata.creationTimestamp | tail   # 4. what's been happening?
kubectl get endpoints nginx                           # 5. is traffic still being served?

Scroll to the Events block at the bottom of describe pod:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  45s   default-scheduler  Successfully assigned default/nginx-6b4d9f2c1e-t5qmx to k8s-lab-worker
  Normal   Pulling    44s   kubelet            Pulling image "nginx:this-tag-does-not-exist"
  Warning  Failed     42s   kubelet            Failed to pull image "nginx:this-tag-does-not-exist": not found
  Warning  Failed     42s   kubelet            Error: ErrImagePull
  Warning  Failed     15s   kubelet            Error: ImagePullBackOff

There's your answer, in plain English: the image tag doesn't exist. No searching required. And read the From column — default-scheduler succeeded, kubelet failed. You know which component to blame and which node it was on.

Now look at what command 3 gave you

Error from server (BadRequest): container "nginx" in pod "nginx-6b4d9f2c1e-t5qmx" is
waiting to start: trying and failing to pull image

9.3 — Repair it, and finish healthy

kubectl rollout undo deployment/nginx
kubectl rollout status deployment/nginx
kubectl get deployments
kubectl get pods -o wide
kubectl get endpoints nginx
kubectl get deployment nginx -o jsonpath='{.spec.template.spec.containers[0].image}'; echo

You must end with: READY 3/3, three Pods Running across your two workers, three endpoints, image nginx:stable, and http://localhost:30080 still serving.

kubectl apply -f deployment.yaml
kubectl rollout status deployment/nginx

kubectl delete deployment nginx
kubectl delete service nginx nginx-public
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f nodeport.yaml
kubectl wait --for=condition=Available deployment/nginx --timeout=120s

Status → cause → first command

The four questions, in order

1. Is it scheduled? Pending → the scheduler could not place it. Resources, taints, selectors.
2. Did the image pull? ImagePullBackOff → name, tag or registry credentials.
3. Did the process start and stay up? CrashLoopBackOff → read the logs; it's your app or your probe.
4. Is it reachable? Running but unreachable → get endpoints and check your labels.

kubectl patch deployment nginx --type json -p \
  '[{"op":"replace","path":"/spec/template/spec/containers/0/resources/requests/memory","value":"64Gi"}]'
sleep 10
kubectl get pods -l app=nginx
kubectl describe pod -l app=nginx | grep -A6 'Events:' | tail -n 5

0/3 nodes are available: 1 node(s) had untolerated taint
{node-role.kubernetes.io/control-plane: }, 2 Insufficient memory.
preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

kubectl rollout undo deployment/nginx
kubectl rollout status deployment/nginx

Commands worth keeping

If something goes wrong

kubectl delete deployment nginx --ignore-not-found
kubectl delete service nginx nginx-public --ignore-not-found
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f nodeport.yaml
kubectl wait --for=condition=Available deployment/nginx --timeout=120s
kubectl uncordon k8s-lab-worker k8s-lab-worker2 2>/dev/null; true

Check yourself

Answer out loud, in your own words, before opening each one.

Option A — keep the cluster, remove the app

# stop the port-forward first: Ctrl+C in Terminal 2
kubectl delete -f nodeport.yaml
kubectl delete -f service.yaml
kubectl delete -f deployment.yaml
kubectl get all

Only the built-in kubernetes Service should remain. Note that you never deleted a Pod or a ReplicaSet — deleting the Deployment removed everything it owns, through ownership links. That's the chain from step 3, working in reverse.

Option B — delete the whole cluster

kind delete cluster --name k8s-lab
kind get clusters      # prints nothing
docker ps -a           # all three node containers are gone

This destroys the three Docker containers that were your nodes, and releases all fifty-one published ports. Everything inside is gone permanently. Your kind-config.yaml, deployment.yaml, service.yaml and nodeport.yaml on disk are untouched, so you can rebuild the entire lab in about four minutes.

Same cluster, same tools, a new app — plus three things we didn't have room for in the session. You'll deploy a small web service called Notice Board, publish it on your own NodePort, and prove it survives a node going away.

If you did the Docker lab, use the image you built and pushed. If not, any small web image works — httpd:alpine and nginx:stable-alpine both serve a page on port 80.

The checklist

Submit: your YAML files, plus a short document with the terminal output proving each of R1–R10. Screenshots are fine; pasted text is better, because we can read it.

Bonus marks

• S1 — add a PodDisruptionBudget with minAvailable: 3, then drain a worker and show, with timestamps, that evictions happened one at a time instead of all at once.
• S2 — move a setting out of your image into a ConfigMap and mount it as an environment variable, then rollout restart to pick up a change to it.
• S3 — set maxUnavailable: 0 and maxSurge: 1, roll out again, and show with kubectl get pods -w that a fifth Pod appears before any of the four is removed.
• S4 — create the whole Deployment through the REST API with curl and kubectl proxy, as in step 2.2, with no kubectl apply anywhere.
• S5 — rebuild your image under the same tag, kind load it again, and show that kubectl apply reports unchanged while kubectl rollout restart actually picks up the new code. State which imagePullPolicy you used and why that choice was forced by how you got the image onto the nodes.