Step 1 — PXE Network Boot Cluster Infrastructure

1. Scope and hardware architecture

This task documents the complete setup of a PXE network boot cluster for eight Raspberry Pi 3 worker nodes, one Raspberry Pi 5 server, and one Raspberry Pi 4 AI camera node. The goal is a diskless infrastructure where every Pi 3 boots its OS over the network from the Pi 5 — no SD cards required — enabling consistent OS management and reproducible cluster deployments.

Device	Model	Role	IP Address
RPi 5 (8GB)	Raspberry Pi 5	PXE Server	`192.168.10.1`
RPi 4	Raspberry Pi 4	AI Camera Node	`192.168.10.109`
RPi 3 Nodes 1–8	RPi 3B / 3B+	PXE Clients	`192.168.10.101–108`
Switch	TP-Link TL-SG1024DE	24-port Layer 2	`192.168.10.100`

2. Network Topology

Internet

↓

Home Router (Fritz!Box)

192.168.178.100

↓

RPi 5 — PXE Server

wlan0: 192.168.178.x (internet) • eth0: 192.168.10.1 (cluster)

↓

TP-Link Switch

192.168.10.100

↓

rpi3-node1

192.168.10.101

rpi3-node2

192.168.10.102

rpi3-node3–7

192.168.10.103–107

rpi3-node8

192.168.10.108

RPi 4 Camera

192.168.10.109

Boot sequence: RPi 3 powers on → DHCP Discover → dnsmasq assigns IP + TFTP address → node downloads bootcode.bin + kernel via TFTP → kernel mounts /nfs/rootfs via NFS → OS boots. No SD card involved at any point.

3. RPi 5: Static IP on Ethernet

The ethernet port needs a fixed IP so dnsmasq and NFS always run on a known address. WiFi stays as the default route for internet access.

Set static IP via NetworkManager

sudo nmcli connection modify cluster-eth0 ipv4.addresses 192.168.10.1/24
sudo nmcli connection modify cluster-eth0 ipv4.method manual
sudo nmcli connection modify cluster-eth0 ipv4.gateway ""
sudo nmcli connection modify cluster-eth0 ipv4.never-default yes
sudo nmcli connection up cluster-eth0
ip addr show eth0
# Should show: inet 192.168.10.1/24

Lock DNS

echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf
sudo chattr +i /etc/resolv.conf

4. Install & Configure dnsmasq

dnsmasq handles DHCP (IP assignment), TFTP (boot files), and DNS (hostname resolution) in one lightweight daemon.

sudo apt install -y dnsmasq
sudo systemctl stop dnsmasq
sudo mkdir -p /tftpboot && sudo chmod 777 /tftpboot

File: `/etc/dnsmasq.conf`

interface=eth0
bind-interfaces

# DHCP pool
dhcp-range=192.168.10.100,192.168.10.150,infinite

# TFTP
enable-tftp
tftp-root=/tftpboot
dhcp-boot=bootcode.bin

# PXE
dhcp-vendorclass=set:rpiboot,PXEClient:Arch:00000:UNDI:002001
dhcp-option=tag:rpiboot,vendor:PXEClient,43,"Raspberry Pi Boot"
pxe-service=tag:rpiboot,0,"Raspberry Pi Boot"

# NFS root path sent to kernel
dhcp-option=17,"192.168.10.1:/nfs/rootfs"

# Fixed IPs per node (MAC reservations)
dhcp-host=B8:27:EB:AC:29:31,rpi3-node1,192.168.10.101
dhcp-host=B8:27:EB:E4:8E:29,rpi3-node2,192.168.10.102
# ... add one line per node

# DNS hostname resolution
address=/rpi3-node1/192.168.10.101
address=/rpi3-node2/192.168.10.102
# ... add one line per node

Fix Boot Ordering — wait for eth0

sudo mkdir -p /etc/systemd/system/dnsmasq.service.d
sudo tee /etc/systemd/system/dnsmasq.service.d/wait-for-eth0.conf <<EOF
[Unit]
After=network-online.target sys-subsystem-net-devices-eth0.device
Wants=network-online.target sys-subsystem-net-devices-eth0.device
EOF
sudo systemctl daemon-reload
sudo systemctl enable dnsmasq && sudo systemctl start dnsmasq

5. Install & Configure NFS Server

sudo apt install -y nfs-kernel-server
sudo mkdir -p /nfs/rootfs
sudo chown -R nobody:nogroup /nfs/

File: `/etc/exports`

/nfs/rootfs 192.168.10.0/24(rw,sync,no_subtree_check,no_root_squash)
/nfs/keys/rpi3-node1 192.168.10.101(ro,sync,no_subtree_check,no_root_squash)
# ... add one line per node for SSH keys

no_root_squash is critical

Without it, root on RPi 3 maps to nobody on the server — the kernel cannot mount root and the node fails to boot silently.

sudo exportfs -a
sudo systemctl enable nfs-kernel-server
sudo systemctl start nfs-kernel-server
showmount -e localhost

6. Prepare RPi 3 OS Image

Download RPi 3 compatible 64-bit image

cd /tmp
wget https://downloads.raspberrypi.org/raspios_lite_arm64/images/\
raspios_lite_arm64-2024-11-19/2024-11-19-raspios-bookworm-arm64-lite.img.xz
xz -d 2024-11-19-raspios-bookworm-arm64-lite.img.xz

# Mount partitions
sudo losetup -f --show --partscan /tmp/2024-11-19-raspios-bookworm-arm64-lite.img
# Note output e.g. /dev/loop0
sudo mkdir -p /mnt/rpi3boot /mnt/rpi3root
sudo mount /dev/loop0p1 /mnt/rpi3boot
sudo mount /dev/loop0p2 /mnt/rpi3root

# Copy to NFS and TFTP
sudo rm -rf /nfs/rootfs/*
sudo rsync -xa --progress /mnt/rpi3root/ /nfs/rootfs/
sudo cp -r /mnt/rpi3boot/* /tftpboot/

7. Configure Boot Files

`cmdline.txt` — must be one single line

sudo bash -c 'echo "console=serial0,115200 console=tty1 root=/dev/nfs \
nfsroot=192.168.10.1:/nfs/rootfs,vers=3 rw ip=dhcp rootwait" \
> /tftpboot/cmdline.txt'

Do NOT include proto=tcp

Older RPi 3 kernels reject it with "bad option proto" and refuse to mount NFS. NFSv3 uses TCP automatically without needing this flag.

fstab — remove SD card references

sudo tee /nfs/rootfs/etc/fstab <<'EOF'
proc            /proc           proc    defaults          0       0
tmpfs           /tmp            tmpfs   defaults          0       0
tmpfs           /var/log        tmpfs   defaults          0       0
tmpfs           /var/tmp        tmpfs   defaults          0       0
EOF

8. Enable SSH on All Nodes

Since all nodes share one rootfs, enabling SSH once applies to all current and future nodes permanently.

sudo ln -s /lib/systemd/system/ssh.service \
  /nfs/rootfs/etc/systemd/system/multi-user.target.wants/ssh.service

9. First Boot & Verification

Power on an RPi 3 (no SD card) and watch the logs on RPi 5:

sudo journalctl -u dnsmasq -f

Stage	Expected Log Message	Meaning
1. DHCP	`DHCPDISCOVER(eth0) b8:27:eb:...`	Node powered on
2. DHCP	`DHCPOFFER(eth0) 192.168.10.101`	IP offered
3. TFTP	`sent /tftpboot/bootcode.bin`	Bootloader delivered
4. TFTP	`sent /tftpboot/kernel7.img`	Kernel delivered
5. Kernel	`vendor class: Linux ipconfig`	Kernel booted
6. NFS	`root-path 192.168.10.1:/nfs/rootfs`	OS mounting from NFS

10. Critical findings and reproducibility traps

no_root_squash is required

Without it, root on RPi 3 maps to nobody on the server — the kernel cannot mount root and the node fails to boot silently. See §5 for the NFS export configuration.

RPi 3B+ boots from network by default

Unlike the RPi 3B, the 3B+ supports PXE boot out of the box — no OTP change is required. Simply remove the SD card and power on. See §11 for the OTP configuration details that apply only to the 3B variant.

11. Switch Setup & Adding Nodes

Enable PXE Boot on RPi 3B (one-time, irreversible)

# Boot from SD card first, then:
echo "program_usb_boot_mode=1" | sudo tee -a /boot/config.txt
sudo reboot
vcgencmd otp_dump | grep 17:
# Must show: 17:3020000a — then remove SD card

RPi 3B+

Supports network boot by default — no OTP change needed. Just remove the SD card and it boots from the network automatically.

Find new node MAC addresses

sudo journalctl -u dnsmasq -f | grep DHCPDISCOVER
# Shows MAC when new node powers on

sudo nmap -sn 192.168.10.0/24
# Shows all devices on cluster network

12. Unique Hostnames Per Node

All nodes share one rootfs so hostnames are set dynamically at boot by reading the MAC address.

File: `/nfs/rootfs/etc/node-hostnames`

b8:27:eb:ac:29:31 rpi3-node1
b8:27:eb:e4:8e:29 rpi3-node2
b8:27:eb:c2:f5:5c rpi3-node3
# ... one line per node

File: `/nfs/rootfs/etc/set-hostname.sh`

#!/bin/bash
MAC=$(cat /sys/class/net/eth0/address)
HOSTNAME=$(grep -i "$MAC" /etc/node-hostnames | awk '{print $2}')
if [ -n "$HOSTNAME" ]; then
    hostnamectl set-hostname "$HOSTNAME"
else
    HOSTNAME="rpi3-$(echo $MAC | tr -d ':' | tail -c 5)"
    hostnamectl set-hostname "$HOSTNAME"
fi
# Mount this node's unique SSH keys
mount -t nfs -o ro,vers=3 192.168.10.1:/nfs/keys/$HOSTNAME/etc/ssh /etc/ssh

sudo chmod +x /nfs/rootfs/etc/set-hostname.sh

13. Unique SSH Keys Per Node

Without unique keys, all nodes have identical SSH fingerprints — a security problem that also causes constant "host identification changed" warnings.

for node in rpi3-node1 rpi3-node2 rpi3-node3 rpi3-node4 \
            rpi3-node5 rpi3-node6 rpi3-node7 rpi3-node8; do
    sudo mkdir -p /nfs/keys/$node/etc/ssh
    sudo ssh-keygen -t ed25519 \
      -f /nfs/keys/$node/etc/ssh/ssh_host_ed25519_key -N ""
    sudo ssh-keygen -t rsa -b 2048 \
      -f /nfs/keys/$node/etc/ssh/ssh_host_rsa_key -N ""
    sudo ssh-keygen -t ecdsa \
      -f /nfs/keys/$node/etc/ssh/ssh_host_ecdsa_key -N ""
    # sshd_config MUST be copied — mount replaces entire /etc/ssh dir
    sudo cp /nfs/rootfs/etc/ssh/sshd_config /nfs/keys/$node/etc/ssh/
done
sudo exportfs -ra

Verify all nodes SSH correctly

for ip in 192.168.10.10{1..8}; do
    echo -n "$ip: "
    ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new \
        server-pi3@$ip "hostname" 2>/dev/null || echo "SSH failed"
done

14. RPi 4: AI Camera Setup

The RPi AI Camera which runs ML inference directly on the camera chip — the RPi 4 CPU receives only detection results, not raw frames.

# Install camera stack
sudo apt install -y python3-picamera2 imx500-all python3-opencv
sudo reboot
# First reboot takes 2–3 min — firmware flashing to sensor

# Verify camera detected
rpicam-hello --list-cameras
# Should show: 0 : imx500 [4056x3040 10-bit RGGB]

# Basic test — should run at ~30 FPS
rpicam-hello --timeout 5000

15. SD Card Backup

Clone the running RPi 5 SD card to a backup using a USB card reader. The result is a bit-for-bit copy — swap it into any RPi 5 and it boots identically.

lsblk
# Source: mmcblk0 (running SD), Destination: sda (blank SD in USB reader)
sudo umount /dev/sda1 2>/dev/null
sudo umount /dev/sda2 2>/dev/null
sudo dd if=/dev/mmcblk0 of=/dev/sda bs=4M status=progress conv=fsync
# Takes ~10 minutes for 32GB card

16. Adding a New Node — Quick Reference

Boot RPi 3B from SD, run echo "program_usb_boot_mode=1" >> /boot/config.txt && sudo reboot, then remove SD. (RPi 3B+ skips this step)
Plug into switch. Watch: sudo journalctl -u dnsmasq -f | grep DHCPDISCOVER — note the MAC address
Add to /etc/dnsmasq.conf: dhcp-host=MAC,rpi3-nodeX,192.168.10.10X and address=/rpi3-nodeX/192.168.10.10X
Add to /nfs/rootfs/etc/node-hostnames: mac rpi3-nodeX
Add to /etc/exports: /nfs/keys/rpi3-nodeX 192.168.10.10X(ro,sync,no_subtree_check,no_root_squash)
Generate SSH keys: sudo mkdir -p /nfs/keys/rpi3-nodeX/etc/ssh && sudo ssh-keygen -t ed25519 -f /nfs/keys/rpi3-nodeX/etc/ssh/ssh_host_ed25519_key -N "" && sudo cp /nfs/rootfs/etc/ssh/sshd_config /nfs/keys/rpi3-nodeX/etc/ssh/
Add 192.168.10.10X rpi3-nodeX to /nfs/rootfs/etc/hosts, /etc/hosts, and the 127.0.1.1 line
Apply: sudo exportfs -ra && sudo systemctl restart dnsmasq
Reboot node — it boots with correct hostname, IP, and SSH automatically

17. Key Files Reference

File	Purpose
`/etc/dnsmasq.conf`	DHCP pool, MAC reservations, TFTP config, PXE, DNS addresses
`/etc/exports`	NFS exports — rootfs and per-node SSH key folders
`/etc/NetworkManager/system-connections/cluster-eth0.nmconnection`	Static IP on eth0, managed via `nmcli`
`/tftpboot/cmdline.txt`	Kernel boot parameters — NFS root path, ip=dhcp
`/nfs/rootfs/`	Shared OS for all RPi 3 nodes (armhf Bookworm)
`/nfs/rootfs/etc/fstab`	Filesystem mounts on nodes — tmpfs only, no SD references
`/nfs/rootfs/etc/set-hostname.sh`	Sets hostname from MAC + mounts per-node SSH keys
`/nfs/rootfs/etc/node-hostnames`	MAC address → hostname mapping
`/nfs/rootfs/etc/hosts`	Hostname resolution for nodes themselves
`/nfs/keys/rpi3-nodeX/etc/ssh/`	Unique SSH host keys + sshd_config per node
`/etc/hosts` (RPi 5)	Hostname→IP mapping for SSH by name from server