> Rendered: https://z29k.github.io/rescriptum/guide/operations/netboot · Docs index: https://z29k.github.io/rescriptum/llms.txt # Netbooting a machine A machine powers on. Four links later it is installing itself the way somebody decided — or, if nobody has decided anything about it yet, sitting in a menu where a human can. ``` power on │ (1) ├── DHCP says where to boot from ......... THEIRS. Two options, and we │ generate the snippet that sets them. ▼ (2) ├── TFTP hands over a loader ............. OURS │ arch-matched iPXE, chaining through ${next-server} ▼ (3) ├── iPXE asks what to do ................. OURS │ known machine → its own unattended answer │ unknown machine → the menu ▼ (4) └── the bits arrive ..................... OURS kernel, initrd, the image itself — HTTP with ranges ``` **Link 1 is somebody else's and stays that way.** rescriptum speaks no DHCP at all — not as a server, not as a proxy, not behind a flag. Sites that deploy this already run one, and pointing it at a boot server is a solved problem with thirty years of tooling. ## Turning it on ```console $ export RESCRIPTUM_MEDIA_DIR=/srv/media # the images $ export RESCRIPTUM_BOOT_DIR=/srv/boot # the loaders $ export RESCRIPTUM_PUBLIC_HOST=192.0.2.10 # what generated scripts will name ``` `RESCRIPTUM_BOOT_DIR` says where the loaders are: unset, there is no TFTP listener and nothing at `/boot/…`. Naming it starts TFTP on `0.0.0.0:69` unless you say otherwise. Port 69 is privileged, and it is the *only* privileged port this server ever wants — with no DHCP responder there is nothing after 67 or 4011. Four ways to deal with it, all portable: ```console $ export RESCRIPTUM_USER=rescriptum # start as root, bind, then drop $ setcap cap_net_bind_service=+ep rescriptum # or grant just that one capability $ export RESCRIPTUM_TFTP_ADDR=0.0.0.0:6969 # or move it, if their DHCP can say so $ export RESCRIPTUM_TFTP_ADDR=off # or have no listener at all ``` **`off` is a value, not an absence** — it is how you say another daemon on this host hands the loader over while rescriptum serves the rest of the chain. The loaders stay served over HTTP at `/boot/…` and stay checked by `boot check`; only the listener is gone. It is a deployment workaround for somebody who wants it, **never how anything here ships**: rescriptum *is* the TFTP server, and a build that turned it off by default would have traded away the thing it is for. The [Synology package](./synology.md) binds port 69 with a `setcap`. **A TFTP port that cannot be bound does not stop the server**, and that is the one place this project's "a listener that cannot bind is fatal" rule inverts. Port 69 is the only privileged port in the design, so it is the only bind that can fail for something nobody configured — a capability an upgrade quietly dropped, most often. Answers are the product; dying here would fail every install in flight to report that a second port could not be opened. So it warns, keeps serving, and `boot check` exits non-zero: ```console $ rescriptum boot check BROKEN nothing answers on 0.0.0.0:69 and it cannot be bound either: Permission denied. Port 69 is privileged: run as root and set RESCRIPTUM_USER to drop afterwards, or grant the binary cap_net_bind_service with setcap — the server still answers and still serves media, but a machine sent here by DHCP asks for a loader and gets nothing ``` It asks the port for a real loader rather than trying to bind it, because binding proves the opposite of what it looks like: a bind that *succeeds* means nothing is listening, and a bind that fails cannot tell this server apart from another daemon squatting the port. **Binding happens first and dropping second**, always. The other order works in testing as root and fails on deployment, at a reboot, which is the one moment nobody is watching. ## Their DHCP server's two lines ```console $ rescriptum boot dhcp-snippet --format dnsmasq # rescriptum 0.2.0 - boot handoff for 192.0.2.10 # Architecture values are IANA option 93 codes; see docs/guide/boot/dhcp. # Generated from the same table the TFTP server serves from. dhcp-match=set:bios,option:client-arch,0 dhcp-match=set:efi64,option:client-arch,7 dhcp-match=set:efi64,option:client-arch,9 dhcp-match=set:efiarm64,option:client-arch,11 … ``` `--format` covers `dnsmasq`, `isc`, `kea`, `powershell`, `pfsense` and `mikrotik`; `--one-loader` emits the single-line form for a fleet that is all one architecture. **The snippet and the TFTP server are generated from one table**, so what you paste in and what the server hands out cannot drift apart. What they *can* do is name a loader nobody has downloaded yet, and that fails silently at the ROM — the machine asks, gets nothing, and stops with no message on any console. One command catches it: ```console $ rescriptum boot check checking boot assets in /srv/boot ok ipxe-arm64.efi (1.0M) MISSING ipxe-undionly.kpxe — every machine the snippet sends here will ask for it, get nothing, and stop ``` Its exit code is a contract, like `check`'s. Put it in the same place. ### Four details the generated snippet gets right Each is a way this fails quietly on somebody else's network, and none is obvious: - **Both the BOOTP `file` field and option 67.** Some ROMs read only one, and which is not predictable from the vendor. - **An untagged default at the end.** Every architecture line is tag-matched, so a ROM that sends no option 93 would match nothing and get no boot file at all. - **`HTTPClient` echoed back in option 60** for UEFI HTTP Boot clients. The firmware *filters offers* on it: a reply carrying only the URL is discarded, silently, which is indistinguishable from having no DHCP server. - **A next-server for those clients too**, even though they fetch over HTTP. Without one the loader's embedded script reads an empty `${next-server}` and chains into nowhere. ::: tip Windows Server A DHCP policy **cannot condition on option 93** — the condition types are vendor class, user class, MAC, client id, FQDN and relay information. The architecture reaches a policy only inside the option 60 string, so the generated PowerShell defines vendor classes on `PXEClient:Arch:00007*` and hangs the policies off those. Same outcome, different mechanism, and it is exactly the sort of thing that gets half-remembered. ::: ## The loader TFTP hands over **one file**, and the rule is written into the code: > **TFTP hands over the loader. Everything after that is HTTP.** At 1468 bytes a round-trip, TFTP moves about 1.4 MB/s on a millisecond of latency. The loader is a megabyte — two seconds. A 1.5 GB image would be the better part of twenty minutes, against fifteen seconds over HTTP on the same wire. Which loader depends on what the firmware announced: | Option 93 | Client | Served | |---|---|---| | `0x0000` | BIOS PXE | `ipxe-undionly.kpxe` | | `0x0007`, `0x0009` | UEFI x86-64 | `ipxe-x86_64.efi`, plus `-snp` / `-snponly` | | `0x000b` | UEFI ARM64 | `ipxe-arm64.efi` | | `0x0010`, `0x0013` | UEFI HTTP Boot | the same files, over HTTP, no TFTP at all | | everything else | 32-bit UEFI, EBC, U-Boot | refused, with the reason | `0x0009` needs a word. RFC 4578 defined it as "EFI x86-64"; IANA's registry, rewritten by RFC 5970, lists it as "EBC". Real x64 firmware sends either, so both map to x64 — a table generated from the registry alone would hand half a fleet nothing. `snponly` exists because the plain UEFI build cannot always see the NIC. All the variants are served and the table picks; this is precisely the knowledge an operator should not have to acquire. ### Getting them Every release attaches `rescriptum-boot-assets-.tar.gz`. Unpack it where the server can read it, name the directory, and check it: ```console $ tar -xzf rescriptum-boot-assets-0.2.0.tar.gz -C /srv $ export RESCRIPTUM_BOOT_DIR=/srv/rescriptum-boot-assets-0.2.0 $ rescriptum boot check ``` It carries the eight loaders, a `SHA256SUMS`, a bootable `ipxe.iso` and `ipxe.usb` for a machine with no usable PXE ROM, and a `NOTICE` — they are iPXE, GPLv2, built from a pinned upstream commit. **They are a separate download and not part of any binary archive or `.spk`**, deliberately: separate files served alongside is mere aggregation, and `packaging/ipxe/` is the written offer that goes with them. To build them yourself instead — the same script the release runs, from the same pin: ```console $ packaging/ipxe/build.sh --out /srv/boot ``` A loader from elsewhere works too, provided it chains to *this* server rather than to the internet — see below for why a stock one does not. ## What happens on the *second* boot The first question anybody asks after a successful install, and it has a real answer. A machine that was just installed reboots, and if network boot is still first in its BIOS order it arrives back here. What happens next is decided by one setting: | `RESCRIPTUM_BOOT_UNCLAIMED` | A machine no answer claims | |---|---| | `menu` (default) | gets the menu, whose first entry is the local disk and whose timeout falls through to it — fifteen seconds, then the disk | | `local` | is handed straight back to its firmware, which moves to the next boot device | **These are opposite readings of what an answer file means**, and the choice belongs to the deployment. With the menu, a file claiming a machine is how you say *leave this one alone* — because without one it lands in a menu somebody could click. That is right while machines are being provisioned, and it is the project's thesis: a machine nobody has decided anything about should end up where a human can decide. With `local`, an answer file means *install this one*, and its absence is the safe state. Nothing happens to a machine you have not written a file for — it boots its own disk, every time, with no menu to click by accident. That is the reading a fleet in production needs, and it is the one that scales: the number of machines you want to reinstall is always smaller than the number you do not. **The payoff is that netboot can stay first in the BIOS order forever.** Reinstalling a machine becomes *add a file, reboot* — no console, no boot menu, no hands on the hardware. Removing the file is what stops it happening twice. ```console $ rescriptum config set RESCRIPTUM_BOOT_UNCLAIMED=local ``` Either way the machine's identity still goes up first. The setting decides only what happens when nothing claimed it — not whether to ask. ## Installing a machine once, and only once A machine claimed by an `.ipxe` answer installs, reboots, is claimed again, and installs again — wiping its disk every time. Every provisioning system answers this the same way: a machine is *armed* for install, and something disarms it afterwards. **The machine is what knows.** Proxmox calls a webhook after a successful install and **before the reboot**, with its network interfaces in the body: ```toml [post-installation-webhook] url = "http://192.0.2.10:8000/installed" auth-token = "nas:s3cr3t" ``` ```console $ rescriptum config set RESCRIPTUM_INSTALLED_TOKEN=nas:s3cr3t ``` That is the whole of it. The machine finishes, says so, and its `.ipxe` moves from `98fa9b50d810/` to `installed-98fa9b50d810/` — a directory name that no longer matches it, because the prefix is part of the name that gets compared. It boots its own disk from then on, and re-arming it is moving the document back. The disarmed document goes to a **sibling directory** rather than staying inside the machine's own, so `98fa9b50d810/` keeps meaning "this machine's configuration" and nothing in it has to be read as switched off. **No token, no endpoint** — absent rather than open. Without one, `/installed` is an ordinary answer request like any other path, which is what keeps a URL bakeable into an ISO. Three things it will not do, and each is deliberate: - **It never touches a group.** A group claims a whole rack, and one machine finishing its install must not disarm its neighbours. The lookup does not consult groups at all rather than filtering them out afterwards. - **It never touches anything but the `.ipxe`.** The machine's own `.toml`, beside it in the same directory, is what the installer read to build it, and it stays as the record of how. - **It moves, it does not delete.** This is the one path where something arriving over the network changes the answer set, so nothing it does is irreversible. Arriving twice is not an error — a webhook may be retried, and a machine installed from the menu was never claimed at all. A disarm that *fails* is logged as `still armed`, because the consequence is otherwise silent: the machine reinstalls on its next boot and nothing else would say so. ### Every other family reports back too Proxmox is the only one with a webhook of its own. **The claim is not Proxmox-specific** — it is an `.ipxe` document, which is about the loader rather than the operating system — so every family needs the same disarm, and every family has somewhere to run one line at the end of its install: ```bash curl -fsS -X POST -H "Authorization: Bearer nas:s3cr3t" \ "http://192.0.2.10:8000/installed?mac=$(cat /sys/class/net/*/address | head -1)" ``` No body, no JSON: the query says which machine, the header says it is allowed. Where that line goes: | Family | Where | |---|---| | Proxmox | `[post-installation-webhook]` — native, nothing to write | | Debian | `d-i preseed/late_command string in-target sh -c '…'` | | Ubuntu | `late-commands:` in the autoinstall document | | RHEL, AlmaLinux, Rocky | the `%post` section of the kickstart | | SUSE | `` in the AutoYaST profile | **Pick the booting interface's MAC, not the first one alphabetically.** The example above takes whichever `/sys/class/net` entry comes first, which is fine on a machine with one NIC and wrong on a machine with four — and the wrong MAC disarms the wrong machine, or nothing at all. On a machine with several, name the interface. The endpoint takes the credential either way — Proxmox's `auth-token` arrives inside the JSON body because that is what Proxmox sends, and a bearer header because that is what a shell script sends. Same secret, same constant-time comparison. ## When everything is right and the machine still will not install The chain can be perfect and fail at the last step, on the machine rather than on the server. Two that have actually happened, both on a Lenovo with vPro: **Intel AMT with a static address, on a NIC it shares with the host.** The installer's own `dhclient` sends two requests about eleven seconds apart and then gives up; if the Management Engine holds the interface with a static configuration while the host asks for DHCP, those eleven seconds pass with no offer and the install aborts with `Fetching answer file via HTTP failed: Network is unreachable`. **Set AMT to DHCP too.** Running `dhclient -v eno1` by hand from the installer's shell afterwards succeeds immediately, which is what makes this so confusing to diagnose: the network is fine, the timing is not. **A switch port that does not forward straight away**, for the same reason and with the same symptom — RSTP converging, or a link still negotiating after the kernel takes the NIC over from iPXE. There is nothing this server can do about either: eleven seconds is the installer's window, not ours. The installer drops to a root shell when it aborts, and that shell is the fastest diagnosis there is: ```console # ip link # is the interface up at all? # dhclient -v eno1 # does an offer come back when asked by hand? # ip addr show eno1 ``` An address appearing there and not during the install means the network works and the machine simply asked too early. ## How iPXE ends up talking to *us* The question nobody expects to have to answer. Whatever delivers the loader: - A plain `undionly.kpxe` from ipxe.org does DHCP, is told to load iPXE, and **loads itself forever** — iPXE's documented chainloading loop. - A stock netboot.xyz binary has an embedded script that goes straight to the **public** `boot.netboot.xyz`. No loop, but your menu and your answers are never consulted. The loaders rescriptum ships carry a three-line script that chains through `${next-server}` — the value option 66 already set, which is how the loader arrived in the first place. That makes one generic build work in every deployment, with no second condition in a configuration file somebody else owns. The script chains to **port 8001**, and that is a contract rather than a preference: it is baked into the loader before any deployment exists and can read no configuration. Moving `RESCRIPTUM_MEDIA_ADDR` is allowed and `boot check` warns about it. ## What a machine sees Stage two puts the machine's identity in the query string, which is the one thing DHCP cannot do — a DHCP option cannot carry `${net0/mac}`: ```console $ rescriptum boot bootstrap #!ipxe chain http://192.0.2.10:8000/ipxe/boot?mac=${netX/mac}&uuid=${uuid}\ &serial=${serial:uristring}&asset=${asset:uristring}\ … || chain http://192.0.2.10:8001/ipxe/menu ``` Two details in there are load-bearing. **`netX`, not `net0`** — `net0` is merely the first interface, so a server booting from its second port would identify as its unused first. And **`:uristring`** on every SMBIOS string, because `${manufacturer}` expands to `Dell Inc.` with the space and iPXE percent-encodes nothing on its own. That final `||` is the whole of "a menu is the default answer": a machine something claims gets its own unattended answer, and a machine nothing claims falls through to the menu. It is `default/`'s job description word for word, applied to a different format. ## The menu ```console $ rescriptum boot menu ``` Rendered from the catalogue **at request time**, not kept in sync as a file: drop an ISO in the media directory and it is in the menu on the next fetch. - **`Boot from the local disk` is first, and the timeout falls through to it.** A machine that PXE-boots by accident, and that nothing claims, ends up on its own disk after fifteen seconds. It never sits waiting for a human who is not coming, and it never installs anything. Combined with the rule that an unclaimed machine gets a menu rather than an install, **the worst case of being wrong about which machines reach this server is a few seconds added to a boot.** - Entries are **gated on the client's architecture**, so an ARM64 image is not offered to an x86 machine — that is an entry that boots the wrong kernel. - An image no probe could place is still offered, as a CD. - The diagnostics entries — a shell, `netinfo`, and one that boots a *different* rescriptum — are what every boot server ends up needing. The last is how you test a candidate server on site, from the running one, without touching DHCP or the loaders. `RESCRIPTUM_BOOT_TIMEOUT_SECS` (default 15) sets the wait, and `RESCRIPTUM_BOOT_TITLE` the title bar. The logo is fetched with `console --picture … ||`, which **tolerates its own failure**: a serial console over IPMI has no framebuffer, and that is how half of all datacenter installs are watched. ## What breaks when this server is down Worth stating plainly, because "a boot server" sounds load-bearing and is not: | | rescriptum down | |---|---| | DHCP addressing, DNS, routing | **unaffected** — it speaks none of those protocols | | Machines already installed and running | **unaffected** | | Machines rebooting | **unaffected** — they boot from disk | | A machine that PXE-boots by accident | falls to its next boot device, as it would anyway | | Starting a *new* installation | stops | **Nothing rescriptum installs depends on rescriptum afterwards.** The answer endpoint is consulted during an install and never again. ## Security Boot traffic is unauthenticated, and necessarily — a PXE ROM has no credentials, the same necessity that already governs the answer endpoint. So the controls are structural, and one of them can say *not you*: ```console $ export RESCRIPTUM_BOOT_ALLOW=10.0.0.0/8 # shared by TFTP and media ``` UDP is forgeable and TFTP is UDP, so the server **never answers a broadcast or multicast destination** — amplification hygiene rather than politeness — caps concurrent transfers in total and per peer, and logs every one. It is read-only: a write request is refused as an access violation, because writing a loader over unauthenticated UDP would be a way to change what every machine on the segment boots. **A boot VLAN is the honest recommendation** and the one that actually works. See [Security](./security.md). ::: tip Secure Boot Our loaders are unsigned, and shim only loads what its distro's vendor key signed — so serving a shim beside an unsigned iPXE is not Secure Boot support, it is a boot that stops at a signature error. What does work: turn Secure Boot off, enrol a MOK, or let firmware PXE-boot the target distro's *own* signed shim and GRUB, served from the media listener like any other file. We sign nothing and strip nothing, and nothing here weakens a machine that has Secure Boot on. ::: ## When their DHCP genuinely cannot be touched None of this costs a line of code, and all three work: - **UEFI HTTP Boot with a URL typed into firmware setup.** Modern server firmware lets you enter a boot URL directly. The chain then starts on the media listener with no DHCP option involved at all. - **iPXE from IPMI virtual media, a USB stick, or the NIC's own ROM**, carrying this server's address. A one-megabyte image, mounted once per machine. - **dnsmasq in proxy-DHCP mode**, for a site that truly has a DHCP server it cannot edit. It exists, it is mature, it is three lines of configuration, and it is not ours to rewrite. Naming it is the honest answer.