Transfers silently stalled Network assumptions | A secondary drifted behind the signer | Port 5353 is multicast DNS and was swallowed leaving the VMs | Transfer route moved to 8053 | Probe every new route from the server that will use it |
|---|
False DNS failures after retiring an address Network assumptions | Readiness fell to 0 of 35; cluster cards out of sync | Three checks still queried the retired public address | One endpoint rule: LAN, then route, then public | Sweep every query target when an address is retired |
|---|
New VM had no network Network assumptions | No address after first boot | Copied settings let the hypervisor regenerate the MAC address | Static MAC matched to the first-boot network settings | Identity is explicit in every build |
|---|
External router faults Network assumptions | Agent flapping; router admin page crashed | Consumer router at a remote site | Treated as an external dependency and monitored (open) | Watch every external dependency; never assume it |
|---|
Hidden two-or-three-server limits Hidden coupling | A fourth server failed to join, step by step | Assumptions scattered through the code | Membership generalised to any number | Membership rules live in one place |
|---|
Address uniqueness clash Hidden coupling | A second server could not be built at one site | Public address treated as unique per server | Uniqueness by hostname and LAN address, now in the build queue too (release 7.3) | One rule, applied everywhere it is checked |
|---|
Standby could not run Platform portability | The cloned VM refused to start | An ARM VM was copied to an Intel host | Portable encrypted replica, rebuilt per architecture | Check the architecture first; state travels as data, not disk images |
|---|
A standby could still be started Platform portability | Masking services silently failed | Unit files sat where the mask had to go | A do-not-start-while-standby rule tied to one marker | Promotion is removing one file |
|---|
Signing keys on public servers Safety design | Found reviewing the key mirror | Keys were mirrored to every secondary | Keys only on hidden standby signers | No private key on a published server |
|---|
Split brain: two signers Safety design | Found in design review, before it happened | Each side able to act alone when cut off | Witness plus two checks; isolation hold; epoch fencing | No side acts without independent agreement |
|---|
False Steward down Check validity | The watcher saw Steward down while it was up | A request method the server does not implement | A real request; only a missing backend counts as down | Test every check against reality before it can act |
|---|
Standby user IDs differed from home Platform portability | The routing gate refused at the standby | Users created with whatever numbers were free; settings store numbers | Standby IDs match home by construction | Explicit identity, the same at every site |
|---|
Forced promotion had no signing profile Hidden coupling | The new signer refused to configure | The profile normally travels in the old signer’s fence receipt | Profile recorded every minute and checked before a drill | A forced move ends confirmed, as a planned one does |
|---|
A finished build agent held the signer’s database Hidden coupling | Promotion could not install its database | The commissioning agent kept running for 15 days | Agent stopped; to close its database and exit after joining | Never install a database beside another’s journal |
|---|
Signer’s LAN address missing as a notify source Network assumptions | A secondary ignored changes until it polled | The member record held only the public address | LAN address added at every member and Steward | Sweep every query target when an address retires |
|---|
Catalog left frozen after a forced move Hidden coupling | New zones would not have propagated | The forced path froze catalogs but never thawed them | Each follower thaws its catalog | A forced move ends confirmed, as a planned one does |
|---|
Notifications refused at forwarded ports Network assumptions | Secondaries caught up only by polling, since 4.22 | A primary configured as address:port never matches the sender | Each secondary trusts its primary’s own addresses; signed notifies only | Probe every new route from the server that will use it |
|---|
Database installed beside another’s journal Safety design | Steward refused a damaged database; recovered intact | Replica applied without removing the old write-ahead journal | Old journals removed before any database is installed | Never install a database beside another’s journal |
|---|
Forced records never closed Hidden coupling | A planned handover could not start afterwards | The forced path had no confirm step | force-confirm closes every member’s record | A forced move ends confirmed, as a planned one does |
|---|
Diagnostics held a running handover Check validity | A handover paused as if Steward had restarted | A status check started a second handover worker | Status read from the gate or the page; never the module | Observe before acting; diagnostics never act |
|---|
Handover probe assumed the old shape Network assumptions | The handover held at its baseline | Probe used a retired address and a same-site destination | Hidden servers probed through their routes; same-site read directly | Sweep every query target when an address retires |
|---|
A patch shadowed a variable Check validity | The corrected probe still failed | Test covered part of the file, not the whole function | Tests run the whole function on the exact source | Test every check against reality before it can act |
|---|
Handover check sized for three members Check validity | Repointing stopped after the first secondary | With three members only one secondary was ever untouched | Repointed members measured as repointed | Membership rules in one place, for any number |
|---|
Weekly serial tick sent no notification Network assumptions | Secondaries lagged after Thursday 00:00 UTC | The served serial changes with the week; nothing notifies | A daily NOTIFY of every signed zone follows the weekly change (release 7.3) | Every serial change is notified |
|---|
A changed reply had a second reader Check validity | Every zone lost its external evidence; panel 8 showed 0/35 | The probe’s reply changed; one of its two readers was updated | Every reader updated; the release lists all readers and stops on any unknown | Change a reply, update every reader |
|---|
PowerDNS-Admin’s requests never passed the gateway Check validity | Zones could not be created, nor records edited, from PowerDNS-Admin | The boundary required PowerDNS’s own spelling of a zone’s kind and name | The gateway normalises both spellings before every check | Accept a client’s spelling, check its meaning |
|---|
A new zone missed its notification targets Hidden coupling | Changes to a new zone would not promptly reach the hidden signer | Only promotion set each zone’s notification targets | Zone creation sets them by the same rule as promotion | Membership rules in one place, for any number |
|---|
A fix on disk was not the code running Check validity | A corrected gateway still refused requests | The service had not restarted onto the new code | Every release waits until each service has restarted after the install | Prove the code running, on both Stewards |
|---|
The standby ran older code after a release Hidden coupling | The standby held fewer pinned files than home | The standby pulls hourly, so it could lag by up to two hours | Every release ends with the standby pulling at once, both manifests compared | Prove the code running, on both Stewards |
|---|
Tests passed as root but failed as the service Check validity | Two tests failed on Steward after passing in rehearsal | A test wrote its sample file as whoever ran it | Tests run as the services run; the ownership check has its own test | Test and rehearse as it really runs |
|---|
A release rebuilt by pattern lost a step Safety design | A release’s standby step had nothing to send | One release’s script was edited into the next by text patterns | Each release is built from its parts, and the whole script rehearsed | Test and rehearse as it really runs |
|---|
Fallbacks carried this cluster’s data Hidden coupling | The retired signer address survived in four fallbacks | Each fallback copied the addresses of its day | Fallbacks removed; missing data is refused, never guessed | No cluster data in the code, nor beside it |
|---|
Old backup copies accumulated beside the code Safety design | 201 old copies, carrying retired addresses into every replica | Each release kept its previous version next to the file it replaced | Old copies moved to /var/backups; releases keep their backups there | No cluster data in the code, nor beside it |
|---|
The bundle was not found on the seed disk Platform portability | A clean VM’s first boot could not find Steward | The seed disk reports file names in another case | The bundle travels as one bundle.tar, found whatever the case | Test and rehearse as it really runs |
|---|
Account numbers already taken on the official image Platform portability | Steward’s accounts could not be created on a clean VM | Debian’s cloud image already uses user 997 and group 988 | Accounts created by name, never by number | Explicit identity, the same at every site |
|---|
Services kept the settings they started with Check validity | A new Steward showed the default web server page | Apache and PowerDNS-Admin were configured but not restarted | Both restarted after configuring, then checked | Prove the code running, on both Stewards |
|---|
First-run setup asked for the writer lease Hidden coupling | “Service Unavailable” when a new Steward applied its settings | The lease belongs to a cluster, and there was none yet | First run only queues its request, outside the lease (release 7.17) | A new installation starts empty, and says so |
|---|
An empty cluster crashed start-up Hidden coupling | A brand-new Steward stopped as it started | A membership with no servers was written; start-up assumed two | No membership until there are servers; the page says “No servers yet” (release 7.18) | A new installation starts empty, and says so |
|---|
Tests took the live cluster’s writer lease Safety design | Found while proving the installer | The registrar tests ran with cluster routing switched on | Those tests run with routing off (release 7.19) | Tests never touch the live cluster |
|---|
A build waited for approval at the host’s screen Platform portability | A VM build stopped until someone approved an upgrade on the remote Mac | The hypervisor asks before upgrading a VM’s hardware version | Builds to set the hardware version themselves, so nothing asks (open) | No step waits for a person at a screen |
|---|
A release’s preparation step broke the release Safety design | Release 7.22 stopped at its last step without restoring | The preparation step reused the name of the release’s staging folder | Every step in its own shell; any failure after the install restores everything | A failed release leaves nothing changed |
|---|
Lists sized for today’s 35 zones Hidden coupling | Settings listed every domain on one page | Each list was designed at the size of this cluster | A summary in Settings; zones shown 100 at a time, with search (releases 7.23 and 7.24) | Every list works at 10,000 zones |
|---|
Certificate renewal checked a retired address Network assumptions | Found by reading the hook before the next renewal | The hook waited on every member’s public address, one now answering nothing | Only the published nameservers are checked; the renewal dry run passed (release 7.25) | Sweep every query target when an address retires |
|---|
The registry was built on the control plane Safety design | Release 7.20 put registrant access on the Steward machine | Where the registry runs was assumed, not confirmed | Removed in 7.21; the registry gets its own VMs, with one narrow path into DNS | Public services never run on the control plane |
|---|
Operating choices were written into the code Hidden coupling | Backup time, thresholds and counts changed only by release | Defaults were written where they were used | Settings → Operations, checked again and applied by a root service (release 7.22b) | An owner’s choice is a setting, never code |
|---|
This site’s data was in the Mac app Hidden coupling | The repository scan stopped its first commit | A form defaulted to this site’s Steward address | Defaults cleared; every commit is scanned for site data and secrets | No cluster data in the code, nor beside it |
|---|