Theme. Bring the domains whose DNS answers can be forged without detection to zero, by making DNSSEC simple enough to run unattended.
Don’t repair a server, replace it. Every node is a disposable Linux VM on Apple Silicon, built from a verified image and swapped in whole. The analysis below is why; panel 8 carries the live proof.
Attackers have redirected domain names for years while users saw nothing wrong. DNSSEC was designed so forged answers fail.
The infrastructure is ready: nearly every top-level domain is signed and the big resolvers validate. The gap is the last link, the individual domain.
Each target replaces work an operator would otherwise do by hand. Progress against them is measured live in panel 8.
Observe, approve the plan, then run unattended: each kind of action earns its autonomy separately.
The cryptography was never the obstacle. Running it correctly, forever, across organisations is.
Two sites, each a Mac mini running Linux VMs: a published nameserver, a hidden signer and a Steward. Upgrades build a new VM and swap it in; nothing is patched in place.
Every release checks what it changes before and after, pins the files it installs, and records its own way back.
Most failures came from assumptions about network paths and hidden coupling, not from DNSSEC. Each one now has a standard.
A fix only counts when the system enforces it. Each lesson is held by code, by a check, or by a release step.
No release files yet, by design: Steward is one end-to-end workflow, so it ships when the whole of it is proven from a clean install.
From domain names and DNS to connected systems and live production, our work has always been about using technology to solve problems and improve how things work.
The starting point was always a practical problem, and the opportunity to build a better way of solving it. Sometimes that meant a web form guiding someone through a process so they got it right first time; sometimes it meant automating a business process and making it available around the world.
Smart operations over brute force was the approach then, and it is the approach of DNSSEC Steward now.
| 2002 | The original engineering business is incorporated, beginning a 14-year chapter of using technology to solve problems and improve processes. |
|---|---|
| 2005 | zone-reg.co.uk is registered, giving that engineering approach a home in domain registration and DNS setup. |
| Domain years | A Nominet registrar, then a combination of Netim and Enom. API integration checked both providers’ prices every hour and routed each registration to the lowest-priced one. |
| Today | We maintain our own domain portfolio, now signed and run by DNSSEC Steward. Our business focuses on streaming and recording through e-businesshosting.com, working with APIs such as EVE Online’s ESI, zKillboard, Twitch and Facebook. |
Zone Reg no longer sells or registers domain names for customers.
| What | DNS answers can be forged or redirected, sending users to an attacker’s server while everything looks normal. |
|---|---|
| Where | Between users and the real servers: resolver caches, authoritative nameservers, internet routing, and registrar accounts. |
| When | Repeatedly. The 2008 cache-poisoning disclosure, then a run of hijacks against wallets, governments and providers from 2017 to 2019. |
| Who | Any domain owner and every user of that domain, from individuals to foreign ministries. |
| Why it matters | The user can’t tell. In the Sea Turtle campaign the attackers obtained genuine certificates, so the padlock showed. |
| How it is prevented | DNSSEC signs each answer. A validating resolver rejects any answer whose signature doesn’t check out against the chain of trust. |
| How much | Almost nobody signs. See sheet 2. |
| 2008 | Cache poisoning shown to be far easier than believed; emergency patches industry-wide. |
|---|---|
| Jan 2018 | BlackWallet.com DNS hijacked; over $400,000 in Stellar Lumens stolen. |
| Apr 2018 | MyEtherWallet: routing hijack lets an imposter nameserver answer for myetherwallet.com and direct users to a fake site. |
| Nov 2018 | DNSpionage reported: government and company records altered to harvest credentials in transit. |
| Jan 2019 | CISA Emergency Directive 19-01, its first ever: audit DNS, enforce MFA on registrar accounts, monitor for changes. |
| Apr 2019 | Sea Turtle disclosed: a state-sponsored campaign since 2017, hijacking DNS through registrars and providers. |
Sources: Cisco Talos; CISA ED 19-01; BleepingComputer; The Register; Computer Weekly.
| Root and TLDs | 92.94% of top-level domains signed (ICANN ITHI, January 2025). The top of the chain is done. |
|---|---|
| Resolvers | Google Public DNS has validated every query by default since May 2013. A forged answer for a signed domain gets SERVFAIL, not a fake address. |
| Registrars | Registrars must offer DNSSEC, yet in June 2025 fewer than one in ten signed their own primary domain. |
| Parents | Some parent zones offer free, signed delegation but no API: at eu.org every DS change is a request handled by hand. Automating rollovers ends at whichever link is still manual. |
| Domains | Signed .com zones rose from 1.5 million at the start of 2020 to about 7 million: real growth, still a small share of .com. |
| Browsers | Chrome supported DNSSEC-stapled certificates, then removed them because almost nobody used them. |
Every part of the system that a few large organisations control is ready. The part that belongs to millions of individual owners is not. Validation is free for users of a validating resolver, so each domain that is signed is protected the moment it is signed.
The barrier sits with the owner, which points the analysis at operations rather than technology.
Sources: ICANN ITHI metric M7; Google Security Blog, 19 Mar 2013 and update 6 May 2013; CircleID, June 2025 registrar study; Internet Society Pulse; imperialviolet.org (Adam Langley, 2015).
| Target | Measure | Why this measure | Status |
|---|---|---|---|
| Routine change needs no person | manual interventions per month: 0 | The burden that stops owners signing | rollovers and checks unattended; upgrades next |
| No validation outage | minutes of bogus answers: 0 | A DNSSEC mistake removes a domain entirely | met; doubt pauses change |
| No key on a public server | published servers holding keys: 0 | A compromised public server must not expose keys | met since release 5.4 |
| Survive the loss of a site | time to signing at the other site: 15 min or less | Home-grade sites lose power and links | met: signing at the other site in 34 s (drill, 11 Nov); back by handover 12 Nov |
| Fresh standby | age of standby keys: under 1 hour; state: under 3 hours | A standby is only as good as its copy | keys about 1 minute, state hourly |
| DS automated wherever there is an API | manual DS steps per key rollover: 0 | The parent is the one link a signer cannot sign for itself | eNom and Porkbun by API; the eng.quest API, gate 6 |
| Phase | Delivers | Gate |
|---|---|---|
| P0 Define | theme, 5W2H, targets, this board | targets agreed |
| P1 Prove resilience | standby signer and Steward, watcher, isolation hold, failover drill, failback | a successful drill and 30 days green |
| P2 Unattended upgrades | certified releases, upgrades as the role moves | one full upgrade wave with no touches; scheduled after release 1.0 |
| P3 Productise | installer and image, test lab, eng.quest parent, GitHub, documents | a clean install from the repository, signing lab.eng.quest |
| P4 Publish | this site, live | board reviewed against the QC checklist |
The fishbone above lays out every cause by category. The Process bone carries the most weight, so it is drilled with the five whys below.
Evidence for why 4: 28 to 32% of signed .com, .net and .org domains failed to publish a DS record (IETF 101 MAPRG, 2018, data to 2016).
| MFA on registrar accounts | DNSSEC | |
|---|---|---|
| Stops | an attacker changing your records at the registrar | forged answers from caches, imposter servers and rerouted traffic |
| Doesn’t stop | forged answers in transit | a hijacked registrar account |
| Role | containment, now widespread | permanent countermeasure, rarely deployed |
Each covers what the other cannot. People adopted the easy one. This project makes the other one easy.
| Node | One public address per site. Each site runs a published nameserver, a hidden signer and a Steward, all Linux VMs on a Mac. |
|---|---|
| Signing | One signer at a time, hidden from the internet. Only secondaries answer the world. |
| Standby signer | Holds a verified copy of every key, refreshed about once a minute. |
| Standby Steward | Passive, rebuilt from an encrypted replica every hour, unable to act until promoted. |
| Decisions | Promotion needs two independent views of a site being down: our own checks and an external witness. |
| Registrars | Which registrar holds a domain is kept as data: each domain is bound to the account that holds it, confirmed against every account’s complete inventory, and each registrar declares what its API can do. eNom and Porkbun today. A transfer between registrars is the binding changing over time, so it can become a journalled operation that keeps the DS intact at the gaining registrar, and a domain in transfer is never mistaken for one that has gone. |
| Parent zone | eng.quest is signed by Steward on the same cluster, with its DS at .quest placed through Porkbun’s API. Its names are designed to ensure DNSSEC is implemented at all times: each must be signed, by the registrant’s own systems or by DNSSEC Steward, with its DS placed through the eng.quest interface or API, and every name is checked continually. |
| Upgrades | Build a fresh VM from a verified image, prove it, swap it in, discard the old one. |
| Option | Rejected because |
|---|---|
| Patch servers in place | drift: no two servers stay identical, and a failed patch leaves a broken server |
| One signer with backups | recovery is manual and slow, exactly the expert work being removed |
| Cloud VMs | recurring costs that keep rising, for compute already available on machines running 24/7 |
| Hosted DNS provider | keys and control sit with someone else; no learning to share |
| Keys on every server | any public server compromised would expose them |
| Mac mini | Idle | Max |
|---|---|---|
| M4, base model | 4 W | 65 W |
| M4 Pro, fully configured | 5 W | 140 W |
| 2018 Intel 6-core i7, the remote site today | 10 W | 85 W |
An always-on node at 5 W uses about 44 kWh a year. Office broadband is already on around the clock, and the machines already exist, so the nameservers run on spare capacity rather than rented compute.
The so-what: cloud service costs keep rising with the state of the world at large, and they recur every month. A small, owned, low-power node does not.
Power: Apple Support article 103253, measured from the wall including power-supply losses.
| RFC 2182 | Secondary nameservers at geographically and topologically diverse locations |
|---|---|
| RFC 1034, 1035 | The domain name system itself |
| RFC 1996 | NOTIFY: the signer tells secondaries a zone changed |
| RFC 5936, 1995 | AXFR and IXFR: full and incremental zone transfer |
| RFC 8945 | TSIG: every transfer between sites authenticated |
| RFC 4033 to 4035 | DNSSEC: signatures, keys and the chain of trust |
| RFC 6781 | DNSSEC operational practices |
| RFC 7583 | Key rollover timing |
| RFC 9432 | Catalog zones: secondaries learn which zones to serve |
| Application | Python: the standard library plus dnspython and cryptography. No framework, no build step, small enough to read. |
|---|---|
| Services | Linux services and timers on Debian: the web front end, the signer API gateway, status every minute, key mirror, readiness, backups, the hourly replica. |
| DNS | PowerDNS with SQLite: one signer, secondaries fed by NOTIFY and authenticated transfer. |
| Nodes | Linux VMs built by cloud-init from a verified Debian image. A node is rebuilt, never repaired: maintenance is building the next one. |
| Hosts | Mac minis running VMware Fusion, managed by a small Swift agent that builds VMs and reports health. |
| Date | Release | What it added |
|---|---|---|
| 25 Oct | 4.19 to 4.21 | Off-site watchdog and backups; cluster topology read from data, not code |
| 26 Oct | 4.22 to 4.25 | Signer hidden from the internet; published set held as data; unpublishing made safe |
| 26 Oct | 5.0 to 5.1 | Status checks every minute, event timeline, notifications on both Macs, external witness |
| 27 Oct | 5.2 to 5.3 | Any number of members; a fourth server built, enrolled and joined from the browser |
| 27 Oct | 5.4 | Signing keys only on hidden standby signers; removed from published servers |
| 27 Oct | 5.6 | Standby Steward rebuilt for another CPU architecture from an encrypted replica |
| 31 Oct | 5.7 to 5.8 | Standby kept current hourly; failover watcher judging two independent views |
| 31 Oct | 5.9 to 6.0 | Isolation hold against split brain; forced promotion with epoch fencing |
| 10 Nov | 6.1 to 6.2 | Transfer route off the multicast DNS port; every check reaches members the way Steward can |
| 10 Nov | 6.3 | Public, sanitised live summary for this board, rebuilt at most once a minute |
| 11 Nov | 6.4 to 6.9 | Failover drill: signing moved to the remote site in 34 s; six faults found and fixed on the way |
| 12 Nov | 7.0 to 7.4 | Failback by receipt-based handover; Porkbun connected; eng.quest signed; gate 3; this site’s names moved into settings |
| 13 Nov | 7.5 | No cluster data in the code: roles from the topology, identity from the manifest, dead code and old copies removed; gate 5 |
Every release verifies the files it expects before changing anything, re-pins the hashes it changes, and prints its own revert command.
Each risk met in operation is recorded with how it showed, its root cause, the countermeasure and the standard that prevents a repeat. Open items stay on the board until closed.
| Risk met | How it showed | Root cause | Countermeasure | Standard now |
|---|---|---|---|---|
| Transfers silently stalled Network assumptions | A secondary drifted behind the signer | Port 5353 is multicast DNS and was swallowed leaving the VMs | Transfer route moved to 8053 | Probe every new route from the server that will use it |
| False DNS failures after retiring an address Network assumptions | Readiness fell to 0 of 35; cluster cards out of sync | Three checks still queried the retired public address | One endpoint rule: LAN, then route, then public | Sweep every query target when an address is retired |
| New VM had no network Network assumptions | No address after first boot | Copied settings let the hypervisor regenerate the MAC address | Static MAC matched to the first-boot network settings | Identity is explicit in every build |
| External router faults Network assumptions | Agent flapping; router admin page crashed | Consumer router at a remote site | Treated as an external dependency and monitored (open) | Watch every external dependency; never assume it |
| Hidden two-or-three-server limits Hidden coupling | A fourth server failed to join, step by step | Assumptions scattered through the code | Membership generalised to any number | Membership rules live in one place |
| Address uniqueness clash Hidden coupling | A second server could not be built at one site | Public address treated as unique per server | Uniqueness by hostname and LAN address, now in the build queue too (release 7.3) | One rule, applied everywhere it is checked |
| Standby could not run Platform portability | The cloned VM refused to start | An ARM VM was copied to an Intel host | Portable encrypted replica, rebuilt per architecture | Check the architecture first; state travels as data, not disk images |
| A standby could still be started Platform portability | Masking services silently failed | Unit files sat where the mask had to go | A do-not-start-while-standby rule tied to one marker | Promotion is removing one file |
| Signing keys on public servers Safety design | Found reviewing the key mirror | Keys were mirrored to every secondary | Keys only on hidden standby signers | No private key on a published server |
| Split brain: two signers Safety design | Found in design review, before it happened | Each side able to act alone when cut off | Witness plus two checks; isolation hold; epoch fencing | No side acts without independent agreement |
| False Steward down Check validity | The watcher saw Steward down while it was up | A request method the server does not implement | A real request; only a missing backend counts as down | Test every check against reality before it can act |
| Standby user IDs differed from home Platform portability | The routing gate refused at the standby | Users created with whatever numbers were free; settings store numbers | Standby IDs match home by construction | Explicit identity, the same at every site |
| Forced promotion had no signing profile Hidden coupling | The new signer refused to configure | The profile normally travels in the old signer’s fence receipt | Profile recorded every minute and checked before a drill | A forced move ends confirmed, as a planned one does |
| A finished build agent held the signer’s database Hidden coupling | Promotion could not install its database | The commissioning agent kept running for 15 days | Agent stopped; to close its database and exit after joining | Never install a database beside another’s journal |
| Signer’s LAN address missing as a notify source Network assumptions | A secondary ignored changes until it polled | The member record held only the public address | LAN address added at every member and Steward | Sweep every query target when an address retires |
| Catalog left frozen after a forced move Hidden coupling | New zones would not have propagated | The forced path froze catalogs but never thawed them | Each follower thaws its catalog | A forced move ends confirmed, as a planned one does |
| Notifications refused at forwarded ports Network assumptions | Secondaries caught up only by polling, since 4.22 | A primary configured as address:port never matches the sender | Each secondary trusts its primary’s own addresses; signed notifies only | Probe every new route from the server that will use it |
| Database installed beside another’s journal Safety design | Steward refused a damaged database; recovered intact | Replica applied without removing the old write-ahead journal | Old journals removed before any database is installed | Never install a database beside another’s journal |
| Forced records never closed Hidden coupling | A planned handover could not start afterwards | The forced path had no confirm step | force-confirm closes every member’s record | A forced move ends confirmed, as a planned one does |
| Diagnostics held a running handover Check validity | A handover paused as if Steward had restarted | A status check started a second handover worker | Status read from the gate or the page; never the module | Observe before acting; diagnostics never act |
| Handover probe assumed the old shape Network assumptions | The handover held at its baseline | Probe used a retired address and a same-site destination | Hidden servers probed through their routes; same-site read directly | Sweep every query target when an address retires |
| A patch shadowed a variable Check validity | The corrected probe still failed | Test covered part of the file, not the whole function | Tests run the whole function on the exact source | Test every check against reality before it can act |
| Handover check sized for three members Check validity | Repointing stopped after the first secondary | With three members only one secondary was ever untouched | Repointed members measured as repointed | Membership rules in one place, for any number |
| Weekly serial tick sent no notification Network assumptions | Secondaries lagged after Thursday 00:00 UTC | The served serial changes with the week; nothing notifies | A daily NOTIFY of every signed zone follows the weekly change (release 7.3) | Every serial change is notified |
| A changed reply had a second reader Check validity | Every zone lost its external evidence; panel 8 showed 0/35 | The probe’s reply changed; one of its two readers was updated | Every reader updated; the release lists all readers and stops on any unknown | Change a reply, update every reader |
| PowerDNS-Admin’s requests never passed the gateway Check validity | Zones could not be created, nor records edited, from PowerDNS-Admin | The boundary required PowerDNS’s own spelling of a zone’s kind and name | The gateway normalises both spellings before every check | Accept a client’s spelling, check its meaning |
| A new zone missed its notification targets Hidden coupling | Changes to a new zone would not promptly reach the hidden signer | Only promotion set each zone’s notification targets | Zone creation sets them by the same rule as promotion | Membership rules in one place, for any number |
| A fix on disk was not the code running Check validity | A corrected gateway still refused requests | The service had not restarted onto the new code | Every release waits until each service has restarted after the install | Prove the code running, on both Stewards |
| The standby ran older code after a release Hidden coupling | The standby held fewer pinned files than home | The standby pulls hourly, so it could lag by up to two hours | Every release ends with the standby pulling at once, both manifests compared | Prove the code running, on both Stewards |
| Tests passed as root but failed as the service Check validity | Two tests failed on Steward after passing in rehearsal | A test wrote its sample file as whoever ran it | Tests run as the services run; the ownership check has its own test | Test and rehearse as it really runs |
| A release rebuilt by pattern lost a step Safety design | A release’s standby step had nothing to send | One release’s script was edited into the next by text patterns | Each release is built from its parts, and the whole script rehearsed | Test and rehearse as it really runs |
| Fallbacks carried this cluster’s data Hidden coupling | The retired signer address survived in four fallbacks | Each fallback copied the addresses of its day | Fallbacks removed; missing data is refused, never guessed | No cluster data in the code, nor beside it |
| Old backup copies accumulated beside the code Safety design | 201 old copies, carrying retired addresses into every replica | Each release kept its previous version next to the file it replaced | Old copies moved to /var/backups; releases keep their backups there | No cluster data in the code, nor beside it |
| Standard | Held in code | Checked continuously | Release step |
|---|---|---|---|
| Probe each new route from the server that will use it | yes | yes | yes |
| Sweep every query target when an address retires | yes | yes | |
| No private key on a published server | yes | yes | |
| Check the architecture; move state as data | yes | yes | |
| Explicit identity, the same at every site | yes | yes | |
| Promotion is removing one marker | yes | yes | |
| Observe before acting; diagnostics never act | yes | yes | |
| Membership rules in one place, for any number | yes | yes | |
| No side acts without independent agreement | yes | yes | |
| Watch every external dependency | yes | ||
| Never install a database beside another’s journal | yes | yes | |
| A forced move ends confirmed, as a planned one does | yes | yes | |
| Change a reply, update every reader | yes | ||
| Accept a client’s spelling, check its meaning | yes | ||
| Prove the code running, on both Stewards | yes | ||
| Test and rehearse as it really runs | yes | yes | |
| No cluster data in the code, nor beside it | yes | yes | yes |
DNSSEC Steward is one end-to-end workflow, not a set of independent modules. The signer, the secondaries, the registrar, the standby and the control plane only work safely together. A part released on its own, or a whole that has not been proven end to end, risks the one failure DNSSEC does not forgive: domains that disappear. So release waits for the whole workflow to be proven from a clean install, under regression testing, not for parts of it to look finished.
| Signing and serving | Hidden PowerDNS signer; published secondaries; authenticated transfers; catalog zones |
|---|---|
| Key lifecycle | Unattended KSK and ZSK rollovers; DS updates by registrar API (eNom, Porkbun), and a guided manual path where there is no API |
| Verification | Every zone checked end to end, every minute; status, events and notifications |
| Nodes | Servers built from a verified image by the Mac agent; add and swap from the browser: replace, don’t repair |
| Multi-site | Any number of members across sites; publish and unpublish nameservers safely |
| Resilience | Standby signer and standby Steward, failover watcher, isolation hold, forced promotion with epoch fencing, rejoin, failback |
| Parent zone | eng.quest: free names, each required to be signed, by the registrant’s systems or by DNSSEC Steward; DS through the eng.quest interface or API; every name checked continually |
| Operations | Encrypted backups with an off-site copy; PowerDNS-Admin through the guarded gateway; a public live summary |
| Automatic promotion | 1.0 promotes on a confirmed decision; fully unattended promotion follows |
|---|---|
| Certified upgrades | Unattended upgrades as the signing role moves |
| Make the gap visible | Own validating resolver; unsigned sites get a “not DNSSEC secured” line |
| Transfers between registrars | A journalled transfer that keeps the DS intact at the gaining registrar; the bindings and capabilities ship in 1.0 |
| More registrars | DS by API beyond eNom and Porkbun |
| More hosts | Hypervisors other than VMware Fusion on a Mac |
| Other operators | The board’s step 10 becomes their step 1 |
| Gate | Passing means | State | Evidence | |
|---|---|---|---|---|
| 1 | Failover drill | Forced promotion and rejoin pass, with the time to signing measured | passed 11 Nov 2026 | Forced promotion to signer2 committed in 34 s; a NOTIFY then brought ns1, ns2 and ns3 up to date in 1 s; ns1 and ns3 rejoined, ns3's own signing discarded; zone-reg.net validated throughout. Found and fixed on the way: standby user IDs, the signer profile, a leftover agent holding signer2's database, signer2's notification source, catalog thaw, and NOTIFY refused at forwarded ports since 4.22. |
| 2 | Failback | Signing returns home through the normal, receipt-based handoff | passed 12 Nov 2026 | Signing returned to ns3 at epoch 10 through the receipt-based handoff; all four servers in step; ns2 now follows ns3 by notification (first time since 4.22). Found and fixed on the way: replica database installed beside a stale journal (recovered intact), the forced path's missing confirm step, the handover probe still using retired .195 and probing a same-site signer, a variable shadowing in 7.0's probe patch, the repoint check for four-member clusters, and secondaries lagging after the weekly serial tick. |
| 3 | Open items closed | Build queue uniqueness rule aligned; the two readiness copies reconciled | passed 12 Nov 2026 | Release 7.3: two servers can be built at one site; the dead readiness copy removed and the live one tested; 42 tests passed on Steward; both Stewards proven to hold the same manifest. |
| 4 | Steward installer | A Steward builds from the image on a clean node, on either CPU architecture | to do | |
| 5 | Site data separated from code | Addresses, credentials and registrar settings are configuration; nothing of this cluster is in the repository | passed 13 Nov 2026 | Releases 7.4 to 7.5d: names, sites and probe zone are settings; identity from the manifest; roles from the topology; fallbacks, dead code and 201 old copies removed. A scan finds no site data: home 0 of 108 files, standby 0 of 5. |
| 6 | eng.quest live | eng.quest signed by Steward with its DS at .quest placed through Porkbun's API; a child name.eng.quest delegated and signed, with its DS placed through the eng.quest API, and the continual DNSSEC check running with its notices and suspension page | to do | |
| 7 | Clean-room install | A fresh lab cluster, built only from the repository, signs lab.eng.quest end to end with a public chain of trust | to do | |
| 8 | Full regression on the lab | Build, join, swap, unpublish, rollover, handoff, failover, rejoin and failback all pass in one run | to do | |
| 9 | Documentation | Install, operate, recover, and the architecture | to do | |
| 10 | Security and licence review | No secrets in the history, dependencies pinned, licence chosen | to do | |
| 11 | Tag 1.0 | Signed release files with checksums | to do |
Read live from github.com/DarthCruiser/dnssec-steward (release.json). Each gate is ticked by a commit, so the repository’s history is the record.