PING

In this episode of PING we’re hearing from Duane Wessels, a fellow from Verisign who researches DNS and who we’ve had on PING before. This time Duane is discussing the history of changes of top level Key Signing Key (KSK) called “roll-over” of the KSK at the root of the DNS. This month on the 11th of October, we will complete deployment of the 3rd KSK, in the second rollover. It’s a multi-year process. This second key roll was delayed from the first rollover in 2017 by some unexpected circumstances in 2020 and 2023. The initial KSK was deployed in 2011. In October, the new key pair will enter service for the first time. The old key pair will no longer be used to sign, but will remain visible in the root until early 2027.

Duane discusses some recent DNS research work by himself and Roy Arendts from ICANN, about the observations of resolver behaviour across a root KSK roll-over. Their most recent blog article “The 2024-2026 Root Zone KSK Rollover: Updates and Observations” from July 2026 is a follow-up to their original article “The 2024-2026 Root Zone KSK Rollover: Initial Observations and Early Trends” from March 2025. It’s worth reading both articles to see how experience gained in the initial observations of the key rollover have informed subsequent research.

The DNS KSK at the root is the fundamental trust source for all subsequent DNSSSEC signed data in the global DNS. The processes around management of the private keys of this public-private key pair are described by IANA at their DNSSEC webpage and Verisign play several critical roles in the production, maintenance, distribution and use of the keying materials which come from this activity. Duane monitors this closely, both as a participant in the key ceremonies held under the auspices of ICANN/IANA, and from the vantage points Verisign has into global DNS from operations of the “J” root server anycast cloud.

Duane and Roy had the opportunity of re-analysing the behaviour of resolver systems under the key roll with greater clarity, because since the original key roll in 2017 a larger community of resolvers now use a signalling method defined in RFC8145 which indicates which trust anchors they see and are using. This has improved visibility of the uptake of the new trust anchor and shows very clearly in their time-charts of deployment of the new keying materials.

Key rolls in the DNS are not a “one and done” process, this is a multi-year activity which is about to reach it’s next most significant date later this month in October. On October the 11th, The new key will begin operations signing over the zone and the older keys will be withdrawn from signing. The new KSK has been published and listed in the root zone since January 2025, and was promoted by RFC5011 signalling in Feburary 2025.

In this episode of PING, APNIC Chief Scientist Geoff Huston and I continue a discussion about the repeat queries Geoff sees on the APNIC Labs experiment. Across time this number has changed, as DNS protocol usage has changed. We now have DNS records which didn't exist when the problem was first seen, and we therefore have multiple queries about multiple things.

Geoff has been exploring two facets of the DNS. Firstly, Exploring the effects how you answer "NO" has on the amount of queries you see, as well as the amount of work that has to be done by authoritative server, intermediate resolvers in the chain and the client. There are some surprising outcomes in the various choices of how to say no, including simply not answering.

Secondly, what impact would adoption of alternative DNS transports have on the repeats and query load, which may be a function of the UDP protocol and it's behaviours. Many people believe the overhead of maintaining TCP and related protocols in terms of state on the server was a reason to avoid it, but with the volume of uneccessary repeats rising, it is possible there is less work end-to-end, maintaining state for a protocol like TCP.

In this episode of PING, we talk with Willem Toorop from NLNet Labs and Ilyas Rahimi who recently completed a masters in Security and Network Engineering at the University of Amsterdam (UvA). They’ve been looking at the effects of local root serving.

Ilyas was on a masters programme called “OS3” at UvA (from the triplet of Open Standards, Open Software and Open Security). NLNet labs is on the campus of UvA and Willem has been working with the masters programme for some time, pursuing research in DNS and DNSSEC with the students doing their masters thesis component. They conduct 2 1-month original research activities in the masters course, and write the second one up as their masters thesis.

Locally served root is a model of operating a DNS resolver where it “pre-fetches” the state of the root zone, the apex of the entire DNS naming model. By fetching this data the idea is that both early termination of unresolvable domain names (names where the final terminal label can be seen not to exist under the root of the DNS) and a degree of privacy in fetching resolvable names can be gained, by looking in the local copy. There is a current proposal in the IETF to make this a “best current practice” or BCP.

Ilyas looked at 3 different public code-bases (bind, unbound and the knot DNS resolver) and a total of 4 configurations, exploring both in-band and HTTPS bound fetch of the root zone.

As well as identifying a bug in the model unbound used, which accounted for a spike in traffic, Ilyas has been able to show that under the update cycle typically seen for the root there is a surprising increase in apparent network traffic fetching the zone, compared to the many but far smaller events asking for uncached data from the roots.

There is more work to do in this space, and with the OS3 masters thesis component now given more time in the degree program, there is hope that Willem can continue to work with UvA students, and carry forward the measurements. WIllem has been exploring how the system behaves if converted to incremental updates (IXFR) rather than an entire zone-fetch.

In this episode of PING, APNIC Chief Scientist Geoff Huston and I discuss the behaviour of the DNS system when you have to come up from nothing: the “cold start” where no data exists from prior queries in your “cache”. This stems from a talk given by Ondřej Surý, that Geoff saw at the recent RIPE-92 Meeting held in Edinburgh, in the DNS working group.

Cache, is the things you hold on to, from prior work done. It’s a fundamental technique in computer science to avoid the massive disparity in speed between parts of the systems, keeping things you use a lot in “fast” memory close to the CPU, and avoiding having to go to “slow” memory or even worse disk or tape files, to find the data. The size of your cache and it’s speed can have a huge influence on the speed of your program. Comparing CPU performance with and without prior cached data can be very instructive to it’s benefits! -In the DNS, Cache is how you avoid having to wait for a remote system to answer over the network (with all the round-trip delay of question and answer) because you know parts of your answer from the prior queries and answers you hung onto.

“Cold Start” is a well known problem in large distributed systems. The problem is not just that industrial systems like coal plants and gas turbines need time to warm up and get sufficient energy to drive the generator, in the so-called “black start” there is the problem of energising the magnetic coils associated with generation of electricity. It’s distinct DC energy, used to make the generator enter the state where the rotational force from the turbine actually makes power. Without a source of power, you cannot make the coils “excite” and so the spinning generator won’t actually make any electrical power. Typically, this DC voltage comes from an independent source like a small diesel generator, or a source of electrical power which hasn’t been affected by the outage. If you contract to supply this contingency power, failure to be available is a major issue.

The DNS isn’t an electrical generation network, but it does have this kind of complex dependency, mapping from DNS fully qualified name, to the specific IP addresses bound to the name. The “D” part of DNS stands for “Domain” and the elements of the name in a sequence form “domains” of control (or at least potentially do) separated by the dots in the name which may have a distinct “name server” detailing how things are named under-neath that domain boundary. So a single question “what is www.potaroo.net” has the potential to invoke at least 3 if not more questions, for each “domain” in the name.

When you have a cache. The cache of known name-to-address pairs includes the data you need to know the addresses of the nameservers for each domain you have seen, which can tell you the name-to-address pairs for the other things you are asking: either directly or indirectly by passing you on to another system. But if you have no cache, when you ask (for example) the nameservers for .NET and are told they are A.NET and B.NET you are no wiser: how do you find these hosts, when you don’t know how to find .NET?

Geoff has been chasing down what he sees exploring this “cold start” behaviour, and what it tells us about how people chose to name their hosts and services, how they provide the “name servers” behind these hosts and services, and how intruding intermediaries, or even trying to work around the risks of a cold start can increase the query burden on the clients worldwide.

In this episode of PING, we talk with Job Snijders about secure Internet routing again, focusing on an approach to preserving the state of RPKI as a longterm historical record for research and analysis. Job was last on PING to discuss the “Erik” protocol.

Job has been running an archive of RPKI state for some time, as a volunteer activity but an exercise which was feasible when BGP speakers producing ROA objects was measured in the hundreds to low thousands is significantly more expensive when the population of ROA producers is a more realistic percentage of the around 80,000 AS holders worldwide.

Because Public Key Cryptography depends on a regular re-signing, and re-cataloging (as another signed object, the RPKI “Manifest” file) even when there is no substantive change in the state of signed information, a large amount of “churn” can be seen in the data, and even a compressed form of this state in turn incurs a huge overhead in storage of hard-to-compress data. Job reached the limits of his free activity, and started to explore a more compact and useful representation. This has emerged in the IETF standards process as two related activities. The Canonical Cache Representation or CCR, and the aggregation over this for data represented as the RPKI Spool data model for “materialising” the state of RPKI objects.

Along with his fellow IETF draft authors Bart Bakker , Tim Bruijnzeels, Theo Buehler and Fedor Vompe Job has managed to define a remarkably compact, highly compressible representation of RPKI validated objects, and the cryptographic payloads. These systems have been designed to leverage well known UNIX and other techniques for data management such as the “tar” format for spooled data, ASN.1 for binary encoding, Merkle Trees to hold compact hash state, and data compression using the zstd encoding.

It’s a nice approach to solving the scaling problem.

In this episode of PING, APNIC Chief Scientist Geoff Huston and I discuss a traffic behaviour in the DNS which Geoff has noticed in the labs advertising based experimental data capture. Virtually every DNS query Geoff sees, he sees twice (or more). For a cohort of about 150 million unique DNS labels on a given day, The Labs system is collecting 270 million incoming DNS queries. Thats a lot of duplication. What's going on?

The advertising data collection depends on issuing unique DNS queries, which in turn generate unique web page serves. This allows measurement of internet-wide behaviour on about 30 million browsers, games, devices every day. Not to say that these names and web URLs are not routinely seen by more than one entity, intermediary systems such as caches and proxies as well as re-visiting old open website tabs on a browser can cause this.

The point is that after the "first" fetch, the subsequent fetches can usually be held to be re-presentations of the same experiment and can therefore often be discarded (in the case of the web). But, for the DNS which has always had an element of unreliable transport, and which in turn invites measurement of features like DNSSEC which cause SERVFAIL messages, and demand repeated attempts to "find the DNS name-to-address mapping, re-fetching is itself something under test. How many resolvers lie behind a given users systems? How many kinds of resolver (DNSSEC enabled, or not) does the user depend on?

Finding that almost all DNS queries are repeated, and it turns out repeated very quickly invited Geoff to have a look at what's going on "under the covers". There are some patterns behind what's being seen which Geoff explored in the APNIC Labs Blog.

In this episode of PING, we talk with Willem Toorop about a measurement of 6 choices for naming the root servers of the global DNS. This measurement was motivated by RSSAC 028, a 2017 technical analysis for ICANN of the naming scheme used by the root DNS servers.

Root name servers are part of the global DNS system, and they provide basic bootstrapping for every other DNS resolver and authoritative name server worldwide. This is their primary role, but they also handle query load relating to the global DNS system all day and every day. There are 13 distinctly named root servers, each one identified by a letter from A to M under the domain root-servers.net. Although there are only 13 distinct "labels" there are in fact thousands of machines providing this service as independently operated clusters behind each of these letters, using the BGP "anycast" method we have discussed on PING before.

The special zone, root servers.net was instantiated in 1995. It's delegated as normal under the .net zone, which in turn delegates from the root "dot" zone. If you are bootstrapping a new resolver, to find a root server you appear to need to know about the delegation of .net in order to find root servers.net, to find the given server. This is a circular dependency because you don't know how to find .net until you have asked a root server instance. The circular dependency is resolved by the contents of an initial "priming" response. This response provides the additional "glue" information to seed direct knowledge of how to find each of these named root servers as is, without all the intermediate logic of the circular dependency.

This glue is inherently not signed in DNSSEC. It's insecure. The priming response is very carefully curated for both its size and the content but it would be good to give it some content security and increase trust in the message. RSSAC 028 explored the impact of applying DNSSEC over this domain, which in turn would add DNSSEC signatures to the priming response and impact a goal of keeping this a small 512 byte backwards compatible UDP packet.

By keeping this response small, it would continue to work on all legacy systems worldwide. Nothing happens quickly in DNS evolution, and this naming question has now been under consideration for almost a decade. Willem and his colleagues at NLNet Labs and SIDN produced two reports reflecting on the candidate naming schemes, application of DNSSEC, and the impact on the bootstrap fetch. There are some surprising outcomes!

In recent PING episodes, APNIC Chief Scientist Geoff Huston has asserted that Internet routing has shifted away from traditional IP packet forwarding. Instead, it is increasingly driven by processes that map names to addresses. It’s no longer just about your IP address or the specific endpoint you think you’re connecting to, it’s about your location and which intermediary services can most effectively handle your request.

How does this actually work in practice? What processes determine where your request is served from, and who makes those decisions? In the latest episode of PING, we explore this topic.

Of course, IP-level routing hasn’t disappeared. For optimizing content delivery using a ‘closest’ node model, anycast remains a critical technique, using BGP to direct traffic to the nearest available server, and it is widely deployed at scale. However, this approach is increasingly supplemented by name-based mechanisms, particularly those driven by DNS, to more precisely determine where requests should be directed.

The way endpoints are identified in an Internet protocol exchange is changing, and this shift has broader implications for the nature of the network. As a result, we are seeing some trends emerge:

Control over routing decisions for application content has shifted away from ISPs operating at the BGP layer and towards higher-level logistics functions in the stack. These decisions are now typically made by service providers offering optimized content delivery as a managed service.

Provision of this optimization is typically carried out through DNS-based mechanisms. As a result, organizations often delegate their DNS to the same content delivery provider, allowing it to control request routing. This gives the intermediary significant influence over how and where traffic for a domain is directed.

Together, these trends are likely to reinforce the growth of ‘walled garden’ vertical markets. Application-specific delivery methods are increasingly positioned as competitive advantages over generic services (such as standard video streaming), meaning that both application choice and DNS-based steering can guide users into more closed, vertically integrated environments.

In this episode of PING we’re hearing about secure Internet Routing and its data distribution problem from Job Snijders who has been on PING before talking about his measurements in BGP and RPKI.

We caught up at IETF125 in Shenzhen where Job presented to the SIDROPS working group on a new protocol he’s been designing, called Erik. The Erik protocol was named in honour of Erik Bais who died in May 2024. Erik was a stalwart of the RIPE routing community. He was a chair of the Address policy working group, and active in the Dutch cloud community and the data center association.

RPKI, the principal mechanism for determining secure inter domain routing intent (hence SIDR) depends on every relying party (or RP) validating the data collecting all the signed statements from all the publication points, worldwide. This is a time consuming process which inherently serialises behind the sequence of bytes fetched to form a given repository state at a publication point, and how the protocol works out whats changed since the last fetch by this user, and what to send. It’s not very efficient and it’s not scaling as well as we’d like as the amount of data rises, and the number of validators or RPs are fetching the data.

Job’s “Erik” protocol is designed to improve significantly on the two mechanisms defined at present, the RSYNC protocol, originally designed in the mid 1990s for filesystem synchronisation, and RRDP, a SIDR specific delta protocol which was designed to improve on rsync, using experience gained from the NRTM mechanism used to copy data in the RIR WHOIS databases.

Job has been able to find why RPKI fetch is slow, and design a protocol using the Merkle Tree mechanism which can significantly improve the collection delay, as well as allow for intermediaries such as CDN providers to host services in the cloud.

In this episode of PING, APNIC Chief Scientist Geoff Huston and I discuss Network Time Protocol or NTP. NTP is one of the older systems we depend on, designed and implemented by Dave Mills who died in 2024. Dave had been working on time synchronisation from the mid 1970s, and cared deeply about synchronising the emerging ARPAnet and the Internet, with the pre-existing worldwide collaborative framework which regulates our sense of time, and how it relates to the world of science, astronomy and civil society.

Geoff has been musing about NTP, moves to secure NTP, and the many dependencies in the modern world on the underlying concepts of a coordinated sense of time. This dependency in the modern world on highly synchronised clocks cannot be overstated, it creeps into every sector of daily life from aircraft and space navigation, to finance systems, and event scheduling of all kinds.

But our model of time is fundamentally based on the rotation of the earth, and the length of the second, and unfortunately while we now define the length of the second to astonishingly accurate levels, the rotation of the earth isn't as stable as we'd like. our model of time therefore has to make some adjustments

To make matters worse, our model of time has been coded over the years to varying models of a start date known as an "epoch", and how we represent time inside the machines, systems and services isn't one unified model any more. It's all coming a bit un-stuck.