In the final podcast for 2025, APNIC Chief Scientist Geoff Huston discusses the problem of independent measurement in an Internet which is increasingly “going dark”.

Communications has always included a risk of snooping, and a matching component of work to enhance privacy, from the simplest ciphers used in ancient times, techniques of hiding and discovering messages, attempts to prevent and detect intrusion of the mail, to adoption of telegraph codes, the cutting of telegraph wires in wartime (to force messages into radio where they could be listened to) and the development of modern encryption algorithms typically using the public-private keypair model. There has always been a story of “attack” and “response” in how we communicate privately.

Aside from matters of state security, banking and finance at large depend on a degree of privacy and now require it under legislation to enable use of creditcard information online. Many other contexts have an assumption of privacy, and use technology to try and preserve it. Fundamentally, individuals in their use of the Internet are entitled to expect a level of privacy where the state permits it.

The publication of RFC7258 “Pervasive Monitoring Is an Attack in 2014 formalised a belief that the intrusion of third parties into a communication between two ends demanded a technology response to exclude them, where possible. Protocol designers and Internet Engineers took up the challenge.

This position led over time to a marked increase in the adoption of privacy enhancing protocol features. For example, the web moved from HTTP: denoted URLs to HTTPS: where the content is protected by the Transport Layer Security (TLS) encryption protocol, which now overwhelmingly predominates in the at large.

However, significant aspects of Internet communications “leak” information to third parties. Between an individual and a web service lies their provider, unknown numbers of intermediate providers, typically a content distribution system hosting the web site in a local copy, all of whom have opportunities to see and understand what is being done, and by whom. In particular the DNS typically exposes the name and address of the site being connected to across all kinds of protocols (not just the web) and exposes it to unknown intermediary systems as the DNS lookup is processed.

In response to this, services are emerging which break down the DNS into dissociated queries: what is being looked for, and who is looking for it, and use intermediary services which may know one, but not both: Questions are seen to be asked, but by who is now hidden. If you know who is asking, you don’t know what they are asking for.

Combined with newer network protocols like QUIC which imposes a strong end-to-end encryption model which even hides the inter-packet size and timing information (another form of leak which can be used to reconstruct what kind of traffic is flowing) it has become increasingly hard for an independent researcher to see inside the network: It’s going dark.

Geoff explores the nature of privacy in the Internet at large, and how APNIC Labs gets round this problem with it’s measurement system.

PING will return in January 2026 with another season of episodes. Until then, enjoy this final recording of 2025, and see you online, in the new year.

This time PING features Emile Aben from the RIPE NCC R&D Department. Emile is a Senior Research Engineer, and for over a decade and a half has been looking at Internet Measurement at RIPE in the Atlas system, and in the RIPE RIS BGP data collection.

Emile and a collaborator Romain Fontugne from IIJ Labs in Tokyo have been exploring a model of the influence and effect on global connectivity in BGP for different AS, based on the impact they have on other AS’s transit choices. They call this “AS Hegemony” and Emile has been using it to adjust for sample bias in the data being collected in RIPE RIS and in the Atlas network. This approach to re-balancing the sources helps Emile to understand changes in network topology and routing under rapid shocks like cable cuts and he’s been applying this to the recent spate of cable outages in the Baltic, around Africa, and the power outage on the Iberian Peninsula.

Emile has also been looking at new ways of holding data, and visualising data. His RIPE colleague Ties de Kok has explored use of “Parquet” as a data abstraction tool and this has allowed Emile to perform rapid analysis and experiment in new data visualisations in 3D, rendered in-browser.

Read more about AS Hegemony, and the new data visualisations on the web:

• How RIS Saw the Iberian Power Outage (RIPE90 plenary presentation, May 2025)
• A Deep Dive Into the Baltic Sea Cable Cuts (RIPE Labs, December 2024)
• AS Hegemony: A Robust Metric for AS Centrality (SIGCOMM 2017 poster)

In this episode of PING, APNIC Chief Scientist Geoff Huston explores the complex landscape of undersea cables. They have always had a component of strategic interest, communications and snooping on communications has been a constant since writing was invented, and the act of connecting two independent nation states by a telegraph wire invokes questions of ownership and jurisdiction right from the start.

After the initial physics of running a long distance wire to make an electric circuit was worked out, telegraph services became a vital part of a states economic and information gathering processes. This is why at the beginning of world war 1 and again in world war 2 the submarine cables linking europe out into the world were cut by the British Navy: forcing the communications flows into radio meant it was possible to listen in, and with luck (and some smart people) decode the signals.

Modern day fibre optic communications are no different in this regard. Many incidents of cable cutting have simple explanations, not all paths the subsea cables run through are especially deep and in shallow waters near landfall with lots of fish, trawlers cause a lot of damage. But there is now good reason to believe state actors are also disrupting fiber communications by breaking links, and a strong trend now to direct which sources of equipment (from the physical fibre up to the active routing systems) are used for a landfall into any given economy. This in turn is influencing the flow of capital, and the paths taken by subsea fibre systems, as a result of the competing pressures.

In this episode of PING, Shumon Huque from Salesforce discusses how protocols with extensible flag fields can benefit from regular testing of the values possible in the packet structure. This technique is known as "greasing" and has a strong metaphorical meaning of "greasing the wheels" to ensure future uses aren't blocked by mistaken beliefs about the possible values.

Intermediate systems (so-called "middleboxes") have to try and determine "risky" packetflows, and one of the mechanisms they use is to consider unexpected values in the known packetflows as possibly dangerous. This is an over-simplistic approach, and risks "ossifying" a protocol into the range of values which are actively in use now. Protocols usually include extra potential values for flag-fields, settings, options and the like, and these frequently have a large range of "reserved" values which are held in trust in an IANA registry, for future use. Greasing is a proposed mechanism to test out some of these values, and see what happens "on the wire" for the protocol in question.

Shumon and his co-author and collaborator Mark Andrews from ISC have been applying the greasing model to the DNS, and we talked about it's history in other protocols, and how in practice greasing can be applied on the global internet.

Read more about Shumon, Mark and Roy Arends' greasing activity on the web:

• DNS Grease (IETF draft, in the IETF Datatracker)
• the TLS DNSSEC Chain Extension ( IETF DANE WG, IETF RFC):
• DELEG Testing Report (with Roy Arends, DNSOP WG interim meeting presentation, IETF)

In this episode of PING, APNIC Chief Scientist Geoff Huston discusses a problem which cropped up recently with the location tagging of IP addresses seen in the APNIC Labs measurement system. For compiling national/economic and regional statistics, and to understand the experimental distribution into each market segment, Labs relies on the freely available geolocation databases from maxmind.com, and IPinfo.io -which in turn are constructed from a variety of sources such as BGP data, the RIR compiled resource distribution reports, Whois and RDAP declarations and the self-asserted RFC8805 format resource distribution statements that ISPs self publish.

At best this mechanism is an approximation, and with increasing mobility of IP addresses worldwide it has become harder to be confident in the specific location of an IP address you see in the source of an internet dataflow, not the least because of the increasing use of Virtual Private Networks (VPN) and address cloaking methods such as Apple Private Relay, or Cloudflare Warp (although as Geoff notes, these systems do the best they can to account for the geographic distribution of their users in a coarse grained “privacy preserving” manner).

Geoff was contacted by Ben Roberts of Digital Economy Kenya, a new boardmember of AFRINIC and long-time industry analyst and technical advisor. He’d noticed anomolies with the reporting of Internet statistics from Yemen, which simply could not be squared away with the realities of that segment of the Internet Economy. This in turn has lead Geoff to examine in detail the impact of Starlink on distribution of internet traffic, and make adjustments to his measurement Geolocation practices, which will become visible in the labs statistics as the smoothing functions work through the changes.

Low Earth Orbit (LEO) Space delivery of Internet has had rapid and sometimes surprising effects on the visibility of Internet worldwide. The orbital mechanics mean that virtually the entire surface of the globe is now fully internet enabled, albiet for a price above many in the local economy. This is altering the fundamentals of how we “see” Internet use and helps explain some of the problems which have been building up in the Labs data model.

Read more about Geolocation and Starlink on the APNIC Blog and on the web:

• Geolocation and Starlink (Geoff Huston, APNIC Blog September 2025)
• RFC8805 A Format for Self-Published IP Geolocation Feeds (IETF RFC website)
• The NRO RIR Statistics on delegations with geographic tagging of the delegated entity (NRO Website)
• Maxmind GeoIP resources (maxmind website)
• IPinfo.io (IPinfo website)
• Labs statistics portal (APNIC Labs website)

RSSAC047 - a document from the Root Server System Advisory Committee proposed a set of metrics to measure DNS root servers, and the DNS root server system as a whole. the document was approved in 2020, and ICANN worked on an implementation of the metrics as code, and a deployment into 20 points of measurement distributed worldwide.

ISC and Verisign, two of the root server operators proposed a review of this measurement and retained SIDN Labs (who are part of the Dutch body operating .NL as a CountryCode Top-Level Domain or ccTLD) to look into how well the measurement was performing.

In this episode of PING, Moritz Mullër from SIDN Labs and Duane Wessels from Verisign respectively, discuss this "measurement of the measurement" exercise, what they found out, and what it may mean for the future of metrics at the DNS Root.

It's an interesting "meta conversation" about measuring things which themselves are measurements. We see this all the time in the real world, for example diagnostic imaging machines designed to measure bone density (for osteoporosis checks) require calibration, and when you want to compare a baseline over time that calibration and the specific machine become questions the clinician may want to check, assessing the results. Change machine, you get different sensitivity. So how do you line up the data?

Moritz's investigations show that in some respects, the ICANN implementation of RSSAC047 was incomplete, and didn't tell an entirely accurate story about the state of the DNS Root Server System. Also, there are questions of scale and location which means a re-implementation or future improvement is worth discussing.

Read more about the DNS Root Server System, Moritz's report, and the RSSAC on the APNIC blog and on the web:

• Root-Servers.org website
• Monitoring highly distributed DNS deployments: Challenges and recommendations (APNIC Blog)
• RSSAC047: RSSAC Advisory on Metrics for the DNS Root Servers and the Root Server System (ICANN)
• SIDN Labs
• Verisign Labs

In this episode of PING, APNIC Chief Scientist Geoff Huston shares a story from the recent AusNOG in Melbourne and connects it to measurement work at APNIC Labs, exploring how modern IP flow control manages ‘fair shares’ of the network.

At AusNOG 2025, Geoff attended a talk by Lincoln Dale of Amazon AWS titled “No Packet Left Behind: AWS’s Approach to Building and Operating Reliable Networks”. The presentation examined how AWS scales its data centre networks, highlighting massive investments in high-speed routers and switches to support both global internet services and the vast flows of traffic between servers and other Amazon resources.

What AWS doesn’t do is rely on highly complex protocols like Segment Routing over IPv6 (SRv6), Resource Reservation Protocol (RSVP), or other modern traffic engineering techniques unless absolutely necessary. Instead, they use a radically simplified, on-chip model of data management, pushing as much processing as possible into a single VLSI circuit and minimizing the amount of ‘smart’ work in the network. The question is: How can simplifying the IP stack to this extent actually work?

Geoff has long been sceptical of higher-layer protocols that try to manage bandwidth reservation and shaping. He recalls an earlier attempt by Digital Equipment Corporation (DEC) to signal congestion with Explicit Congestion Notification (ECN), a mechanism that still exists in the protocol stack and now underpins new bandwidth management approaches such as Apple and Comcast’s ‘L4S’.

APNIC Labs has measured how the wider Internet responds to ECN signals using an advertising-based model, and the results suggest this approach struggles outside tightly controlled, ‘walled garden’ networks. He contrasts this with advances in flow control through Google’s BBR, now in its third version, which refines the aggressive, bandwidth-seeking behaviour of TCP window management.

Read more about the story of IP, flow control and the modern Internet on the APNIC Blog, and the AusNOG website (video recordings of Lincon Dale’s talk and others should be released shortly)

• Measuring Explicit Congestion Notification (ECN) (Geoff Huston, APNIC Blog)
• Notes from AusNOG 2025 (Geoff Huston, APNIC Blog)
• The AusNOG 2025 program (AusNOG Website, videos to be released shortly)
• 
  

In this episode of PING, Adli Wahid, APNIC's Security Specialist discusses the APNIC honeypot network, an investment in over 400 collectors distributed throughout the Asia Pacific, collecting data on who is trying to break into systems online and use them for malware, destributed denial of service, and command-and-control systems in the bad traffic economy.

Adli discusses how APNIC Members can get access to the results of honeynet traffic capture coming from their network address ranges, and originated from their AS in BGP using the DASH system. and explores some work planned for the APNIC Honeynet systems to extend their systems coverage.

As well as publishing reports on APNIC's Blog and presenting at NOG meetings and conferences, Adli has coordinated information sharing from this collector network with a range of international partners such as the Shadow Server Foundation. He continues to offer training and technical assistance in security to the APNIC community and works with the CERT, CSIRT and FIRST community at large.

Read more about Honeypots, bad traffic and systems security on the APNIC Blog and the web:

• Blogs on the honeynet (APNIC Blog)
• Adli's posts on the APNIC Blog
• The APNIC Dashboard for AS Health (DASH) (requires an APNIC member account)
• The Shadow Server Foundation dashboard

In this episode of PING, APNIC’s Chief Scientist, Geoff Huston, discusses the economic inevitability of centrality, in the modern Internet. Despite our best intentions, and a lot of long standing belief amongst the IETF technologists, no amount of open standards and end-to-end protocol design prevents large players at all levels of the network (from the physical infrastructure right up to the applications and the data centres which house them) from seeking to acquire smaller competitors, and avoid sharing the space with anyone else.

Some of this is a consequence of the drive for efficiency. A part has been fuelled by the effects of Moore’s law, and the cost of capital investment against the time available to recover the costs. In an unexpected outcome, networking has become (to all intents and purposes) “free” and instead of end-to-end, we now routinely expect to get data through highly localised, replicated sources. The main cost these days is land, electric power and air-conditioning. This causes a tendency to concentration, and networks and protocols play very little part in the decision about who acquires these assets, and operates them.

The network still exists of course, but increasingly data flows over private links, and is not subject to open protocol design imperatives.

A quote from Peter Thiel highlights how the modern Venture Capitalist in our space does not actively seek to operate in a competitive market. As Peter says: “competition is for losers” – It can be hard to avoid the “good” and “bad” labels talking about this, but Geoff is clear he isn’t here to argue what is right or wrong, simply to observe the behaviour and the consequences.

Geoff presented on centrality to the Decentralised Internet Research Group or DINRG at the recent IETF meeting held in Madrid, and as he observes, “distributed” is not the same as “decentralised” -we’ve managed to achieve the first one, but the second eludes us.

Read more about the policy issues of the modern Internet at the apnic labs blog, the DINRG (IETF) and APNIC Blog

• Decentralizing Services? (Geoff Huston, talk to DINRG IETF123 Madrid)
• Centralization topics at the APNIC Blog
• DINRG at the IETF Wiki (IETF web page)

In this episode of PING, Robert Kisteleki from the RIPE NCC discusses the RIPE Atlas system -a network of over 13,000 measurement devices deployed worldwide in homes, exchange points, stub and transit AS, densely connected regions and sparse island states.

Atlas began with a vision of the world at night -a powerful metaphor for where people are, and where technology reaches. Could a measurement system achieve sufficient density to "light up the internet" in a similar manner? Could network measurement be "democratized" to include internet citizens at large?

From it's launch at the RIPE 61 meeting held in Rome Italy. with 500 probes based on a small ucLinux device designed as an ethernet converter, to 5 generations of probe hardware and now a soft probe design which can be installed on linux, and an "anchor" device which not only sends tests but can receive them, Atlas has become core technology for network monitoring, measurement and research.

Rob discusses the history, design, methodology and futures of this system. A wonderful contribution from the RIPE NCC for the community at large.