Chrome (and derivatives) need to start behaving better:
> The motivation for the Google’s Chrome browser to do this is pretty obvious. Many ISPs perform NXDOMAIN substitution in their DNS resolvers and replace the “no such domain” response with a synthetic pointer to their own search page, which allows them to monetize all those No such domain” user typos. From Google's perspective this NXDOMAIN substitution is not well regarded. Search is a major path to advertisement placement and advertising is Google's core revenue. So how can Google's Chrome browser platform detect network environments where NXDOMAIN substitution is happening? Simple. Test the DNS with its own nonce random string queries to see if NXDOMAIN substitution is taking place.
> This supposedly innocuous probe of the DNS should’ve been pretty harmless. But there is a lot of Chrome out there these days. Some two thirds of all browser use in today’s Internet is reported to use the Chrome browser, and the number rises when you include products such as Edge that are based on the Chrome engine. Consequently, it is perhaps unsurprising to learn that according to this report these probes are now taking up some 50% of the total root server traffic (Figure 3).
I understand that hijacking NXDOMAIN is bad, but Google is now 'punishing' the root DNS operators with extra traffic when the actual perps are the bad-actor ISPs. A bit of collateral damage there.
And would it be so bad for Google/Chrome to simply let the NXDOMAIN hijacking slide? They seem to be externalizing costs to help their business model.
You need to know a bit of history, namely that one of the earliest "bad actors" here was Verisign, the operator of the com. and net. registry, and of the "A" and "J" ICANN root content servers.
It's only 1300 QPS per site. Is just plain old better programming on the table? An ordinary PC should be capable of barfing out NXDOMIAN responses at, conservatively, 100000 per second.
I'm looking into this a bit more for some basic math.
It looks like QPS is around 1.3M per second? Am I way off? What the bandwidth? I'm getting 1GBps or less which seems pretty low?
If the root can't handle this how does it handle DOS floods even?
Should we just turn some of the root over to some of the bigger players that can actually scale (AWS / GCP / Azure / Netflix etc)? These rates seem so small for this level of complaining about google et al.
Yeah, the conclusion I'm getting is, how do I sign up to be a root? Chrome's NXDOMAIN detection is good for the world and it sounds like handling 50% of root server traffic would not be that hard.
Digging a bit more it looks like some of just the public DNS servers are doing WAY more traffic than this. Google was doing 810K qps on 8.8.8.8 alone in 2012 (8 YEARs ago).
Can't they offer to operate some of the root zone then if it's so hard for these other folks to manage it (maybe do something with AWS / Azure etc).
Out of curiosity, how big (in GBs/TBs) are the root files? If I setup a local DNS server (Raspberry or something more powerful) could I duplicate or cache the root file somehow?
IMHO the various ISPs should contribute to the cost and technical needs of the root servers or even better utilize blockchain, torrent or other distributed network concepts.
That's in human-readable ASCII. A wire format could probably shave a chunck off that, but distributing deltas instead of re-downloading the whole file each time a change is made would further reduce the required data transfer amounts to mere bytes per day on average.
Thanks for your reply, I was thinking about the actual files containing the DNS entries and not the root file which contains all of the DNS servers. As I understand it the root file is needed in case you want to bootstrap your own DNS server so as to be able to find all the other authoritative servers. What about the actual DNS database files?
You are talking about the root seed list (the NS records for the root label .) - essentially the anycast IPv4/6 addresses for the 13 root servers. This file is provided in the first section of the linked page.
The second file on the page contains the actual current root zone, which is a 2 MB file containing all the delegation entries for TLDs (like com., org. and de.) immediately below the DNS root (.). This is what the article is talking about caching locally at your ISP's recursive name server.
The whole global DNS can never be downloaded or even built as a local database due to its distributed nature, and also for technical reasons. Below the root zone at TLD level (like com.) there is no way to enumerate all the individual registrations (like google.com., ycombinator.com., microsoft.com. etc) You always query for a specific individual name + record pair, instead of being allowed to download the entire zone. So even if you somehow would be able to contact every authoritative DNS server on Internet at some point in time, there is no standard query to ask them to give you all their records, making building a full local copy of the global DNS database impossible.
Anyone can serve their own root.zone. I have done so for many years. The web might be "big" but the portion of it you as a personal user are accessing is comparatively small. If you tally up all the domains you are currently visiting (excluding nonsense like ad servers), chances are, a zone file that contains the RRs for every domain you will ever need to visit in your entire lifetime would fit on a small-sized USB stick from 10 years ago. You can effectively avoid recursion. 100% authoritative DNS, all served from your computer.
Today's USB sticks can accomodate the largest zones like .com several times over. That is more web than you could possibly surf in your lifetime.
I wonder if the only change that needs to happen is that applications driving a significant amount of traffic (i.e. Chrome) & DNS owners of other domains need to start bearing the costs here. Then we wouldn't be dealing with cost externalities & tragedy of the commons. On the other hand, introduce revenue streams & you start focusing on optimizing the revenue stream instead of solving the problem.
I wonder if there's a different way to make the costs felt that isn't directly revenue to align incentives. Like slow down or fail queries for zones who's authoritative servers don't implement the features needed to keep scaling under control. Then make sure that there's revenue to the root servers from non-root servers based on load so that costs are tracked & managed.
Just let google run some of the roots - I'm serious. For all the whining about bandwidth and traffic, this issue is so small on the scale of google (which includes youtube) and netflix traffic flows it's hard to even imagine.
Their secondary resolvers run at huge volumes, everything from 8.8.8.8 to the dns for their mail and other web properties.
A root is not one static machine. Google can serve 1/3 of traffic for all routes, AWS could do the same.
I don't think folks realize how SMALL the root dns traffic is. Youtube / netflix / amazon prime video are just orders and orders of magnitude more demanding in terms of bandwidth - every second they are pumping MULTIPLE HD streams to millions of folks. They measure bandwidth terrabits per second. Even at 30 frames per second, we are talking incredible bandwidth.
DNS responses should be 1K or less (often .5K). About 40 netflix viewers at their top quality use the same bandwidth? Netflix has far more than 40 users.
You think the root operators should just delegate part of the traffic of all letters' VIPs to google and amazon? It's not crazy, but it would mean the entire root would have a shared-fate failure mode.
Yeah, and it makes perfect sense for these big tech cos to run all or some of one root letter. But it violates the spirit of resilience to have one company participate in multiple letters.
Someone at Google pushes yet another invalid config of death, taking out their global network and, with this proposal, also taking down 1/3rd of all DNS root capacity.
I'm just pointing out that for all the complaining about google efforts to block domain jacking (which was endemic before they started their efforts) scaling the root is doable, and if the existing maintainers are too incompetent to scale it then maybe others should be allowed to.
I mean, how do they handle denial of service attacks which are going to be orders of magnitude higher in volume?
Rather than the solution being to allow for domain hijacking by turning off the redirect detectore, how about verisign, which kicked this whole thing off recently AND way back with their "sitefinder" garbage get a clue and either propose another solution to domain jacking or scale the root a bit.
You've conflated the proxy DNS servers, that do grunt work of query resolution (or forward to other proxies that do), with the content DNS servers. It is not the content DNS servers that are causing the scaling problems, it is the proxy DNS servers, and the DNS clients and applications softwares that drive them.
The change that needs to happen is the thing that most people nowadays tell one first, that this paper in contrast leaves all of the way to the end: run your own content DNS server(s) for the root, on your own LAN or Intranet. It's fairly simple to do, and it's an idea that has been around for over two decades, now.
It takes a long time in the DNS world for such grassroots ideas to percolate to the level that they get circulated by corporate officers. This article is, for example, still propounding that every resolving proxy DNS server works like ISC's BIND, and has a "priming query". This is untrue in reality, and is one of the differences between BIND and some of the softwares (e.g. dnscache and MaraDNS) that came along to replace it a couple of decades ago.
Similarly the argument that Google should use a subdomain, presented in this article without any counterpoint at all, has long since been countered by pointing out that this makes the malicious actors' lives easier; as conditional forwarding and split-horizon DNS service, which are well-known things, would then make it easier to fool Google Chrome. And there have been other counterarguments made, years ago.
These aren't types of query, they are types of server. viovich123 was talking about content DNS servers that cause the scaling problems, and penalizing the people who run them. But it is the proxy DNS servers, and the DNS clients and applications softwares that drive them, that are the cause here.
The terms are correct, and are derived from HTTP. You aren't grasping them correctly.
> The terms are correct, and are derived from HTTP. You aren't grasping them correctly.
This, I think, is the source of the problem. DNS servers have their own terminology that pre-dates HTTP. The servers that take/do lookup requests and cache them are known as recursive servers. Yes, they act similarly to HTTP caching proxies, but that's not how they are referred to. These make up the vast majority of DNS servers.
The other type is authoritative, which is called that as it is the 'athoritative'source of truth for some domain's DNS data. Where the root servers look it up to feed into the system. These would be akin to your basic HTTP servers, but again the terminology is different.
This is the source of the problem. People don't understand the DNS terminology. Very few people indeed get "resolver" right, for example, and they only then end up confusing the rather larger number of people who get it quite wrong. Similarly, "authoritative" confuses people when they have to try to understand why it isn't actually the same as an "authoritative answer". Whereas they do understand the idea of DNS records being content, that is published by content servers.
Hence why many years ago I took to explaining DNS using HTTP terminology. It works quite well.
I can see where you're coming from, as it does work as a close metaphor and people are generally more familiar with HTTP. I think the disconnect is that while that is a good way to relate it, seems like teaching the correct terminology along with the basic idea is generally better as it also gives them the tools (eg. as search criteria) to research it more themselves.
Long experience shows that it doesn't. It just adds layers of obfuscation to the subject. You spend all of the time explaining what a "resolver" is, for starters, and then have to explain how that model then doesn't really match how the softwares fit together on their operating system and LAN, because it was intended to be an operating system neutral model. And then you get to explain that almost no-one will understand you correctly if you use the term properly.
Whereas explain to them that it's a form of publication, from content servers through proxies to client libraries in applications, like HTTP, and it generally clicks what part of the overall system talks to what other parts. People have some absurd models of how things work, from ISPs talking to ISPs to relays along chains of content servers.
Explaining that it's actually like HTTP gets people using the correct mental model, and matches how the softwares in reality fit together. There really is a DNS client library, and it has a name such as (for example) libresolv. There really is a proxy DNS server, and one can point to a specific process running its program, again with names (dnscache, pdns-recursor, et al.), to its back-end and front-end traffic, and to whether it is configured to forward or to resolve. There really are content DNS servers, again with nameable programs (nsd, tinydns, walldns, pdns_server, et al.), with specific locations, and with content sources that one can point to.
The fact that RFC 7719 exists should tell one something, even if it weren't spelled out in its introduction that the RFC terminology is confusing, contradictory, and going to change wholesale "real soon now".
Here's a mental exercise:
Configure your machine with 1.1.1.1 in /etc/resolv.conf. The RFCs, including RFC 7719, tell you both that your "resolver" is the program that does the entire job of "resolution" and that it is located on the same machine as the program requesting the lookups. You have a DNS client library locally, linked into your applications, that is talking to a resolving proxy DNS server running on a machine operated by Cloudflare. The resolving proxy DNS server is doing all of the grunt work of query resolution, retrieving content from content DNS servers with back-end transactions, and Cloudflare's machine is where its front-end and back-end are. It's simple to explain that way. Now point to your "resolver". Then further try explaining it to someone else. Then try explaining that Cloudflare's server is a DNS server, but not a "name server". See what happens when you tell them that for an explanation of "name server" they just need to use RFC 7719 to "research it more".
No, content and proxy are types of server. There's no such thing as a "content query" nor a "proxy query".
And DNS very much is content. The content is a whole bunch of resource record sets. Published by content servers, much as in HTTP. And it isn't all name to address stuff.
How about a system for root zone distribution akin to rpki using bittorrent where updates are signed by a trusted key? Then any resolver could easily be a root server.
My knowledge is limited here but at some level there is a limit to the amount of traffic a single endpoint can handle right? For example if something is distributing queries in a round robin manner, isn’t there a limit on how much load that can ultimately handle and distribute to other computers? What is that limit and what constrains it?
Moving the DNS root to blockchain is one of the best use cases for blockchain [1] given the numerous stakeholders (name owners) and type of information required to be in consensus. This instantly serves to distribute DNS root queries, instantly scaling the root.
If one of the best use cases for blockchain is "data about name owners" and "DNS information", the future of blockchain is bleaker than I'd imagined.
Handshake isn't a super-persuasive example to cite, since it's backed by a minor altcoin that's primary use case (according to CoinDesk) is crypto day-trading.[1][2]
If only there were some technology that could verify whether a DNS query/response had been spoofed, without interfering with caching and proxying, and without the privacy and centralization-of-control problems of DoH. If only such a technology had already been widely deployed in the dns root and many caches and resolvers.
> The motivation for the Google’s Chrome browser to do this is pretty obvious. Many ISPs perform NXDOMAIN substitution in their DNS resolvers and replace the “no such domain” response with a synthetic pointer to their own search page, which allows them to monetize all those No such domain” user typos. From Google's perspective this NXDOMAIN substitution is not well regarded. Search is a major path to advertisement placement and advertising is Google's core revenue. So how can Google's Chrome browser platform detect network environments where NXDOMAIN substitution is happening? Simple. Test the DNS with its own nonce random string queries to see if NXDOMAIN substitution is taking place.
> This supposedly innocuous probe of the DNS should’ve been pretty harmless. But there is a lot of Chrome out there these days. Some two thirds of all browser use in today’s Internet is reported to use the Chrome browser, and the number rises when you include products such as Edge that are based on the Chrome engine. Consequently, it is perhaps unsurprising to learn that according to this report these probes are now taking up some 50% of the total root server traffic (Figure 3).
I understand that hijacking NXDOMAIN is bad, but Google is now 'punishing' the root DNS operators with extra traffic when the actual perps are the bad-actor ISPs. A bit of collateral damage there.
And would it be so bad for Google/Chrome to simply let the NXDOMAIN hijacking slide? They seem to be externalizing costs to help their business model.