Sign In
Access your IPWhois.net account
No account? Create one
DNS

How One Routine Maintenance Error Took Down Germany's Internet (The .de DNSSEC Outage Explained)

Ivan May 18, 2026 13 min read 31 views
How One Routine Maintenance Error Took Down Germany's Internet (The .de DNSSEC Outage Explained)

On the night of May 5, 2026, a large part of the German internet simply stopped working. Not slowed down. Not degraded. Gone. Amazon.de returned errors. Deutsche Bahn's ticketing system was unreachable. Banking apps like N26 failed to connect. eBay, Web.de, government portals, mainstream news sites, all of it dark for users across Germany, Switzerland, Italy, and Sweden.

There was no cyberattack. No cable cut. No data center fire. No server crash. The websites themselves were running perfectly fine the entire time. What broke was something most people have never heard of and never think about: the cryptographic signatures that prove a domain name answer is authentic.

This is the story of how a single faulty piece of code, deployed during routine maintenance, made millions of domains vanish from the internet for roughly three hours. It is also a useful lesson in how the modern internet is held together by systems that fail in dramatic, all-or-nothing ways, and what you as an ordinary user can actually do about it.

What happened on the night of May 5, 2026

Here is the timeline, reconstructed from DENIC's own post-incident analysis, Cloudflare's incident report, and outage monitoring data.

Around 19:30 UTC (21:30 in Germany), DENIC, the registry operator that runs Germany's .de top-level domain, began publishing incorrect DNSSEC signatures for the .de zone. This happened during what was supposed to be a routine cryptographic key rollover.

Within minutes, every DNS resolver on the planet that validates DNSSEC (and that includes Cloudflare's 1.1.1.1, Google Public DNS, Quad9, and countless ISP resolvers) started receiving these broken signatures. The DNSSEC specification is strict about what to do with a signature that does not check out: reject it completely and return an error called SERVFAIL.

Over the next three hours, the outage spread like a slow wave. This part confuses people, so it is worth explaining. The breakage was not instant for everyone, because resolvers cache DNS answers. A domain whose answer was already cached kept working until that cache entry expired. As each cached record aged out and the resolver went back to DENIC for a fresh copy, it got a broken signature and that domain went dark. So the outage got progressively worse for about three hours as more and more cached records expired.

At roughly 21:57 UTC, DENIC's engineers detected the problem on thier signing infrastructure.

At 22:17 UTC, Cloudflare deployed a mitigation (more on what that was below).

At 00:08 on May 6, DENIC began distributing a corrected DNS zone.

By 01:15 on May 6, normal operations were fully restored.

The .de TLD has about 17.9 million registered domains. It is consistently one of the most queried country-code domains on the entire internet. An outage at this level of the DNS hierarchy is roughly the digital equivalent of an entire country's phone book briefly becoming unreadable.

4gWhude DNSSEC incident. AI grok image

 

First, what DNSSEC actually is (in plain English)

To understand why this was so catastrophic, you need to understand two pieces of internet plumbing. The first is DNS. The second is DNSSEC.

DNS (Domain Name System) is the internet's phone book. When you type example.de into your browser, your computer does not know where that is. It asks a DNS resolver, "what is the IP address for example.de?" The resolver does some lookups and comes back with an answer like 192.0.2.45. Your browser then connects to that address. This happens billions of times a second across the internet and you never notice it.

The problem with classic DNS is that it was designed in a more trusting era. There was originally no way to verify that the answer you got back was actually the real answer and not something an attacker injected. DNS hijacking, where a malicious party feeds you a fake IP address to send you to a phishing site, is a real and serious threat.

DNSSEC (DNS Security Extensions) was created to fix that. It adds a cryptographic signature to DNS answers. Think of it like a tamper-evident seal on a medicine bottle. When a resolver gets a DNSSEC-signed answer, it checks the signature against a seperate chain of trust that goes all the way up to the root of the internet. If the signature is valid, the answer is genuine. If the signature does not check out, the resolver assumes the answer has been tampered with and refuses to use it.

That last part is the key to understanding May 5. DNSSEC is designed to fail closed. If something looks wrong with the signature, it does not shrug and let you through anyway. It blocks the connection entirely. This is intentional. The entire point of DNSSEC is to never let a potentially forged answer reach you.

Why a single error took down an entire country's domains

Here is the chain of cause and effect.

DENIC signs the entire .de zone with cryptographic keys. Periodically, those keys have to be rotated for security reasons, a process called a key rollover. This is routine. Registries around the world do it regularly, and it is largely automated.

In April 2026, DENIC had deployed the third generation of their DNSSEC signing system. This system uses standard software (a DNS server called Knot) combined with their own in-house code and Hardware Security Modules for the cryptographic operations. According to DENIC's post-mortem, the new system was tested in advance and externally audited before going live.

But there was a flaw. A faulty piece of code in the in-house development was not fully covered by the test scenarios. It passed the test runs. It even passed "cold" parallel operation before commissioning. The defect simply was not in any of the scenarios anyone thought to test.

When the key rollover ran on May 5, that faulty code generated signatures that did not match the keys. Specifically, resolvers hit a malformed signature on an NSEC3 record tied to key tag 33834. NSEC3 records are part of how DNSSEC proves that a domain does not exist (used to prevent attackers from faking nonexistent subdomains). Because the broken signature was on this foundational record, it did not just break individual signed domains. It broke the delegation information for the entire zone.

This is the detail that made it catastrophic. Even .de domains that do not use DNSSEC at all became unreachable, because the broken signature was at the zone level, not the individual domain level. To a validating resolver, the whole .de zone looked compromised.

Only about 3.6 percent of .de domains are individually DNSSEC-signed (per ICANN data), which sounds small. But because the failure was at the zone delegation level, the practical impact was far broader than 3.6 percent. Major services across the German internet went down regardless of whether they personally used DNSSEC.

Why some people stayed online and others didn't

This is the part that matters most for you personally, and it is the part almost no news coverage explained well.

Not everyone lost access at the same time, and some people barely noticed anything. Whether your internet worked that night depended almost entirely on which DNS resolver you were using.

If you used Cloudflare (1.1.1.1), Google (8.8.8.8), or Quad9 (9.9.9.9), you were hit hard. These resolvers enforce DNSSEC validation strictly. When they saw the broken signatures, they did exactly what the protocol told them to do: reject the answers and return errors. Ironically, the most security-conscious DNS providers gave their users the worst experience that night, because they were following the rules correctly.

If you used a less strict ISP resolver that does not fully validate DNSSEC, you might not have noticed anything at all. Those resolvers saw the broken signature and effectively let it through, because they were not checking. Technically less secure, but on the night of May 5, those were the users who could still check their bank balance and buy train tickets.

There is a real tension inside the security community over this, and the .de incident put it back in the spotlight. The resolvers doing the "right" thing security-wise produced the worst availability outcome. The resolvers doing the "wrong" thing kept their users online. Neither answer is obviously correct, and the tradeoff is genuinely hard.

If you want to see which resolver you are actually using right now and whether it validates DNSSEC, you can run a quick dns leak test. It shows you exactly which DNS servers are answering your queries, which is the single biggest factor in whether an incident like this affects you.

How Cloudflare and Google limited the damage

The big resolver operators did not just sit there returning errors for three hours. They deployed two specific mitigations that are worth understanding because they show how the modern internet absorbs these shocks.

Serving stale (RFC 8767). Normally a DNS resolver discards cached answers when they expire. But there is a formalized behavior called "serve stale" that says when upstream resolution is failing, a resolver is allowed to keep serving expired cached records rather than returning an error. Cloudflare's 1.1.1.1 did exactly this. For .de domains that were still in cache, it kept serving the old (correct) answers past their official expiry, which significantly cushioned the impact and bought time. This is why the outage ramped up gradually over three hours instead of hitting everyone instantly. Domains with longer cache lifetimes survived longer.

Negative Trust Anchors (RFC 7646). This is the bigger hammer. A Negative Trust Anchor is an explicit, temporary exception that tells a resolver "stop validating DNSSEC for this specific zone." Cloudflare deployed an NTA for the .de zone at 22:17 UTC, roughly three hours after the incident began. Once that was in place, 1.1.1.1 stopped checking .de signatures and started returning answers normally again, broken signatures or not. Google and other operators applied similar measures. This is essentially the emergency brake: deliberately turning off the security feature that was causing the outage, accepting the small risk, to restore service.

Both of these are interesting because they are admissions that strict DNSSEC enforcement, while correct in theory, is sometimes operationally unworkable. The internet's biggest DNS operators have built in deliberate ways to violate the DNSSEC specification when following it would cause more harm than good.

What DENIC's post-mortem revealed

DENIC was relatively transparent, publishing an analysis within days. The key findings:

The new third-generation signing system, deployed in April 2026, contained a code defect in the in-house portion (not the standard Knot software). The defect was in a code path that the test scenarios did not exercise. It survived testing, external audit, and cold parallel operation undetected.

The broken signatures over NSEC3 records caused the entire delegation to be classified as bogus by validating resolvers, which is why even non-DNSSEC .de domains were affected. The failure occured at the zone delegation level, not at individual domains.

DENIC thanked the large resolver operators who deployed Negative Trust Anchors, explicitly acknowledging that those operators temporarily disabling DNSSEC validation is what mitigated the impact for most users.

DENIC paused future key rollovers pending a full post-mortem.

One detail worth highlighting: the faulty signature was technically "formally valid" until May 19, 2026, in the sense that the signature's own validity window had not expired. The fix required DENIC to specifically re-sign the affected records with correct signatures, not just wait for the bad ones to age out. This is why it required active intervention rather than simply resolving itself.

What this means for you

The uncomfortable truth is that if a TLD registry pushes broken signatures, there is nothing you as an end user can do to fix it. You cannot patch DENIC's signing system from your laptop. But there are practical takeaways.

Know which DNS resolver you use. This is the single biggest factor in how incidents like this affect you. If you do not know what resolver is answering your queries, it is worth finding out. During the .de incident, your resolver choice was the difference between a normal evening and a broken internet.

Understand the tradeoff you are making. Strict DNSSEC-validating resolvers (Cloudflare, Google, Quad9) protect you from real DNS hijacking attacks, which happen and which are dangerous. The cost is that when a registry makes a mistake, you feel it immediately and completely. Non-validating resolvers are more forgiving of registry mistakes but leave you exposed to actual attacks. There is no free option here. Pick the tradeoff consciously rather than by accident.

Have a backup resolver in mind. During the incident, technically savvy users who switched from a strict resolver to a non-validating one temporarily restored their own access. This is not something most people will do mid-crisis, but knowing it is possible is useful. Switching your DNS resolver takes about a minute (we have a separate guide on configuring DNS over HTTPS that walks through it for every platform).

Do not over-correct. The lesson of May 5 is not "DNSSEC is bad, turn it off." DNSSEC prevents real attacks. The lesson is that single points of failure at the registry level are a structural risk in how the internet is built, and that the mitigation tooling (serve-stale, Negative Trust Anchors) mostly works. The system absorbed a major shock and recovered in about three hours. That is not nothing.

For a fuller picture of what your current network setup exposes, an ip lookup on your own connection shows your IP, your ISP, and details about how your traffic is routed. Combined with a DNS leak test, it gives you a clear view of the infrastructure you are actually depending on, most of which is invisible until the night it breaks.

The bigger question: is DNSSEC worth it?

The .de outage reignited a long-running debate, and it is worth presenting honestly because reasonable experts disagree.

The case for DNSSEC: DNS hijacking is a real and active threat. Attackers redirect victims to phishing sites by poisoning DNS answers. DNSSEC's cryptographic signatures genuinely prevent this. The protocol works as designed. The May 5 incident was not a failure of DNSSEC's security model, it was a registry operational error that DNSSEC's strictness then amplified.

The case against (or for looser enforcement): DNSSEC has no graceful degradation. There is no warning period, no soft-fail mode in strict configurations. A signature is valid until it is not, and the moment it is not, an entire zone can vanish. A single botched key rollover at one registry, at 10 PM on a Tuesday, stranded millions of people. Critics argue the real-world availability cost of strict enforcement may exceed the security benefit for many users, since registry errors have caused more downtime than DNSSEC has demonstrably prevented attacks for ordinary users.

The middle ground that most large operators have effectively adopted: enforce DNSSEC by default, but build in emergency mechanisms (serve-stale, Negative Trust Anchors) to override it when an operational failure is causing more harm than the security it provides. This is pragmatic but it is also an admission that the protocol as strictly specified is not operationally sustainable on its own.

There is no clean answer. The .de incident is a data point for the "looser enforcement" side, but a DNS hijacking attack that DNSSEC prevents would be a data point for the other. Both risks are real.

Wrap up

On May 5, 2026, Germany's internet largely disappeared for three hours because a piece of undertested code generated bad cryptographic signatures during routine maintenance. No attacker, no malice, just a code path nobody tested and a security protocol that does exactly what it is told even when what it is told is catastrophic.

The internet is full of systems like this. They work invisibly almost all the time, and then occasionally fail in dramatic, total ways that remind you how much you depend on infrastructure you have no control over. The best you can do as a user is understand which pieces you rely on, know what tradeoffs you are making (often without realizing it), and have a rough idea of what to do when something upstream breaks.

Knowing which DNS resolver answers your queries is a good place to start. Most people have no idea, right up until the night it matters.


Sources for this article include DENIC's official post-incident analysis, Cloudflare's incident report on the .de TLD outage, and outage monitoring data from public DNS resolvers. Technical details reflect the information available as of mid-May 2026 and may be updated as DENIC completes its full root cause analysis.

Did you like this?
I
Last updated May 18, 2026 · 13 min read · 2,698 words

Comments 0