Gandalf_The_Grey

Level 55
Verified
Helper
Top poster
Content Creator
Well-known
Apr 24, 2016
4,442
A detailed explanation of what happened to #AdGuardDNS and why by company's CTO
We just had a serious outage of @AdGuard DNS, but it was actually caused by @Facebook. What hapned and how on earth @AdGuard may depend on FB? Let me try to explain.

Everything started with Facebook name servers going down today. AdGuard DNS connects to them in order to find out the addresses of Facebook domains. So, they went down and now AdGuard DNS was responding with error to every request for FB domains’ addresses.

This caused a considerable spike in the overall number of requests. What happened? Every app, every device was now repeatedly requesting FB domains as if they can’t live without it.

The high number of requests is not much of a problem for us, we’re ready for higher load so this went almost unnoticed. So everything was working well until one crucial moment when Facebook engineers decided to null-route their nameservers.

What does this mean? From now on requests to FB name servers not just fail, they TIME OUT. Now we could not respond quickly with an error and have to wait for a few seconds until we’re sure there will be no response.

The worst part is that we weren’t doing any negative caching. It means that if we cannot resolve a domain, we were trying to do that again and again until it finally succeeds (it never did) instead of caching the negative result at least for a few seconds.

So we had an overwhelming number of incoming queries that time out and simply exhaust the servers resources. This all lead to one of the worst outages we ever had with AG DNS.

At some point we almost hit 1M queries per second (our normal load is about 250-300k). The most of the queries are encrypted (DoT/DoH/DoQ) so this is like 10x regular DNS load.

It took us about an hour to figure all that out, implement a fix (negative caching it is) and deploy it to every AG DNS server. Everything works well now, but we learned a couple of very useful lessons. Thanks, FB!
 

Freud2004

Level 10
Verified
Well-known
Jun 26, 2020
466
He really doesn't sound like a lunatic, wise words.

John David McAfee (/ˈmækəfiː/ MAK-ə-fee;[2][3] 18 September 1945 – 23 June 2021)[4][5] was a British-American computer programmer, businessman, and two-time presidential candidate who unsuccessfully sought the Libertarian Party nomination for president of the United States in 2016 and 2020. In 1987, he wrote the first commercial anti-virus software, founding McAfee Associates to sell his creation. He resigned in 1994 and sold his remaining stake in the company.[6] McAfee became the company's most vocal critic in later years, urging consumers to uninstall the company's anti-virus software, which he characterized as bloatware. He disavowed the company's continued use of his name in branding, a practice that has persisted in spite of a short-lived corporate rebrand attempt under Intel ownership.

McAfee's fortunes plummeted in the financial crisis of 2007–2008. After leaving McAfee Associates, he founded the companies Tribal Voice (makers of the PowWow chat program), QuorumEx, and Future Tense Central, among others, and was involved in leadership positions in the companies Everykey, MGT Capital Investments, and Luxcore, among others. His personal and business interests included smartphone apps, cryptocurrency, yoga, light-sport aircraft[7] and recreational drug use. He resided for a number of years in Belize, but returned to the United States in 2013 while wanted in Belize for questioning on suspicion of murder.[8]

 

plat1098

Level 26
Verified
Top poster
Well-known
Sep 13, 2018
1,519
OK, so I did some real-life browsing and went back to two places in my immediate neighborhood that made the news earlier this year. One has a hopeful ending and one is waiting for the judge's gavel, I think.

The first one, which I link here is now a burned out shell. home to pigeons and one burned-out vehicle.

IMG_0081.jpg
IMG_0083.jpg

The second one involved an city bus slamming into a 100 yr. old townhouse, injuring 16 people, including the driver who prob. saw the Gates of Heaven opening as he lost control. I really hope he's OK now and back on the job. But looks like the building is saved and is currently undergoing repairs. They don't make 'em like they used to, it seems.

IMG_0080.jpg

 

brambedkar59

Level 23
Verified
Top poster
Well-known
Apr 16, 2017
1,223
Tech-illiterates aka majority doesn't even care about it anymore, they forgot the headline they saw on a news site about something called Pegasus. The amount of outrage this should have caused is just underwhelming.


Now something to watch with my morning cup of tea. Glorious IS-2 heavy tank equipped with 122mm cannon, developed as a successor to the KV-series by the Soviet Union during World War II. That orgasmic sound when 122mm fires 🙀.

 

SpiderWeb

Level 7
Verified
Well-known
Aug 21, 2020
326

Storage bandwidths are catching up to RAM bandwidth very fast. I can imagine a future where storage and RAM will be the same thing and people won't even remember when they were separate.
 

Moonhorse

Level 31
Verified
Top poster
Content Creator
Well-known
May 29, 2018
2,064
Tech-illiterates aka majority doesn't even care about it anymore, they forgot the headline they saw on a news site about something called Pegasus. The amount of outrage this should have caused is just underwhelming.


Now something to watch with my morning cup of tea. Glorious IS-2 heavy tank equipped with 122mm cannon, developed as a successor to the KV-series by the Soviet Union during World War II. That orgasmic sound when 122mm fires 🙀.

Tiger h1 vs is 2 was probably even, who shot first won


I like the desing of this tank
 
Top