App Review Windows Defender Firewall Critique Part 2

[Andy Ful]

Researchers and enthusiasts expose are all the corner cases and things not covered by the dedicated professional test labs.
(...)
Knowing you, your counter-argument is going to immediately go to "Home users are not targeted so that kind of testing is not required and the home users need not worry."

OK, it seems that I misunderstood your meaning of enthusiasts with common YouTube testers.
My argument is slightly different. As you noticed, the tests made by professional labs are usually very different from tests made by researchers and enthusiasts. Home users should not panic after reading/watching tests made by researchers and enthusiasts, because the vulnerabilities exposed, have a very small impact on the infection rate at home. This is possible because Microsoft and most AV vendors are not blind to exposed vulnerabilities. Furthermore, some vulnerabilities are strictly related to Enterprises. The importance of tests made by researchers and enthusiasts is much greater for Microsoft and AV vendors. So, all kinds of tests are welcome.

 

[cruelsister]

who are not professional researchers or security testers that test do not make videos. They make demos at conferences such as Wild West Hackin' Fest or Black Hat

And some actively do both.

They are self taught

God, that would have been soooo much cheaper.

 

[bazang]

Home users should not panic after reading/watching tests made by researchers and enthusiasts, because the vulnerabilities exposed, have a very small impact on the infection rate at home.

They should not, but when it comes to all things related to cybersecurity, the average user is vulnerable to panic due to the Ignorance Tax.

OK, it seems that I misunderstood your meaning of enthusiasts with common YouTube testers.

A lot of those testers are data point. Some are pure entertainment. An exception is Leo. He is not popular here because he does not make videos that a lot of MT members find acceptable. The reason is that they are not at his level nor his approach to Microsoft Defender. Even though he is more than capable of performing methodical, carefully documented and explained testing, that is not his objective. He takes the approach of an average Windows user that does not understand or know Microsoft Defender beyond the basics. Leo is one of those professional testers who is not going to explain in-detail what is being done in the video. He expects the viewer to either figure it out or to educate themselves to figure it out.

And some actively do both.

Yes they do.

Some of the people I know have a their primary relationship with a keyboard and their entire social lives are 100% digital. They're the ones that go to Black Hat and the entire time they are on their device. They attend presentations and never look up. Their fingers constantly doing the klackity-klack. Somehow they absorb stuff. I suppose they have that capacity to multi-focus without looking or tuning other things out.

God, that would have been soooo much cheaper.

You attended Dartmouth undergrad when tuition & housing during the era of what, ~ $20,000 per year? Not sure if you did a grad school program afterwards or not.

Did you not do the Rhodes Scholar thing, or was it Sorbonne?

As undergrad were you partial aid, no aid, or a merit full-ride scholarship recipient?

Self taught is all about paying the Ignorance Tax with a lot of patience, frustration management, dedication, perseverance, and pounding it out on the keyboard. In a few words, sacrifice and hard work. There's a good bit of neuroticism sprinkled on that process (anyone that completes any degree program has the basic elements of neurotic behaviors - it is not anything derogatory; it is the behaviors that matter and not the labels). You of all people know that entire drill very well.

 

[Andy Ful]

An exception is Leo.

Yes, many people like such retro-tests to recall the times from 15 years ago. The malware world was much simpler and much more understandable.

Edit.
Fortunately, he published many good videos unrelated to AV comparison testing.

 

[Andy Ful]

Some readers may think that I am prejudiced against Leo's tests. So let's consider three important facts:

1. The infection rate of tested AVs is incredibly low - typical for several days-old samples. For Kaspersky, the infection rate is close to 0. A similar effect can be seen for AV-Test samples in the non-real-world part of tests (reference set).
2. Only EXE files are tested.
3. Files are executed on the system (non-web threats).

In such tests, the differences between AVs follow mainly from the completeness of the AV signatures (just like 15 years ago).
I can understand that such tests can still be useful for some people, especially when the infections commonly propagate via non-web sources. If I recall correctly, a few years ago MT members from Ukraine reported such a situation, related to sharing pirated software on USB drives. In many countries, the software is hardly available from legal sources and there are no anti-piracy regulations.

 

[bazang]

Some readers may think that I am prejudiced against Leo's tests. So let's consider three important facts:

1. The infection rate of tested AVs is incredibly low - typical for several days-old samples.

Most testers are trying to prove a specific point. Their objective is not to say "It is unlikely you will be targeted or infected, even through your own actions." Why? Because they want to prove their specific point, despite the low probabilities of system infection reality.

2. Only EXE files are tested.

Leo has talked about this before stating that the average user only downloads or launches the .exe or .msi file types. He knows all the various malware types and file types. He is not interested in demonstrating that. He just collects .exe for his demonstrations. He also knows that if a security software publisher's behavioral protections do not properly handle .exe, then those protections are very likely to fail at containing malicious scripts. That is not a given, but I understand Leo's logic. In most cases he is correct but his testing is just many others - it is very specific as opposed to being comprehensive.

 

[Andy Ful]

In most cases he is correct but his testing is just many others - it is very specific as opposed to being comprehensive.

Let's agree on that. to stop irritating @cruelsister (thanks for your patience).

 

[bazang]

Let's agree on that. to stop irritating @cruelsister (thanks for your patience).

I expect that @cruelsister shares Leo's general criticisms of Microsoft Defender. Maybe it's disdain or a combo of the two.

Leo don't use Defender. @cruelsister don't use Defender. Either "Two peas in a pod" or "Like peas & carrots."

 

[Andy Ful]

I expect that @cruelsister shares Leo's general criticisms of Microsoft Defender.

One can share the @cruelsister criticism based on her video:

App Review - Microsoft Defender vs Magniber

Once More unto the Breach...

malwaretips.com

This is a good example, that independent testers can show something interesting.

One can criticize Defender based on professional tests which show that Defender free (on default settings) is not among the top AVs.

One can criticize Defender based on my POC, for example:

Advice Request - Windows Defender disabled by malware

After making several successful POCs and submitting them to Microsoft, I noticed the first signs of Defender's behavior-based detections. One of the executables involved in the UAC bypass is recognized as malicious (I did not submit it to Microsoft). That is good. But, after doing some...

malwaretips.com

Leo's general criticism would be a mistake. Based on his videos, he has the right to criticize only the completeness of the Defender's signatures.

 

[Game Of Thrones]

In India and SE Asia, infections from USB flash drive sharing is rampant.

Not all infections or attacks happen via a file download from a network and execution. That is a real-world fact.

How malware is delivered is a moot point in testing unless the objective is to show how a solution protects at the delivery stage. Default deny solutions predominantly act to block execution or interrupt the malicious run sequence within the post-download or post-exploit environment.

yeah, we have a lot of infected USB devices here too, we tested, and Bitdefender, Eset, and Kaspersky did not care where the file came from and used their cloud and full protection layers on every unknown and new file no matter the entry point. Bitdefender uses its cloud on execution but Eset and Kaspersky use it on access.

Windows Defender is somehow weak in terms of this entry point, in default USB devices can infect a system, iv seen some in people's laptops and PC in shops and copy stores and stuff, and the attack chain gets interrupted when that flash drive arrives in a system with Eset, Kaspersky, BitDefender installed.

the beast is Bitdefender, back in the day (many years ago) there was a virus that kept files hidden in USB drives and in those days Kaspersky, Eset, and Bitdefender detected it. they were the first. all the security solutions just detected the virus and deleted it. but Bitdefender was a surprise, it not only detected the virus but repaired the USB drive and made the files unhidden and like a normal USB drive. if you see in tests Bitdefender and Kaspersky are the two best in repairing an infection.

windows defender? it needs the point of entry to have a better repair or it will just react on simple mechanisms (detect and delete)

 

[Andy Ful]

Windows Defender is somehow weak in terms of this entry point, in default USB devices can infect a system, iv seen some in people's laptops and PC in shops and copy stores and stuff,...

Yes, that issue can be important, when people often share/execute files without an Internet connection. This is still the case in many countries, and Defender + Windows Firewall cannot be a good protection there. It is a kind of paradox that Defender free + Windows Firewall works best in rich countries, where most people use legal software and could easily afford to buy a top AV.

 

[mlnevese]

Yes, that issue can be important, when people often share/execute files without an Internet connection. This is still the case in many countries, and Defender + Windows Firewall cannot be a good protection there. It is a kind of paradox that Defender free + Windows Firewall works best in rich countries, where most people use legal software and could easily afford to buy a top AV.

That's exactly why I say the best antivirus depends on situation. I have seen in my work occasions where a person had to get files from their clients, often pdf files that can be infected, using an offline laptop, going to the client's place and copying from a USB drive for instance. Any security in that machine would often be updated sometime in the morning then only in the next day when he returned with the files.

In this case any solution that strongly depends on internet connection is useless. He'd be much better with ESET, Avast or Bitdefender that have strong local signatures.

Mostly I see people in this and other forums considering Internet connection will be available everywhere, but that's not the case in many places.

Also most seem to dismiss other infection vectors like usb drives only considering infections from the net, which in many places, will not be the main source of infection.

 

[Andy Ful]

In this case any solution that strongly depends on internet connection is useless. He'd be much better with ESET, Avast or Bitdefender that have strong local signatures.

That is true.
Of course, using Defender would still be OK after enabling ASR rules (mainly independent of Internet connections).
However, ASR rules will block pirated software (not a solution for everyone).