App Review Best Antivirus vs Windows Defender: What's the difference? (PC Security Channel)

[Captain Awesome]

I've proven MS Defender's effectiveness many times over

Leo... please watch my videos

He is anti-WD🫢

 

[Harlequin]

AV-Comparatives is not the best indicator. In reality, against new malware, Microsoft Defender performs in the 40% to 60% detection rate. When it comes to any type of advanced tests - such as protections against active banking trojans or info stealers - Microsoft Defender always performs very poorly. One fatal flaw with Defender is that if it does not detect an infection by signature, the malware can do a lot of damage on the system.

"I never had detection in 15 years" is proof that your behaviors are what protect your systems. Microsoft Defender is not even involved. 15 years of no detections means Defender and its forerunner were silent.

I don't know what you are talking about, and my big mistake was to participate in such a discussion, I really don't give a hoot about any third party AVs, although I will enjoy to see their demise in the near future. What 'protects' my system is uniquely Windows Security, and let me reiterate, there is no malware around for average users, if I could disactivate Windows security, I would , but as I have said several times my system does not change in terms of speed. "One fatal flaw with Defender..." Are you really serious? I can recover my system in less than 5 minutes. The only fatal mistake in a system is to install a third party AV and believe it is doing something productive and paying for it...

 

[Andy Ful]

Leo... please watch my videos

Leo's view of Defender evolved recently, so he might watch some of your videos.

 

[mlnevese]

If one has the patience to handpick samples you should be able to do a video where both Microsoft and Kaspersky detect it and Bitdefender misses and another where Bitdefender and Microsoft stop the sample and Kaspersky misses.

Just don't look at me, I'm not patient enough to test thousands of samples to find the ones that fit.

 

[Andy Ful]

In the wild, the sets of missed samples can significantly differ for any two AVs, even if the total number of missed in-the-wild samples would be the same. So, the sample missed by one AV can often be detected by another AV. If one tests two AVs (AV2 and AV3) on the sample already missed by AV1, those AVs (AV2 and AV3) have a fair chance to detect the sample.
From such a test one cannot conclude, that AV1 is inferior to AV2 and AV3. Such (usually unconscious) manipulation can often happen when the tester believes that AV1 is inferior to AV2 and AV3.

 

[Andy Ful]

In reality, against new malware, Microsoft Defender performs in the 40% to 60% detection rate.

Interesting. Could you provide the source of these statistics (if it is not confidential)? What about the other AVs?

 

[bazang]

Interesting. Could you provide the source of these statistics (if it is not confidential)? What about the other AVs?

Internal government statistics. The US NIST and UK NCSC are not going to ever publish the results lest they be sued by the antivirus companies. There are also tests performed by the US DoD and UK MoD which produce essentially the same results.

If no one had ever guessed (probably not, who here actually reads government cybersecurity frameworks and regulations), the reason that you will never see "Active Directory" or "BitLocker" in any government framework is because the agencies and programs do not want to name a specific technology as it will be interpreted as endorsement of that technology.

In the same manner, government agencies that perform tests never publish them as a reported poor result will almost always create outcry and a reported very good result would be interpreted as endorsement.

The range 40% to 60% is widely accepted within the AV industry. Most of that type of data is derived from internal AV industry group studies and is behind paywalls. 10,000 Euro paywalls.

There is not a single AV out there that consistently performs at the 90% level against highly skilled malware campaigns (these are directed at enterprises, governments, and home users; anyone that can be infected). All AV falter to the 40% to 60% range when subjected to highly skilled malware campaigns.

To properly test AV's capabilities, the tester has to have the ability to create all kinds of different types of new malware on-the-fly. Then throw them at AV test boxes in real-time. Harvesting malware from abuse.ch and other sources is not permitted. Youtube testing is not very indicative of AV effectiveness. Malware packs with 1000 samples is not permitted. As you already know, the only organizations that have those kinds of capabilities are nation state, government agencies.

 

[Andy Ful]

@bazang,

We should not rely on data that cannot be verified. I do not say that 40% to 60% is a false range, but we do not know the methodology used to gather the data and how faithful are the results. Furthermore, we do not even know what was researched. You used the term new malware which must be used within a concrete period of time. So we can have new malware samples in the year 2024, or new malware less than 1 day old, etc. Did the data include FUDs (Fully Undetected malware)? FUD is an older well-known malware that is packed/encrypted/protected to avoid detection. Without knowing those (and some other factors) 40% to 60% are only pure numbers.

 

[Andy Ful]

If one has the patience to handpick samples you should be able to do a video where both Microsoft and Kaspersky detect it and Bitdefender misses and another where Bitdefender and Microsoft stop the sample and Kaspersky misses.

Just don't look at me, I'm not patient enough to test thousands of samples to find the ones that fit.

Here is an example for the above post:
https://avlab.pl/en/results-may-2023/
https://avlab.pl/en/wp-content/uploads/2023/06/Report-Summary-May-2023.csv

In this test, Microsoft Defender missed 4 samples and Bitdefender 5 samples. All samples missed by Microsoft Defender were detected by Bitdefender + other tested AVs, and all samples missed by Bitdefender were detected by Microsoft Defender + other tested AVs,.
So, we could use any of 4 samples to show that Microsoft Defender is inferior to Bitdefender, and any of 5 samples to show the opposite. That is how such manipulations work.

Edit.
Leo's test is slightly manipulated, but most presentations include manipulations supported by some comments. For me, Leo's comment makes his (unconscious) manipulation acceptable. He commented that still "Microsoft Defender is very effective at blocking most malware that you'll see...".

 

[Vitali Ortzi]

Leo's view of Defender evolved recently, so he might watch some of your videos.

He definitely watches forums or at least used to
Btw I remember him using a mix of emsisoft and comodo on his system and he worked for emsisoft (they have a good behavior blocker and was especially good for the time back then )

 

[bazang]

@bazang,

We should not rely on data that cannot be verified. I do not say that 40% to 60% is a false range, but we do not know the methodology used to gather the data and how faithful are the results. Furthermore, we do not even know what was researched. You used the term new malware which must be used within a concrete period of time. So we can have new malware samples in the year 2024, or new malware less than 1 day old, etc. Did the data include FUDs (Fully Undetected malware)? FUD is an older well-known malware that is packed/encrypted/protected to avoid detection. Without knowing those (and some other factors) 40% to 60% are only pure numbers.

I can understand your view, but the numbers are real.

New malware means newly created malware. The time of introduction of the malware and the test systems is less than 24 hours. The samples are harvested globally so as not to give the AV geo bias. With the geo bias alone removed, signatures begin to show their weaknesses.

Beyond signature detection, Microsoft Defender provides weak protection.

I've seen the documents. I've seen numbers. I can quote them. But I have no means of obtaining documents and then posting those documents. Even if that were possible I cannot as I have always had to sign non-disclosure agreements (NDA) to gain access from commercial entities that perform such research. For access to government generated data one needs a security clearance. To post that information is a crime.

I'll tell you what, I will help people out here because I know the truth hurts and they get extremely upset emotionally and mentally when the truth is posted. I've been in this row before where the real numbers of 40% to 60% have caused people to smash the report button because they were outraged.

There is all kinds of infos out there that reinforce the notion that all AV is highly effective. People can believe this guy and his team of researchers. LOL.

https://scholarworks.gsu.edu/cgi/viewcontent.cgi?article=1000&context=ebcs_tools

David Maimon

David Maimon is an Associate Professor in the department of Criminal Justice and Criminology at Georgia State University.

aysps.gsu.edu

His researchers collect samples from abuse.ch and similar sources. The AV industry loves these people because they say the things that the AV wants to hear. They also satisfy people who want to believe that their AV of choice is a 98% effective security solution.

 

[Vitali Ortzi]

Internal government statistics. The US NIST and UK NCSC are not going to ever publish the results lest they be sued by the antivirus companies. There are also tests performed by the US DoD and UK MoD which produce essentially the same results.

If no one had ever guessed (probably not, who here actually reads government cybersecurity frameworks and regulations), the reason that you will never see "Active Directory" or "BitLocker" in any government framework is because the agencies and programs do not want to name a specific technology as it will be interpreted as endorsement of that technology.

In the same manner, government agencies that perform tests never publish them as a reported poor result will almost always create outcry and a reported very good result would be interpreted as endorsement.

The range 40% to 60% is widely accepted within the AV industry. Most of that type of data is derived from internal AV industry group studies and is behind paywalls. 10,000 Euro paywalls.

There is not a single AV out there that consistently performs at the 90% level against highly skilled malware campaigns (these are directed at enterprises, governments, and home users; anyone that can be infected). All AV falter to the 40% to 60% range when subjected to highly skilled malware campaigns.

To properly test AV's capabilities, the tester has to have the ability to create all kinds of different types of new malware on-the-fly. Then throw them at AV test boxes in real-time. Harvesting malware from abuse.ch and other sources is not permitted. Youtube testing is not very indicative of AV effectiveness. Malware packs with 1000 samples is not permitted. As you already know, the only organizations that have those kinds of capabilities are nation state, government agencies.

Leo Mal X is much better then just dropping malware when maybe some will win based on intelligence of the common malware
So I hope he continues to develop it anyway you're totally correct and it's annoying that labs don't use enough apt tests and even when they do they are mediocre considering it's a research "lab" at least from what's available to the public


Anyway I really hope project zero will go back to doing research into sandboxes (emulation) ,the great research they did showing how the mini filters of avs can be easily waeponized by a talented researcher (worse of all they all run at trusted installer , system and they can be trigger to scan , emulate even by code inside html or scripts aka zero click )

Anyway really interesting what you personally use in the "civilian" life (not an endorsement or anything related to any agency , business rather personal preference)

 

[Vitali Ortzi]

I can understand your view, but the numbers are real.

New malware means newly created malware. The time of introduction of the malware and the test systems is less than 24 hours. The samples are harvested globally so as not to give the AV geo bias. With the geo bias alone removed, signatures begin to show their weaknesses.

Beyond signature detection, Microsoft Defender provides weak protection.

I've seen the documents. I've seen numbers. I can quote them. But I have no means of obtaining documents and then posting those documents. Even if that were possible I cannot as I have always had to sign non-disclosure agreements (NDA) to gain access from commercial entities that perform such research. For access to government generated data one needs a security clearance. To post that information is a crime.

I'll tell you what, I will help people out here because I know the truth hurts and they get extremely upset emotionally and mentally when the truth is posted. I've been in this row before where the real numbers of 40% to 60% have caused people to smash the report button because they were outraged.

There is all kinds of infos out there that reinforce the notion that all AV is highly effective. People can believe this guy and his team of researchers. LOL.

https://scholarworks.gsu.edu/cgi/viewcontent.cgi?article=1000&context=ebcs_tools

David Maimon

David Maimon is an Associate Professor in the department of Criminal Justice and Criminology at Georgia State University.

aysps.gsu.edu

His researchers collect samples from abuse.ch and similar sources. The AV industry loves these people because they say the things that the AV wants to hear. They also satisfy people who want to believe that their AV of choice is a 98% effective security solution.

You gotta mix everything from packing to non packing , obfuscation and no obfuscation

Sometimes being packed , obfuscated can actually cause some very aggressive av software to mark anything malicious
Meanwhile it can help bypass some modules of less aggressive avs


So even the most basic stuff needs variety

Oh and sometimes making simple scripts especially using non common languages ,less commonly used sub types of mitre tactics can be very effective

So you don't need to do anything fancy to bypass majority of av software and ai can help with coding in uncommon languages the threat actor may not have experience required in an obscure language ,less common

So an attacker who doesn't know how to code can bypass an av with simple ai assistance

Oh and definitely you don't need any overflow , rop , or any memory corruption, cve to bypass majority of products by the way they are usually configured , stay at default

 

[Andy Ful]

@bazang,

The article mentioned by you cannot bring anything useful to the discussion.

I am not sure if you understood my post. The detection of 40% to 60% of new malware used in the targeted attack on Enterprises is nothing new. It is consistent with the statistics already posted by me:
https://malwaretips.com/threads/mic...s-malware-gets-around-it.133857/#post-1109465

But, the same source shows that in the real world, other popular AVs also have similar detection. Of course, there can be some advanced EDR solutions that can detect/block 90% of new malware. However, most of the increased detection does not follow from better behavior blocking, but mainly from system/network restrictions and allowlisting.
According to available sources, Microsoft Defender has good behavior-based detection for new previously unseen malware.
https://learn.microsoft.com/en-us/d...k-at-first-sight-microsoft-defender-antivirus

I could see it working many times when my freshly compiled tools were detected and blocked. I must always submit my tools to Microsoft before publishing them.

 

[Vitali Ortzi]

@bazang,

The article mentioned by you cannot bring anything useful to the discussion.

I am not sure if you understood my post. The detection of 40% to 60% of new malware used in the targeted attack on Enterprises is nothing new. It is consistent with the statistics already posted by me:

Security News - Microsoft Defender Is Not Enough Anymore—This Malware Gets Around It

Malware hidden inside a fake NFT game ignores two-factor authentication and gains access to a victim's Google account via a malicious Chrome Extension to steal your money. Microsoft Defender, the antivirus app that comes enabled in Windows Vista and later, reportedly failed to catch a type of...

malwaretips.com

But, the same source shows that in the real world, other popular AVs also have similar detection. Of course, there can be some advanced EDR solutions that can detect/block 90% of new malware. However, the increased detection does not follow from behavior blocking, but mainly from system/network restrictions and allowlisting.
According to available sources, Microsoft Defender has good behavior-based detection for new previously unseen malware.

Enable block at first sight to detect malware in seconds - Microsoft Defender for Endpoint

Turn on the block at first sight feature to detect and block malware within seconds.

learn.microsoft.com

I have seen it many times working when my freshly compiled tools were detected and blocked. I must always submit my tools to Microsoft before publishing them.

Microsoft behavior monitoring has improved a lot since the days of windows 10 but still not enough on default settings (no asr rules etc ) yet it can block stuff even some of the top av software wouldn't on default

 

[oldschool]

I'll tell you what, I will help people out here because I know the truth hurts and they get extremely upset emotionally and mentally when the truth is posted. I've been in this row before where the real numbers of 40% to 60% have caused people to smash the report button because they were outraged.

I'm reporting your right now, because you've made me emotionally and mentally upset. I don't know if I can handle the truth. Time for jisatsu.

 

[Andy Ful]

Microsoft behavior monitoring has improved a lot since the days of windows 10 but still not enough on default settings (no asr rules etc ) yet it can block stuff even some of the top av software wouldn't on default settings (no asr rules etc )

Yes, the ASR rules can make the attacks harder to perform and can also discourage some attackers. However, I do not know any tests that could measure properly the impact of ASR rules on security. Furthermore, the Enterprises can use different ASR rules. It is a complex problem.

Edit.
Please note that I do not think that Microsoft Defender (MD) on default settings is a top AV. There are several AV protection tests. In some tests MD scores just like top AVs. In others, the results are not so good. If you include the tests from two or three years, the difference between MD and top AVs is not big but statistically significant. I agree with Leo, that decision to use 3rd party AV over MD should depend on other factors like using additional features, possible performance problems, or control over advanced settings. Many people will skip MD for that. If those factors are not important, then MD + SmartScreen is OK. Advanced users can also apply tools to control the MD advanced features and Windows built-in security.

 

[simmerskool]

Internal government statistics. The US NIST and UK NCSC are not going to ever publish the results lest they be sued by the antivirus companies.

Not disputing you, but IIRC companies like AppGuard would advertise the gov't uses our software. Their ads sorta read like a gov't endorsement. My wife was director of local federal gov't office about 15 years ago, but she (now retired) has no recollection of security software they used, although I do recall gov't flying in techs 2 techs to work on their computers, they seemed like serious people.

 

[monkeylove]

It’s always been the same main issue, what’s the point in detecting something that behaves like malware but it is not? In AV tests this called false positive. There is no doubt that Kaspersky and Bitdefender are good AVs but so is Windows Security. I’ve had MS Defender (Windows Security) for 2 years on Win11 and 7 years on Win 10. No problems whatsoever. I also believe there is no malware around for average users, if I wanted to infect my computer purposely, I wouldn’t know what to do.

That's part of hasty generalization, where one assumes that one's experiences represent that of the majority. Add to that lack of verification.

There's also the possibility that more malware don't show themselves, remain hidden just to steal data, can run without user interaction, can target embedded software directly, and can be found in "legitimate" software and even websites, including in parts of web pages or from data fed by various servers to websites.

Add to this related points for which users can do little, such as their data stolen from various businesses and government agencies targeted by malware.

Finally, I do remember reading one thread in this forum about studies across the years, and I think the infection rate is around 5 percent, or something like that.

 

[jamey910111]

Talking with a deeply devout person is mainly useless.

it’s based on my experience using it for many years in the past.

But i understand ur point, you are all about facts and numbers. Me on the other hand i make useless points cause I use human devout passion. microsoft always sticks to facts and numbers too, that’s why it’s such a successfully flawed product.