AV-TEST Advanced Threat Protection (ATP) AV.TEST test January - June 2025

[ForgottenSeer 114717]

13 security products in an endurance test: here are the best packages for Windows

Many consumer users don't see themselves as being in the crosshairs of cybergangsters. A major fallacy: because attackers are constantly foraging for interesting data, passwords or e-mail log-in details. This also includes entire Windows systems, as attacking botnets are always on the lookout...

www.av-test.org

View attachment 290779

AV-Test, like all AV labs, focuses upon certain things and it is very rare for any participating AV to perform poorly. I cannot recall an instance of one scoring 4 and lower for a very long time.

If a user does not regularly use a Standard User Account (SUA), is in a high risk environment, a prolific downloader, does significant online transactions, has a need to protect sensitive data, then Microsoft Defender at its default settings is not the best option. Against the newest and certain classes of malware and attacks, it is difficult to recommend Microsoft Defender confidently.

Nobody tests and reports Malware Defender at default configurations, at maximum configurations, and then in combination with other Microsoft security features. It is considered "permutation testing" and Microsoft is not going to pay for it.

Microsoft Security - which includes many components of which Malware Defender is only a single part - properly configured (for maximum security; security is prioritized over convenience/productivity) and managed is robust, but those features are not available to consumers.

Malware Defender at default configuration provides "adequate" protection for low-risk, everyday, typical consumer use.

Real-world, practical results - which means reports from the field, as opposed to lab test reports - indicate that all AV fail at a rate that is significant at the moment of truth. The % of active infections (even if it is only PUA/PUP) on systems with "top performing" AV is astounding.

All that said, my observations of consumer infected systems are mostly those under the control of 1) people ignorant/with a basic understanding of security, 2) have no inclination to be secure or make very little effort to be secure, and 3) even when instructed on security, they don't change their behaviors.

If I was given 1,000,000 Euros and instructed use it for betting on security solution outcomes, then I would bet only 1 Euro on any of them. Because all the meaningful and secure outcomes are largely dependent upon the people, their decisions, and their behaviors. Plus things out of their control which degrade the AV performance, such as internet outages, problems with their devices or applications or OS, etc.

One should not extrapolate or project security performance or expectations of security performance based upon test lab results. And since no vendor - not even Comodo - will provide the detailed field reports to the world, nobody except for a very few people know the truth about AV. Get 1,000 top pen testers and malcoders, throw them against every single security solution out there, even with different configurations, and all marketing claims and claims of "You are protected" would be unraveled. Some AV will do worse than others overall. Some AV will be weaker and stronger in different protection areas. But overall, the results would be dismal with protection rates in the 40% or lower range.

Security solution testing is about the same as testing vehicle engine oil to ensure that it meets a minimum standard of both specification and quality. However, when testing oil in real world conditions it becomes clear that negative outcomes are significant after reviewing the full available data. Many times the oil itself is not the variable that led to the failure.

The AV industry has capitalized on peoples' propensity to fully trust based upon "Five Stars and All Green Bars" lab test results. Those tests just establish a minimum baseline of performance using concocted scenarios that do not account for all the stuff that happens in the real world. So, in short, the results are "synthetic" and it takes a lot of knowledge and experience to understand what they really say, and most importantly, what they do not say.

 

[ForgottenSeer 123960]

AV-Test, like all AV labs, focuses upon certain things and it is very rare for any participating AV to perform poorly. I cannot recall an instance of one scoring 4 and lower for a very long time.

If a user does not regularly use a Standard User Account (SUA), is in a high risk environment, a prolific downloader, does significant online transactions, has a need to protect sensitive data, then Microsoft Defender at its default settings is not the best option. Against the newest and certain classes of malware and attacks, it is difficult to recommend Microsoft Defender confidently.

Nobody tests and reports Malware Defender at default configurations, at maximum configurations, and then in combination with other Microsoft security features. It is considered "permutation testing" and Microsoft is not going to pay for it.

Microsoft Security - which includes many components of which Malware Defender is only a single part - properly configured (for maximum security; security is prioritized over convenience/productivity) and managed is robust, but those features are not available to consumers.

Malware Defender at default configuration provides "adequate" protection for low-risk, everyday, typical consumer use.

Real-world, practical results - which means reports from the field, as opposed to lab test reports - indicate that all AV fail at a rate that is significant at the moment of truth. The % of active infections (even if it is only PUA/PUP) on systems with "top performing" AV is astounding.

All that said, my observations of consumer infected systems are mostly those under the control of 1) people ignorant/with a basic understanding of security, 2) have no inclination to be secure or make very little effort to be secure, and 3) even when instructed on security, they don't change their behaviors.

If I was given 1,000,000 Euros and instructed use it for betting on security solution outcomes, then I would bet only 1 Euro on any of them. Because all the meaningful and secure outcomes are largely dependent upon the people, their decisions, and their behaviors. Plus things out of their control which degrade the AV performance, such as internet outages, problems with their devices, etc.

One should not extrapolate or project security performance or expectations of security performance based upon test lab results. And since no vendor - not even Comodo - will provide the detailed field reports to the world, nobody except for a very few people know the truth about AV. Get 1,000 top pen testers and malcoders, throw them against every single security solution out there, even with different configurations, and all marketing claims and claims of "You are protected" would be unraveled. Some AV will do worse than others overall. Some AV will be weaker and stronger in different protection areas. But overall, the results would be dismal with protection rates in the 40% or lower range.

Security solution testing is about the same as testing vehicle engine oil to ensure that it meets a minimum standard of both specification and quality. However, when testing oil in real world conditions it becomes clear that negative outcomes are significant after reviewing the full available data. Many times it is not the oil that is not the variable that led to the failure.

While it's true that the user is the most critical element in any security strategy, dismissing the value of independent lab tests is a mistake.

Reputable labs like AV-TEST and AV-Comparatives provide the only objective, data-driven way to evaluate security software, and their methodologies are designed to mimic real-world threats.

This data consistently shows that Microsoft Defender, even in its default state, offers robust, top-tier protection against zero-day and widespread malware, forming an excellent baseline for consumer security.

The most effective approach is a layered "defense-in-depth" strategy that does not rely on a single tool. This involves using a proven endpoint protection solution like Defender, hardening its settings based on your risk profile, and, most importantly, practicing vigilant security hygiene.

Ultimately, strong password management, the universal use of Multi-Factor Authentication (MFA), and a keen awareness of phishing and social engineering are what truly fortify your defenses against the vast majority of modern threats.

 

[ForgottenSeer 114717]

While it's true that the user is the most critical element in any security strategy, dismissing the value of independent lab tests is a mistake.

I did not say that the tests should be dismissed.

The test results are applicable only to the test methodology and the samples used, and should not be used to extrapolate or infer protection performance beyond those variables.

Reputable labs like AV-TEST and AV-Comparatives provide the only objective, data-driven way to evaluate security software, and their methodologies are designed to mimic real-world threats.

They do their best, but the mimic is conducted under idealized, controlled test lab conditions.

This data consistently shows that Microsoft Defender, even in its default state, offers robust, top-tier protection against zero-day and widespread malware, forming an excellent baseline for consumer security.

That is where the problem lies. The "test data" does not reflect real world performance across the entire spectrum of device types, user types (e.g. very young children), conditions, etc. I have access to non-publicly available non-consumer data aggregated and filtered from real-world field reports and over the years it has shown with remarkable consistency that security solutions - in real world deployments - perform in the 40% protection range. That data considers all possible vectors, and with that, a lot of the solutions do not have all of the requisite capabilities to protect against all vectors.

There is a lot of "territory" or "conditions" not covered by synthetic lab tests. The labs themselves have openly stated this fact. Even the same AV deployed in different regions of the world can have a significant variance in protection performance over time.

"Top-tier" versus "Second-tier" and so on is not very meaningful as the comparisons are relative and arbitrary. It's the best that test labs can do, and to this day vendor participants often dispute the accuracy, validity, and "fairness" of the tests. AMSTO has tried over the decades, but there is still heated debates because not everyone agrees on the AMSTO guidelines.

I would agree that the security baseline of most AV have improved greatly over the decades, but still that baseline performance is not truly indicative of overall real-world protection. The differences are only revealed under the heaviest use, higher risk conditions.

These are things that almost never discussed. I will leave it that.

 

[annaegorov]

I use nothing but Kaspersky on the desktop. In terms of protection features, it is the only program that has not left the top for years in my opinion.

AND THIS IS THE RUB........

Kaspersky being banned in the USA is just horse, plus chicken sh-t...

How do we know, that is was a petty, fake news, Russia, Russia hoax?

Easy, no other country in the world has abandoned Kaspersky, because it is run by the KGB....Except maybe Canada, which is USA 1.0 !

 

[ForgottenSeer 123960]

I did not say that the tests should be dismissed.

The test results are applicable only to the test methodology and the samples used, and should not be used to extrapolate or infer protection performance beyond those variables.

They do their best, but the mimic is conducted under idealized, controlled test lab conditions.


That is where the problem lies. The "test data" does not reflect real world performance across the entire spectrum of device types, user types (e.g. very young children), conditions, etc. I have access to non-publicly available non-consumer data aggregated and filtered from real-world field reports and over the years it has shown with remarkable consistency that security solutions - in real world deployments - perform in the 40% protection range. That data considers all possible vectors, and with that, a lot of the solutions do not have all of the requisite capabilities to protect against all vectors.

There is a lot of "territory" or "conditions" not covered by synthetic lab tests. The labs themselves have openly stated this fact. Even the same AV deployed in different regions of the world can have a significant variance in protection performance over time.

"Top-tier" versus "Second-tier" and so on is not very meaningful as the comparisons are relative and arbitrary. It's the best that test labs can do, and to this day vendor participants often dispute the accuracy, validity, and "fairness" of the tests. AMSTO has tried over the decades, but there is still heated debates because not everyone agrees on the AMSTO guidelines.

I would agree that the security baseline of most AV have improved greatly over the decades, but still that baseline performance is not truly indicative of overall real-world protection. The differences are only revealed under the heaviest use, higher risk conditions.

These are things that almost never discussed. I will leave it that.

That's a significant claim that runs counter to all the available public data. Could you help me understand the methodology behind those field reports?

How was the data collected and aggregated?

What were the specific criteria used to define a security 'failure'?

How was bias (like reporting bias) accounted for in the collection process?

Without that information, it's impossible to treat it as more than an interesting but unverified assertion. The reason I rely on established test labs is because they publish their methodologies, allowing for exactly this kind of scrutiny.

 

[ForgottenSeer 114717]

Easy, no other country in the world has abandoned Kaspersky, because it is run by the KGB....Except maybe Canada, which is USA 1.0 !

There's a whole bunch of countries that have banned or that have proposed larger Kaspersky bans that are pending.

The decisions and actions taken against Kaspersky were carefully considered based upon credible intelligence. They weren't hoax-based or hysteria.

What is true of Kaspersky is true of virtually every digital product or service.

If you trust Microsoft not to use Windows against you at the direction of the U.S. Government if that government sees fit to do so, then you're living in denial.

 

[ForgottenSeer 114717]

How was the data collected and aggregated?

Forensic analysis.

What were the specific criteria used to define a security 'failure'?

A security 'failure' is defined as any threat that makes it to the user or a process working on behalf of a user/users.

For example, a single phishing email that makes it to a system's email inbox is considered a failure, because once the email is capable of being accessed by an end user the system and network must be considered compromised. All bets are off.

The criteria can best be summed up as no compromise whatsoever, not even preliminary stage. Because once that happens, it is inevitable that eventually there will be attack success. So proper security criteria considers any form of breach as "fully compromised." That is the basis of enterprises and governments assuming breach 100% of the time - of which very, very few actually do successfully. And most of the success it at a limited, small scale.

How was bias (like reporting bias) accounted for in the collection process?

If there is, then I don't have those details.

Without that information, it's impossible to treat it as more than an interesting but unverified assertion.

I never claimed that what I stated is verifiable. My intent is not to provide verification of anything. Based upon decades of experience and observation though, the model used and the data collected has proven reliable and reflects the protection results of real world deployments.

I think AV lab test results are accurate for a specific set of conditions. No more. No less. I do not think they are highly predictive of actual, real world results as the real world is too highly variable.

They are however, a data point that can be used as part of a larger overall analysis.

In every single test that a security solution posts poor results, the vendor invariably argues that the test methodology or criteria were not valid in one way or another to discredit the test results. So the labs adopted methodology that most product vendors agree upon, but in specific tests each vendor often files complaints or contests results that they don't like. Plus undetected "gaming of the tests" by publishers happens.

I am not dismissing AV test lab testing. I am merely stating that there are considerations that are legitimate. If the data that one has access to is only what the AV test labs publish, then that's what one base their decisions upon.

 

[stonjean633]

AND THIS IS THE RUB........

Kaspersky being banned in the USA is just horse, plus chicken sh-t...

How do we know, that is was a petty, fake news, Russia, Russia hoax?

Easy, no other country in the world has abandoned Kaspersky, because it is run by the KGB....Except maybe Canada, which is USA 1.0 !

Kasperksy's flagship has always been the Very good Detection rate. As long as it stays that way, they will be battling for the top.

 

[Andy Ful]

Most of us will agree that AV tests cannot be ignored.
However, I analyzed many tests and concluded that the results can be significantly affected by short-lived malware and short-lived malicious URLs. The current testing methodology is pretty much blind for both. So, the more the short-lived stuff in the wild, the less correct the test results.

Even a few years ago, most malware samples in the wild affected only one machine protected by a particular AV. This was an effect of the high rate of malware polymorphism. If this fact is combined with a short time of activity, then many AV signatures did not protect users, even if those signatures still can be an important factor for winning in the AV tests.

What we are doing now is testing (and comparing) how good bulletproof jackets are against arrows.
Next, we assume that the best jacked against arrows must also be the best against bullets. But there is no evidence that such an assumption is true.
Of course, it would be much better to test jackets against bullets and assume that they are also good against arrows.

 

[Parkinsond]

Most of us will agree that AV tests cannot be ignored.
However, I analyzed many tests and concluded that the results can be significantly affected by short-lived malware and short-lived malicious URLs. The current testing methodology is pretty much blind for both. So, the more the short-lived stuff in the wild, the less correct the test results.

Even a few years ago, most malware samples in the wild affected only one machine protected by a particular AV. This was an effect of the high rate of malware polymorphism. If this fact is combined with a short time of activity, then many AV signatures did not protect users, even if those signatures still can be an important factor for winning in the AV tests.

What we are doing now is testing (and comparing) how good bulletproof jackets are against arrows.
Next, we assume that the best jacked against arrows must also be the best against bullets. But there is no evidence that such an assumption is true.
Of course, it would be much better to test jackets against bullets and assume that they are also good against arrows.

To recapitulate and stay on-topic, is the specific test of this thread considerable and based on, McAfee and K are top products, of should be ignored and we have no proof regarding McAfee and K? Y/N

 

[ForgottenSeer 114717]

However, I analyzed many tests and concluded that the results can be significantly affected by short-lived malware and short-lived malicious URLs. The current testing methodology is pretty much blind for both. So, the more the short-lived stuff in the wild, the less correct the test results.

This is the core issue. Combine that with the number of malicious emails that do not get detected and blocked, and it means that AV test lab methods do not "measure" the effectiveness of a significant aspect of the real world threat landscape. It is an inherent limitation of AV testing, but the best that can be accomplished currently.

Create 100 true zero-hour malware with combined true zero-day exploits using a range of infection vectors, throw them all at AVs, and test results would be much worse than the typical AV test lab results. An unprovable assertion? Yes. Probability of being accurate? Very high. Counterargument? "Users do not experience such conditions. Plus the conditions are synthetic and not real world in-the-wild."

It largely depends upon one's perspective.

I tend to view things as pass/fail. Either a product protects or it does not, regardless of the specific conditions required because software publishers obviously are limited in their ability to cover every possible potential set of conditions. Many will argue that my perspective is harsh, unforgiving, and unrealistic - but I don't subscribe to "The probability is low that a user will encounter such circumstances." That's like saying that for a user out there who is never targeted and never does anything to infect their systems, that "They are protected." Such a statement is highly misleading.

 

[Andy Ful]

To recapitulate and stay on-topic, is the specific test of this thread considerable and based on, McAfee and K are top products, of should be ignored and we have no proof regarding McAfee and K? Y/N

1. This test, combined with other tests, shows that McAfee and Kaspersky are top against arrows.
2. We can make some additional (imperfect) conclusions after analyzing the details of the protection layers used to fight bullets.
3. The reputation of products should also be included.

The information from points 1-3 strongly suggests that Kaspersky can be among the top products.
I do not know the details of McAfee protection.

Edit.
I agree with some MT members that currently (war time) it would be risky to use Kaspersky in critical or government institutions (in Western countries).
However, this does not follow from the past and present reputation of Kaspersky.
The same would be objectively true for using Microsoft Defender in Russia.

 

[Trident]

I do not know the details of McAfee protection.

Always happy to share details.

Highly centred around pre-execution blocks, McAfee cares about pre-execution a lot.

The following is used:

Yara Rules (updated daily)

Very generic detections/code level heuristics (updated daily).

Pre-execution emulation with its own set of rules (heuristics). Updated daily.

AI pre-execution analysis on scripts, office files, executables and non-pe (modules). There are local and online models. The models were recently retrained, not sure how often it happens.

Offline Trust Detection repository, which is used for whitelisting and is inverted as well, when something is not whitelisted it becomes suspicious

Online reputation (unknown, safe, malicious, suspicious), based on TLSH

Origin Heuristics (where the file comes from, how it came)

Post-execution analysis through Real Protect based on local and cloud machine learning models. These as well were recently retrained on the client side.

Every component is called on every file in a jury sort of way to provide verdict expressed as a whole number. The higher the number, the more confident is the engine that this is malware. It is all linked via the Fusion framework.

In addition, McAfee blocks dodgy websites system-wide, and removes dodgy emails from inboxes.
On the cloud side they use many forms of AI/ML including on raw bytes, eXplainable AI, malware as image representation, TLSH machine learning and so on.

 

[Andy Ful]

Always happy to share details.

Highly centred around pre-execution blocks, McAfee cares about pre-execution a lot.
(..)
In addition, McAfee blocks dodgy websites system-wide, and removes dodgy emails from inboxes.

Looks good.

 

[ForgottenSeer 123960]

Thank you both for a truly insightful and high-level discussion. This is one of the most important conversations in security, bridging the gap between theoretical testing and the operational reality of the threat landscape. I'd like to offer my perspective.

@Andy Ful

"Arrows vs. Bullets"

Andy, your "arrows vs. bullets" analogy is perhaps the best I've seen for explaining the limitations of standardized testing. You are absolutely correct. The primary value of modern endpoint protection is not its ability to react to known threats ("arrows") but its capacity to proactively stop unknown, novel attacks ("bullets") through advanced, non-signature-based methods.
Your focus on analyzing a product's specific protection layers, its behavioral blockers, exploit mitigation, and cloud heuristics, is precisely the right approach for a technical evaluation. Lab tests establish a baseline of competence, but a deep dive into the architecture is what helps us estimate its effectiveness against the zero-day threats that define the modern landscape.

@bazang

The Primacy of the Human and the Limits of Testing

Your skepticism of relying solely on "five stars and all green bars" is not only valid but essential. You are right to champion the reality that the human element is a critical, and often decisive, factor in any security incident. Your assertion that no tool is a silver bullet and that real-world conditions are far messier than a lab environment is an undeniable truth. The ultimate goal of any security system is to function amidst that chaos, not in a sterile testbed.

Synthesizing These Views into a Practical Framework

Where I believe we can build a bridge is by moving away from a pass/fail perspective and toward a model of layered risk reduction.

In this model, the points you both raise are not contradictory, they are simply different, essential layers of a single, cohesive strategy.

Lab Tests as the First Filter (The "Arrow" Test)

The tests from AV-TEST and others serve as a baseline competency check. As Andy noted, they show who is effective against "arrows." A product that cannot consistently stop thousands of widespread, known threats has no chance of stopping a targeted, novel attack. This isn't the end of the analysis, but it's a non-negotiable starting point. It helps us Identify and Protect.
Modern Architecture as the Proactive Defense (The "Bullet" Shield), This is Andy's core point. Once a product passes the baseline, we must evaluate its ability to Detect and Protect against "bullets." This means prioritizing solutions with robust behavioral analysis, anti-exploit technology, and real-time threat intelligence. This is the layer that functions when a brand-new threat bypasses signatures.

The Human Layer as the Gatekeeper.

This addresses bazang's argument. The most effective way to stop a "bullet" is to prevent it from ever being fired. Strong user hygiene, vigilant phishing awareness, universal use of MFA, consistent patching, and strong password policies—is the single most effective way to reduce the attack surface. This layer strengthens the Protect function more than any other.

Regarding the claim of ~40% real-world effectiveness, I suspect that figure is born from a definition of "failure" that includes any intrusion, such as a phishing email successfully reaching an inbox.

From a threat management perspective, this is a valid data point. From an endpoint security perspective, however, the more common definition of failure is a malicious payload successfully executing and achieving its objective. Both are valid lenses, but they measure different things.

Ultimately, I believe we are all advocating for the same thing, for users to move beyond a simplistic "install and forget" mindset and adopt a mature security posture. This posture relies on using tools that are verified by baseline tests, architected for modern threats, and operated by an informed, vigilant user.

Thank you both for elevating the conversation.

 

[Andy Ful]

Thank you both for elevating the conversation.

You're welcome.

 

[rashmi]

Thank goodness I'm not as wise as my friends here; Comodo and Kaspersky are our superheroes, offering 101% and 100% protection—the joy of simplicity!

 

[Trident]

Thank goodness I'm not as wise as my friends here; Comodo and Kaspersky are our superheroes, offering 101% and 100% protection—the joy of simplicity!

Deploy Comodo Antivirus and Pro-Firewall Securities today.
Your total securities are just one click away.
Comodo Antivirus and Pro-Firewall Securities do not rely on testings, marketing and green bars. They rely on word of the mouth as marketing strategy.

Comodo Antivirus and Pro-Firewall Securities.
Trust the red. Forget the green.

 

[stonjean633]

Deploy Comodo Antivirus and Pro-Firewall Securities today.
Your total securities are just one click away.
Comodo Antivirus and Pro-Firewall Securities do not rely on testings, marketing and green bars. They rely on word of the mouth as marketing strategy.

Comodo Antivirus and Pro-Firewall Securities.
Trust the red. Forget the green.

Hehe. Color wars also in the AV now.

 

[Victor M]

perform in the 40% protection range.

I do not know a percentage per se, but I can tell you this: my red team has time and time again penetrated my defenses, part of which includes Microsoft Defender, and MD did not even beep. But the red team is of course intimately familiar with our defenses. We need them to squeeze into those nooks and crannies, and help fortify us.

As I have said before, adversaries test their ware against major AV's before deployment. So does my red team, and so do my pen testers.

To cite one recent example which I can tell you about, my wazuh SIEM agent was exploited to run lol bins. And if it were not for my secure configuration against lol bins, the attacks would have suceeded. Again MD was silent. And wazuh says it's not a vulnerability on their part. My defenses improved with the red team's help. And I await their next test.

I personally think that APT test are useless, because the test orgs take known, documented and dated TTPs from Mitre Att&ck. Any APT worth it's salt would not simply re-use an old TTP in new attacks; there would be changes and improvements.