AV-TEST Advanced Threat Protection (ATP) AV.TEST test January - June 2025

[Victor M]

Or maybe I shouldn't say these APT Tests are useless. They do prove that those AV vendors can overcome some script kiddies who mimic past APT TTPs.

 

[Zero Knowledge]

All hail McAfee! Now I wish I didn't uninstall it when I got my current machine. Offering a 3 month license and renew @ a reasonable price of $79 USD a year was well & truly appreciated.

The problem is the goal posts have moved and attackers have pivoted from home, to enterprise, to cloud and now it's all about the browser and identity management tools/software. From my understanding and experience attackers & APT are more likely to drop a backdoor and implants on hardware and stealing tokens/session cookies than drop info stealers and malware these days. The risk of getting detected and your fine crafted malware getting a nice blog post on projectzero's website is very high these days.

In the end you have to laugh just as the big boys in intelligence probably do @ home users trying to protect themselves against huge government spy agencies & APT's.

 

[Parkinsond]

I personally think that APT test are useless

I hope you and the rest of experts on MT get a good, long discussion based on you declare a list of the valid tests to be considered by MT users according to your approval of methodology; if there is no approved tests at all, just say it frankly.
I have a feeling that approval of denial of each single test of each singe lab or tester is according the results per test; if it is in agreement of my own conviction, I praise the test, if it is not, the test is flawed!

 

[monkeylove]

AV-Test, like all AV labs, focuses upon certain things and it is very rare for any participating AV to perform poorly. I cannot recall an instance of one scoring 4 and lower for a very long time.

If a user does not regularly use a Standard User Account (SUA), is in a high risk environment, a prolific downloader, does significant online transactions, has a need to protect sensitive data, then Microsoft Defender at its default settings is not the best option. Against the newest and certain classes of malware and attacks, it is difficult to recommend Microsoft Defender confidently.

Nobody tests and reports Malware Defender at default configurations, at maximum configurations, and then in combination with other Microsoft security features. It is considered "permutation testing" and Microsoft is not going to pay for it.

Microsoft Security - which includes many components of which Malware Defender is only a single part - properly configured (for maximum security; security is prioritized over convenience/productivity) and managed is robust, but those features are not available to consumers.

Malware Defender at default configuration provides "adequate" protection for low-risk, everyday, typical consumer use.

Real-world, practical results - which means reports from the field, as opposed to lab test reports - indicate that all AV fail at a rate that is significant at the moment of truth. The % of active infections (even if it is only PUA/PUP) on systems with "top performing" AV is astounding.

All that said, my observations of consumer infected systems are mostly those under the control of 1) people ignorant/with a basic understanding of security, 2) have no inclination to be secure or make very little effort to be secure, and 3) even when instructed on security, they don't change their behaviors.

If I was given 1,000,000 Euros and instructed use it for betting on security solution outcomes, then I would bet only 1 Euro on any of them. Because all the meaningful and secure outcomes are largely dependent upon the people, their decisions, and their behaviors. Plus things out of their control which degrade the AV performance, such as internet outages, problems with their devices or applications or OS, etc.

One should not extrapolate or project security performance or expectations of security performance based upon test lab results. And since no vendor - not even Comodo - will provide the detailed field reports to the world, nobody except for a very few people know the truth about AV. Get 1,000 top pen testers and malcoders, throw them against every single security solution out there, even with different configurations, and all marketing claims and claims of "You are protected" would be unraveled. Some AV will do worse than others overall. Some AV will be weaker and stronger in different protection areas. But overall, the results would be dismal with protection rates in the 40% or lower range.

Security solution testing is about the same as testing vehicle engine oil to ensure that it meets a minimum standard of both specification and quality. However, when testing oil in real world conditions it becomes clear that negative outcomes are significant after reviewing the full available data. Many times the oil itself is not the variable that led to the failure.

The AV industry has capitalized on peoples' propensity to fully trust based upon "Five Stars and All Green Bars" lab test results. Those tests just establish a minimum baseline of performance using concocted scenarios that do not account for all the stuff that happens in the real world. So, in short, the results are "synthetic" and it takes a lot of knowledge and experience to understand what they really say, and most importantly, what they do not say.

Consumers have to rely on something, and not just for vehicles and AVs but for many other things. In which case, these tests and others remain relevant.

 

[monkeylove]

We are left with test results, system impact, costs, and annoyances.

For the first, it's pointless to argue that AVs are the same because the other two points remain.

For the second, there are tests and one can do that by oneself. And from what I gathered, there appears to be only little system impact given Windows Defender, the same with almost all security features turned on, and the same with almost all features turned plus some (not all) third-party AVs installed. That means system impact isn't that much of a factor anymore.

Third are costs. Given online stores selling McAfee for less than two euros for a device per year, it looks like those aren't a serious factor, either, unless one buys from the manufacturer. There's also the issue of being scammed, but it looks like most sellers are legit, and the price is still reasonable even in well-known online software stores, e.g., 4 USD for Kaspersky Standard, one device/one year.

Lastly are annoyances, and those could be worse, e.g., you turned on Controlled Folder Access and can't access some files, or the AV decides to quarantine a safe app and either tells you or not, and probably popups for some AVs (but not likely if they're paid and set to game mode or similar).

With that, Defender looks fine, and with almost all features in Windows Security turned on, but the results for various tests aren't that good in some cases, and there's like a good reason why various security features are turned off by default. And if costs and system impact for various AVs are negligible, then one can just look for the AVs with the best balance of system impact, real-time protection, and malware protection across several years but also few false positives and preferably almost set-and-forget, and then buy them for 2-4 USD a year for each device.

 

[ForgottenSeer 114717]

Consumers have to rely on something, and not just for vehicles and AVs but for many other things. In which case, these tests and others remain relevant.

I never said a word about the relevance of these tests. What I said was that they are limited in what they prove.

AV test labs do not exist on behalf of consumers. They exist to create a marketing tool for the AVs that participate in the tests.

The flaw in AV test lab testing are the "5 Stars and All Green Bars" cannot be understood by the average world citizen. They do not have the knowledge to understand what the tests say, and more importantly what they do not say and that they cannot be extrapolated generically to the entirety of real world possibilities.

Most of the world's population does not even know about AV test labs and AV lab test results.

 

[ForgottenSeer 114717]

I have a feeling that approval or denial of each single test of each singe lab or tester is according the results per test; if it is in agreement of my own conviction, I praise the test, if it is not, the test is flawed!

Well, what you state here describes how people interpret and judge stuff online across all types of content sources, regardless of the subject matter.

This is just the nature of people and the digital world.

 

[ForgottenSeer 114717]

I do not know a percentage per se, but I can tell you this: my red team has time and time again penetrated my defenses, part of which includes Microsoft Defender, and MD did not even beep.

Microsoft Defender does not do protect well against malware that it does not block.

To cite one recent example which I can tell you about, my wazuh SIEM agent was exploited to run lol bins. And if it were not for my secure configuration against lol bins, the attacks would have suceeded.

LOLBin blocking or heavily restricted-constrained LOLBins remain the single most effective method of breaking the vast majority of kill chains. What remains are kernel/firmware malware/exploits, malicious bootloaders, etc - all operating at a low level - which some, probably most, will bypass ANY security solution. If that happens, there is a chance to break the kill chain by blocking abused LOLBins, but if the attack is operating at an adequately low level and needs nothing else, ALL security solutions are beat.

Obtain a contract with Microsoft Security and one of the first things they will advise are "least privilege (SUA)," "attack surface reduction (install only what is absolutely needed; block users from executing and installing stuff)," "least functionality (disable as many features and services as possible)," and "block as many of the LOLBins as is possible that ship with Windows."

When a client objects to any of these the general reply from Microsoft Security is "change your ways" or "find a different way." "If not, accept and assume the risk. You are both accountable and responsible for your decisions."

 

[Victor M]

valid tests to be considered by MT users

One has to consider the cadence of Mitre Att&ck, which is updated bi-annually. That is like saying an AV providing malware signatures bi-annually. Would that AV still be useful ? Would that APT test still be useful ?

Like I said yesterday, the AVs that passed the test are still good for blocking script kiddies who mimic past ATP TTPs. And there are a Lot of script kiddies. But the ATP test is not useful for determining how good an AV is at blocking real ATPs.

Use what you are given, and then keep updating your defenses as revealed by purchased threat intelligence, OSINT and security blogs, your red team and pen tests. Don't count on the AV being able to protect you.

 

[ForgottenSeer 123960]

It is important to distinguish that these are baseline tests, designed to establish a foundational benchmark.

 

[Parkinsond]

Don't count on the AV being able to protect you

I don't; actually I am sa cautious to the extent that sometimes I feel I am not in need.

 

[Parkinsond]

It is important to distinguish that these are baseline tests, designed to establish a foundational benchmark.

Test as much as they need; I move between MD and AVG free (K free, previously), and not all the tests in the world can make me change the limited route

 

[Victor M]

sometimes I feel I am not in need.

That is not true. You can keep updating your defenses, and post and Update the details of your configuration here on MT.

 

[ForgottenSeer 114717]

One has to consider the cadence of Mitre Att&ck, which is updated bi-annually. That is like saying an AV providing malware signatures bi-annually. Would that AV still be useful ? Would that APT test still be useful ?

Like I said yesterday, the AVs that passed the test are still good for blocking script kiddies who mimic past ATP TTPs. And there are a Lot of script kiddies. But the ATP test is not useful for determining how good an AV is at blocking real ATPs.

Use what you are given, and then keep updating your defenses as revealed by purchased threat intelligence, OSINT and security blogs, your red team and pen tests. Don't count on the AV being able to protect you.

Security is not software. It is a process (that takes way more than just 'security solution for the masses'; it takes user knowledge, users changing their behaviors, users wanting to prioritize security and accepting the required overhead,...). And with that - the mention of the word "users," the entire security process is unraveled.

Offering security software as the protection model to non-professional users as their primary online security is such a flawed security model as to almost be worthy of being labeled "A scam."

Even the professionals who would provide secure configurations, baselines, and management within enterprise and government are undermined and compromised by the CEO that is angry because he could not willy-nilly download WebEx and install it to ProgramData or the Program Manager that is placed in overall accountability and responsibility, but yet they don't know the first thing about security and don't listen to their competent security staff.

The "Users want to use stuff" paradigm is dinosaur thinking.

 

[Trident]

Security is definitely a process, but security software is not as bad as people want to think.
Many attacks are blocked successfully, it should be kept in mind that there many malicious actors (probably millions), most of them run several campaigns and every campaign results in millions of emails and malicious files.

Majority of failed defences are not some sort of EDR failing, it is corporate passwords reuse, happy clicking, phishing and so on.

For consumers, the picture is not so apocalyptic, solutions like Avast, McAfee, Bitdefender and so on with effective web blocking fend off most of the dodgy stuff in the browser and malware is pretty much never downloaded.

For businesses, the problem is a lot of them do not train employees and don’t want to pay more than €2.59 per account per month on email security—one that will actually remove Phishing and malicious attachments instead of merely injecting banners, which nobody notices.

 

[Parkinsond]

For businesses, the problem is a lot of them do not train employees and don’t want to pay more than €2.59 per account per month on email security—one that will actually remove Phishing and malicious attachments instead of merely injecting banners, which nobody notices.

Also corporate sector has more holes than home users for file sharing, shared printers, and remote working.

 

[ForgottenSeer 114717]

Majority of failed defences are not some sort of EDR failing, it is corporate passwords reuse, happy clicking, phishing and so on.

People problem.

For consumers, the picture is not so apocalyptic, solutions like Avast, McAfee, Bitdefender and so on with effective web blocking fend off most of the dodgy stuff in the browser and malware is pretty much never downloaded.

Even consumers that can afford paid security software, never bother. Even then, whether they choose to install or use security software, paid or free, it only provides limited overall protections. Security software is nothing more than a limited insurance policy providing protections within a small scope of what is required in total.

I never stated nor implied that protecting localhost with security software is the most important thing for consumers to do to protect themselves. In fact, it is not.

There is far too much focus on software, and much too little on what is required to be secure. The importance of security software in being secure across the entire digital space is way over-exaggerated.

Most entities - consumer, enterprise, or government - want to solve their security problems with software, and not the stuff that will actually provide a level of security far above what software can provide.

For businesses, the problem is a lot of them do not train employees and don’t want to pay more than €2.59 per account per month on email security—one that will actually remove Phishing and malicious attachments instead of merely injecting banners, which nobody notices.

People problem.

People are always the problem. ALWAYS.

 

[ForgottenSeer 114717]

Also corporate sector has more holes than home users for file sharing, shared printers, and remote working.

Productivity and profit will always be prioritized before security, even within regulated industries that mandate a heavy security requirement burden upon the industry participants. Even when the organizations are required to be assessed and audited by government or independent 3rd party auditors, and the organization and/or people within it can be charged civilly or criminally for non-compliance.

People problem.

 

[stonjean633]

Also corporate sector has more holes than home users for file sharing, shared printers, and remote working.

More holes with more Cyber Defenses BUT holding more Confidential Data that is prone to Cyber Attacks.

A cyber attack on an enterprise level needs proper reporting to auditors and are responsible for being transparent to customers affected. If Cyber professionals are not doing their job, likely they will in the CNN headlines sooner or later

In home/consumer world, it's the opposite.

 

[Parkinsond]

Productivity and profit will always be prioritized before security

According to the nature of the facility; if data encryption and paying ransom chances are not low, they will cost way more than the security solutions fees for years.