AV-Comparatives AV-Comparatives - Real World Protection Test Feb-May 2024

[Spartan]

Chart: Comparison

Article: Real-World Protection Test February-May 2024

 

[TairikuOkami]

"Useless" Panda Free performed pretty well.

 

[RoboMan]

Avast/AVG once again consistent with being top notch. I remember the days when they were just "free antiviruses" and nobody would recommend them.

Still waiting for the day they add Application Control.

 

[Templarware]

AVG and Avast are the best, no surprises.. And they do it for free. Microsoft Defender was probably in the default settings though

 

[TairikuOkami]

Microsoft Defender was probably in the default settings though

Well the test was made for people not for MalwareTips.

AVG and Avast are the best, no surprises.. And they do it for free.

This proves that the default is everything. 99% do not change a thing, they would not know how nor why.

 

[kailyn]

Avast/AVG once again consistent with being top notch. I remember the days when they were just "free antiviruses" and nobody would recommend them.

Still waiting for the day they add Application Control.

Even though Avast does well in tests it is often criticized for bugs. It is the usual thing of those saying they do not experience the bugs and those that say they do and they cannot cope with the bugs. Yet its free version still remains the most widely installed free product just because it is free and consumer testing organizations promote it as "good enough." More or less the same thing with AVG.

The majority of the world still wants free software.

Avast lost a lot of credibility because it was selling user data. In fact it just lost against administrative actions and lawsuits over its privacy violations in multiple nations.

AVG and Avast are the best, no surprises.. And they do it for free. Microsoft Defender was probably in the default settings though

1.2% difference is not even statistically meaningful. The difference has to be above 10% to be statistically significant. The flaw in this type of testing is multi-fold. For one it is not deterministic nor predictive. The test results apply only to the malware used in the test. Therefore the test results are only a speculative indication of protections. The second thing is that typical users do not download hundreds of files. The small number of files that users download in normal use means that any statistical difference between the various vendors are leveled-out and make no real difference in day-to-day computing practices; Avast, AVG, Microsoft and others will protect essentially the same.

 

[RoboMan]

Even though Avast does well in tests it is often criticized for bugs. It is the usual thing of those saying they do not experience the bugs and those that say they do and they cannot cope with the bugs. Yet its free version still remains the most widely installed free product just because it is free and consumer testing organizations promote it as "good enough." More or less the same thing with AVG.

The majority of the world still wants free software.

Do you think the huge user database Avast/AVG has directly affects its detection capabilities, hence why it's first in the test? I'm sure the telemetry gathered from the millions users' infections has something to do.

 

[kailyn]

Do you think the huge user database Avast/AVG has directly affects its detection capabilities, hence why it's first in the test? I'm sure the telemetry gathered from the millions users' infections has something to do.

The size of the userbase does factor into performance but it is only a small piece of the puzzle. If userbase were the primary factor then Microsoft would blow every other security product out of the water since its base are those in its 1+ billion Windows systems that provide telemetry to its threat intelligence ecosystem. What is never discussed - because there is virtually no visibility into any of it - is how wildly different vendors' backends and technologies vary.

Beginning in 2016 Avast and AVG made infrastructure investments that improved their detection rates.

The other thing to be aware of that vendors have gamed AV Comparatives tests for years. Every one of the vendors knows the criteria and the test methodologies. Not one bit of it is kept secret from them. Most of them tune things to optimize test results. I don't think Emsisoft did. In fact, besides the cost (approximately $50,000 per test) Emsisoft I recall because it felt cheated by those that were tuning their product and AV Comparatives did not do a very good job about policing it. Some are better at the tuning than others. The ones that are blatant about it get busted and banned by AV Comparatives, like QiHoo.

The other thing you never learn about are which vendors that do a good job of making improvements to their products based upon lab feedback. Some merely make signatures for those items they missed while others make efforts to improve their other non-signature capabilities. None of the vendors are very forthright about it unless they are called-out, as in the case of Google's Project Zero. When called-out Kaspersky reacts quickly. Norton is reasonably quick. Microsoft is always the slowest to react and that's not because it is dragging its feet. Microsoft's protections are spread across many different points in its operational structure and it takes time to coordinate and orchestrate within such a large environment.

The only thing you can do is look at test results across multiple test labs over a number of years to get a real understanding of how well a product will protect. A product like Avast will do great in AV Comparatives tests but then fail in MRG Effitas simulated banking trojan tests. It's just an example, but the product generally does better than average.

5 Stars & 100% Bars mislead uninformed test result readers to the extent that they have little to no understanding of all the variables involved in testing, what is actually being measured, and what is not included in the measurements. It's the reason that people select the 5 Stars & 100% Bar products and then end up with malware or worse. But the world is a bunch of societies that demand shallow measurements that people do not have the understanding to realize what they're actually looking at.

 

[RoboMan]

The size of the userbase does factor into performance but it is only a small piece of the puzzle. If userbase were the primary factor then Microsoft would blow every other security product out of the water since its base are those in its 1+ billion Windows systems that provide telemetry to its threat intelligence ecosystem. What is never discussed - because there is virtually no visibility into any of it - is how wildly different vendors' backends and technologies vary.

Beginning in 2016 Avast and AVG made infrastructure investments that improved their detection rates.

The other thing to be aware of that vendors have gamed AV Comparatives tests for years. Every one of the vendors knows the criteria and the test methodologies. Not one bit of it is kept secret from them. Most of them tune things to optimize test results. I don't think Emsisoft did. In fact, besides the cost (approximately $50,000 per test) Emsisoft I recall because it felt cheated by those that were tuning their product and AV Comparatives did not do a very good job about policing it. Some are better at the tuning than others. The ones that are blatant about it get busted and banned by AV Comparatives, like QiHoo.

The other thing you never learn about are which vendors that do a good job of making improvements to their products based upon lab feedback. Some merely make signatures for those items they missed while others make efforts to improve their other non-signature capabilities. None of the vendors are very forthright about it unless they are called-out, as in the case of Google's Project Zero. When called-out Kaspersky reacts quickly. Norton is reasonably quick. Microsoft is always the slowest to react and that's not because it is dragging its feet. Microsoft's protections are spread across many different points in its operational structure and it takes time to coordinate and orchestrate within such a large environment.

The only thing you can do is look at test results across multiple test labs over a number of years to get a real understanding of how well a product will protect. A product like Avast will do great in AV Comparatives tests but then fail in MRG Effitas simulated banking trojan tests. It's just an example, but the product generally does better than average.

5 Stars & 100% Bars mislead uninformed test result readers to the extent that they have little to no understanding of all the variables involved in testing, what is actually being measured, and what is not included in the measurements. It's the reason that people select the 5 Stars & 100% Bar products and then end up with malware or worse. But the world is a bunch of societies that demand shallow measurements that people do not have the understanding to realize what they're actually looking at.

Thanks for your input

 

[oldschool]

Therefore the test results are only a speculative indication of protections.

We can say the same for most, if not all, tests. They are snapshots in time, like election polls.

 

[monkeylove]

The size of the userbase does factor into performance but it is only a small piece of the puzzle. If userbase were the primary factor then Microsoft would blow every other security product out of the water since its base are those in its 1+ billion Windows systems that provide telemetry to its threat intelligence ecosystem. What is never discussed - because there is virtually no visibility into any of it - is how wildly different vendors' backends and technologies vary.

Beginning in 2016 Avast and AVG made infrastructure investments that improved their detection rates.

The other thing to be aware of that vendors have gamed AV Comparatives tests for years. Every one of the vendors knows the criteria and the test methodologies. Not one bit of it is kept secret from them. Most of them tune things to optimize test results. I don't think Emsisoft did. In fact, besides the cost (approximately $50,000 per test) Emsisoft I recall because it felt cheated by those that were tuning their product and AV Comparatives did not do a very good job about policing it. Some are better at the tuning than others. The ones that are blatant about it get busted and banned by AV Comparatives, like QiHoo.

The other thing you never learn about are which vendors that do a good job of making improvements to their products based upon lab feedback. Some merely make signatures for those items they missed while others make efforts to improve their other non-signature capabilities. None of the vendors are very forthright about it unless they are called-out, as in the case of Google's Project Zero. When called-out Kaspersky reacts quickly. Norton is reasonably quick. Microsoft is always the slowest to react and that's not because it is dragging its feet. Microsoft's protections are spread across many different points in its operational structure and it takes time to coordinate and orchestrate within such a large environment.

The only thing you can do is look at test results across multiple test labs over a number of years to get a real understanding of how well a product will protect. A product like Avast will do great in AV Comparatives tests but then fail in MRG Effitas simulated banking trojan tests. It's just an example, but the product generally does better than average.

5 Stars & 100% Bars mislead uninformed test result readers to the extent that they have little to no understanding of all the variables involved in testing, what is actually being measured, and what is not included in the measurements. It's the reason that people select the 5 Stars & 100% Bar products and then end up with malware or worse. But the world is a bunch of societies that demand shallow measurements that people do not have the understanding to realize what they're actually looking at.

It's a choice between published test results and accusations by forum anons of gamed results.

 

[blackice]

I’M GOING TO PAY ALL OF THE MONEY FOR DETECTION OF JUST A FEW MORE SAMPLES THAT I WOULD NEVER ENCOUNTER IN REAL LIFE!!!!!!

 

[Digmor Crusher]

I’M GOING TO PAY ALL OF THE MONEY FOR DETECTION OF JUST A FEW MORE SAMPLES THAT I WOULD NEVER ENCOUNTER IN REAL LIFE!!!!!!

Why are you yelling, we all know these tests are for entertainment purposes only.

 

[Jengo]

G DATA ! Always top contender. I dont care Avast score more, they implemented too much in their software compared to GDATA. Less is More.

 

[Behold Eck]

Why are you yelling, we all know these tests are for entertainment purposes only.

Please don`t make him even angrier, DON`T!!!

Regards Eck

 

[blackice]

Why are you yelling, we all know these tests are for entertainment purposes only.

YELLING IS FOR ENTERTAINMENT PURPOSES ONLY!!!

 

[kailyn]

We can say the same for most, if not all, tests. They are snapshots in time, like election polls.

This is a fact of protection life cycle, with some being much more resistant to changes in the threat landscape and others not so much.

If you would throw a lot of 16 and 32 bit malware at signature based solutions, people would have meltdowns and indignation that they are not detected. It does not matter to them that those malware cannot even run on their 64 bit system but we're talking about people here and their irrational beliefs so... Also malware signatures do not last forever. There are short term signatures and then those in turn get converted into long term signatures. Beginning about 5 years older signatures begin to be purged from signature databases. Like, vendors cannot keep every signature that they ever created and cache them. It is not practical nor feasible.

Just look at the range of results from one test to the next. In particular, the synthetic banking tests by MRG Effitas. One solution does great, but then fails the next time. They fix it, and do great, but then fail on the next test. Solutions such as Avast, Norton, and ESET seem to follow that cycle, or they did. Not sure if they even participate in MRG tests any longer. Every time MRG switches up the tests you can pretty much predict which one will fail and which will pass.

The protections that are most resistant to changes in the threat landscape are default deny: that which is not allowed is denied or virtualization. But those too have to receive periodic updates, mostly because Microsoft makes changes to their OS which breaks stuff. Not because there is a revolutionary change in the threat landscape. That landscape is pretty much unchanged in terms of Attack Kill-Chain and Tactics, Techniques, and Processes (TTPs). LOLBins, GTFOBins, etc have been more or less static for years. Recently there was a few Azure LOLBins discovered, but customers do not have access to those; only Microsoft does and they have either locked them down on their infrastructure or made changes to how they work to mitigate their abuse.

One cannot extrapolate any test results with absolute certainty, but people do it all the time. Tests are not predictive over the long term and definitely not deterministic.

 

[kailyn]

It's a choice between published test results and accusations by forum anons of gamed results.

Vendors gaming the tests is common knowledge amongst those in the know within the industry. It is no false accusation. It is a statement of fact.

 

[monkeylove]

Vendors gaming the tests is common knowledge amongst those in the know within the industry. It is no false accusation. It is a statement of fact.

You can make it a statement of fact by proving it.

 

[kailyn]

You can make it a statement of fact by proving it.

Just a single example:

Anti-virus product caught cheating by independent test agency

AV-Comparatives, one of the world’s leading independent testers of anti-virus products, says that it has uncovered that at least one product isn’t playing by the rules.

grahamcluley.com

Tencent, Qihoo antimalware firms are accused of cheating, stripped of rankings in antivirus tests

Several independent testing firms claim the accused companies submitted programs specially tuned to ace the benchmarks.

www.pcworld.com

There are discussions here at MT about vendors optimizing their product for the AV-Comaparatives tests. Fabian Wosar made posts here here about the problem and it does not just apply to Tencent and Qihoo.

Did you know that the signature engines on VirusTotal are not the ones that are in the consumer and enterprise products (this has always been the case)? Did you know that vendors copy each others' signatures from VirusTotal (Proven by Eugene Kaspersky and his team)?