Advice Request Why I think testing "labs" are useless

[RoboMan]

This is my opinion, and the fact I'm dropping it here, in the general security discussion section, is with hope we can establish a serious discussion about this subject; about why I think testing labs are useless.

We often come across testing labs, some which we believe to be more accurate than others. For example, we've seen PC MAG "reviews", many which are garbage and $$ focused. While, in the other hand, many here seem to trust sites such as AV-Comparatives or AV-Test. We often recommend to take these tests with a grain of salt, but, do we? Do we really? Do all average users really?

Take as an example, TPSC (Leo) and the comments section, full of users saying "wow, ESET sucks, worst AV ever, it scores 93% protection only", because Leo threw 1K files and then did basic math to calculate a protection rate. Well, I believe testing labs have the same impact.

I wish not to imply labs like AV-Comparatives are a fraud; but instead they aren't accurate.

CASE SCENARIO

For example, they test 20 antivirus software, with a pack of 15,000 malware files. Thrown all at each antivirus, according to several factors like database version, tweak settings, each can perform different (one single antivirus can perform A today, and B tomorrow). At the end of the day, they will do simple math to calculate "protection scores", and will tell you that, for example, Kaspersky detected 19,990 files and it's protection score is, for example, 99.9%; and Windows Defender detected 18,998 and its protection score is, for example, 94.5%, so it scores last out of the 20 tested antivirus.

AVERAGE USERS RESPONSE (90% of internet users)

Reading this, average users will go online claiming Windows Defender is TRASH and sucks as an antivirus since it scored LAST of all antivirus, therefore it's the worst. How can Microsoft protect their users will this garbage?! It misses almost a thousand malware files in one test! I am probably already infected without having noticed!

When, in real-life, average users WILL NEVER EVER face 20,000 threats in one day, and Windows Defender will still have a 100% protection rate against the 2 or 3 possible malware they face online each month.

MY COMMENTS

This works entirely as marketing. Suddenly seeing 5 or 6 antivirus score 99.9% protection rate, and seeing my humble Windows Defender (or any other) score 94%, will make me think I'm so exposed and vulnerable, feeling the NECESSITY to buy or switch to the best one, which of course, is a lab's gold partner. Snake oil on its best. Creating irreal scenarios, forcing antiviruses to work on a case that they should NEVER face in their lives, creating paranoia amongst users and giving them the false sensation that lab tests are a great tool to be informed.

I haven't been infected in ages. Any family member has been infected in ages, and they some use Windows Defender, one pays for Kaspersky, another one uses Avast. The reason why they haven't been infected is because they're ordinary users, whom do not recieve targeted attacks, and maybe face a couple of possible threats a month trying to download a game, torrent, or crack a software. Three or four cases a month, where the antivirus, no matter which one, always protects them, because its common, old threats. Therefore, the only actually application I see for lab tests is corporate antivirus software, which maybe will recieved special targeted attacks and may need some more feedback on the subject.

 

[Cortex]

I agree entirely, thousands of viruses/malware never suddenly appear on your desktop with AV protection disabled, that is not going to happen not on Earth anyway - In real life scenario they would have to get there somehow usually via a browser with it's protection - Most of the time the persecuted AV never has the opportunity to have a reboot either - I now look a these tests for entertainment only as for me they are not realistic?
.

 

[EndangeredPootis]

Agreed, I also think they are shady due to the fact they simply never show their tests taking place and the inconsistencies in their tests aswell.

A product can become last place because of a having 1% less detection ratio, how would the average user respond to that? without actually looking at the detection rates they would think that product must be absolute garbage, its probably how the rumor of windows defender being garbage started.

I also tested Kaspersky Security Cloud Free in my spare time and it scored a detection rate of 73% of samples that were over 2 days old (this was on demand, I opened the folder containing malware and let kaspersky do its, albeit slow, job), yet in tests such as by Leo he got a 100% detection ratio with over 1500 samples, and he did it on execution so kaspersky had to take care of 3-4 sample launches every second without any prior analysis, it just does not make any sense.

 

[danb]

Yes, designing a perfectly valid AV test is no easy task. The question is, what specifically would you guys do to design a better test?

Most people are not going to get infected tomorrow. Most people will not get into a car accident tomorrow either, but if they do they better be wearing their seat belt.

 

[RoboMan]

Yes, designing a perfectly valid AV test is no easy task. The question is, what specifically would you guys do to design a better test?

Most people are not going to get infected tomorrow. Most people will not get into a car accident tomorrow either, but if they do they better be wearing their seat belt.

That's exactly my point! In what real world, can me (Robbie) decide which antivirus is the best for you (Dan)? It's nonsense. I can even test 1' to 20 antiviruses in your PC right now over TeamViewer and decide Kaspersky protects you the best, and maybe tomorrow that's not true anymore. I believe finding a good antivirus for YOU depends entirely on YOU. It's a daily task of trying to find something that suites you.

 

[sepik]

I believe finding a good antivirus for YOU depends entirely on YOU. It's a daily task of trying to find something that suites you.

'nuff said

 

[James246]

This is my opinion, and the fact I'm dropping it here, in the general security discussion section, is with hope we can establish a serious discussion about this subject; about why I think testing labs are useless.

We often come across testing labs, some which we believe to be more accurate than others. For example, we've seen PC MAG "reviews", many which are garbage and $$ focused. While, in the other hand, many here seem to trust sites such as AV-Comparatives or AV-Test. We often recommend to take these tests with a grain of salt, but, do we? Do we really? Do all average users really?

Take as an example, TPSC (Leo) and the comments section, full of users saying "wow, ESET sucks, worst AV ever, it scores 93% protection only", because Leo threw 1K files and then did basic math to calculate a protection rate. Well, I believe testing labs have the same impact.

I wish not to imply labs like AV-Comparatives are a fraud; but instead they aren't accurate.

CASE SCENARIO

For example, they test 20 antivirus software, with a pack of 15,000 malware files. Thrown all at each antivirus, according to several factors like database version, tweak settings, each can perform different (one single antivirus can perform A today, and B tomorrow). At the end of the day, they will do simple math to calculate "protection scores", and will tell you that, for example, Kaspersky detected 19,990 files and it's protection score is, for example, 99.9%; and Windows Defender detected 18,998 and its protection score is, for example, 94.5%, so it scores last out of the 20 tested antivirus.

AVERAGE USERS RESPONSE (90% of internet users)

Reading this, average users will go online claiming Windows Defender is TRASH and sucks as an antivirus since it scored LAST of all antivirus, therefore it's the worst. How can Microsoft protect their users will this garbage?! It misses almost a thousand malware files in one test! I am probably already infected without having noticed!

When, in real-life, average users WILL NEVER EVER face 20,000 threats in one day, and Windows Defender will still have a 100% protection rate against the 2 or 3 possible malware they face online each month.

MY COMMENTS

This works entirely as marketing. Suddenly seeing 5 or 6 antivirus score 99.9% protection rate, and seeing my humble Windows Defender (or any other) score 94%, will make me think I'm so exposed and vulnerable, feeling the NECESSITY to buy or switch to the best one, which of course, is a lab's gold partner. Snake oil on its best. Creating irreal scenarios, forcing antiviruses to work on a case that they should NEVER face in their lives, creating paranoia amongst users and giving them the false sensation that lab tests are a great tool to be informed.

I haven't been infected in ages. Any family member has been infected in ages, and they some use Windows Defender, one pays for Kaspersky, another one uses Avast. The reason why they haven't been infected is because they're ordinary users, whom do not recieve targeted attacks, and maybe face a couple of possible threats a month trying to download a game, torrent, or crack a software. Three or four cases a month, where the antivirus, no matter which one, always protects them, because its common, old threats. Therefore, the only actually application I see for lab tests is corporate antivirus software, which maybe will recieved special targeted attacks and may need some more feedback on the subject.

I agree the average user is not going to face 20000 viruses at once but could face any one at any point of time, from the 350000 that are apparently released every day, so the tests are not completely useless but also are not the perfect representation of true security.. Despite what these test results might imply it is most unlikely that any antivirus is going to be able to stop everything.
A good addition to a simple antivirus would be :-
1) Voodoo Shield or
2) Spyshelter Firewall (for more advanced users) or
3) Comodo (with the Cruel Lady settings)
There are of course one or two others.

 

[toto]

I think that's why for the average user (not a happy clicker) it's more important to see the performance loss by the specific product as most companies are good enough in regards to protecting the system from possible threats encountered when browsing or downloading a file or any everyday task.

 

[Atlas147]

Definitely a fair point about testing labs throwing AVs in a scenario that they would otherwise never face ever. However such a test also provides certain insights as to how behavioural blockers are adding to the mix. Given that many of the AVs like to use bitdefender or avira signatures, it would be safe to assume that most if not all of them can cover at least 95% of all malware. What separates the good from the best in AV tests are those last few percentages that most AVs don't have signatures for. And it's not to say that a higher percentage would mean better BBs. There is just not enough information released on how the threats were blocked based on the the different AVs.

However there are a few things to note about windows defender scoring low. Yes it makes sense that WD scores much lower than other AVs and is perceived as the weaker product. And honestly that is 100% true, WD is a weaker product in general, the numbers show it. While the average user wouldn't face thousands of malware threats everyday, a higher protection score gives users a slightly greater sense of security. If I were to protect my family computer with something, I would 100% choose a more robust product because even though I know that I won't be clicking on phishing or malicious links, my family members might not be able to identify such links.

Another side of throwing unrealistically huge amounts of malware at an AV is that an average Joe won't be able to replicate such an attack to evaluate AV products. AV test and AV comparatives are providing data from such tests to the general public. However, the choice on which AV is better for your own use cases are still dependent on a ton of other things, and everyone should try a few different AVs until they find one that suits them.

 

[monkeylove]

I don't see much sense in claims that "my PC has never been infected for years" because people use computers differently. Neither do I see the point in arguing that something that a program is either the best or "garbage."

That said, I look at tests because it's better than nothing, and what I'm looking for isn't what's "garbage" or not but what provides the best protection without affecting performance that much.

With that, I select free versions of the programs with the best protection given different tests, and then I do the ff.

1. I remove the ones that have popup notifications for upgrades or notifications that can't be disabled;

2. I install each one and see which slows down the PC the least;

3. From there, I select one which has the most features and use it for all PCs.

 

[James246]

"WD is a weaker product in general"
Actually thee are plenty of test results that suggest WD is not weaker in general, certainly it is not the equal of "K" or "BD" but there are others that finish below it in some lab results.. However I think the Average Joe would be mad to use WD without some form of complementary program (for example VS).

 

[Nightwalker]

"WD is a weaker product in general"
Actually thee are plenty of test results that suggest WD is not weaker in general, certainly it is not the equal of "K" or "BD" but there are others that finish below it in some lab results.. However I think the Average Joe would be mad to use WD without some form of complementary program (for example VS).

I respectfully disagree, anyone who knows how to use tools like VoodooShield, Sandboxie and HIPS like software doesnt need them at first place.

IMO Windows Defender is more than enough for everyone, contrary to common sense in security forums, the average joe has never been safer than now.

 

[Vitali Ortzi]

Unfortunately common users in my country have been Targeted by APT
Because of the region and political interest.

 

[James246]

I respectfully disagree, anyone who knows how to use tools like VoodooShield, Sandboxie and HIPS like software doesnt need them at first place.

IMO Windows Defender is more than enough for everyone, contrary to common sense in security forums, the average joe has never been safer than now.

I completely disagree I have a friend who worked for the intelligence community (granted he is not the Average Joe) who on his own personal computer once clicked on something he thought innocuous that sliced straight through Kaspersky, it didn't get him though because he had an "extra special security layer" on his machine.
VoodooShield with WLC (which dramatically reduces user interaction) is easy for the Average Joe to learn and once familiarity is established, one can simply ignore browsing alerts and move on to the next site.
The additional protection VoodooShield will give to WD is comparable to the difference between an Apollo Space Rocket and a Pogo Stick.

 

[danb]

That's exactly my point! In what real world, can me (Robbie) decide which antivirus is the best for you (Dan)? It's nonsense. I can even test 1' to 20 antiviruses in your PC right now over TeamViewer and decide Kaspersky protects you the best, and maybe tomorrow that's not true anymore. I believe finding a good antivirus for YOU depends entirely on YOU. It's a daily task of trying to find something that suites you.

Fair enough, but most users have zero interest in learning about cybersecurity... they simply want to be secure. I was just curious if anyone had any recommendations on how the labs can improve their testing practices, as I am sure they are open to suggestions.

To me the biggest issue with lab testing is the fact that they are only able to utilize known malware. This is no different than the C19 being a novel virus... our defenses our useless against it.

 

[Outpost]

@Robbie
I totally agree with you. But the solution is simple: do not read them, do not consider them.
I don't even read user posts that mention them, let alone the related answers of other users.
When some friends or acquaintances ask me for advice, because they have read one of these tests, I answer that I don't care what they have read or heard.
I have more important things to do than waste time reading these tests.

 

[Nightwalker]

I completely disagree I have a friend who worked for the intelligence community (granted he is not the Average Joe) who on his own personal computer once clicked on something he thought innocuous that sliced straight through Kaspersky, it didn't get him though because he had an "extra special security layer" on his machine.
VoodooShield with WLC (which dramatically reduces user interaction) is easy for the Average Joe to learn and once familiarity is established, one can simply ignore browsing alerts and move on to the next site.
The additional protection VoodooShield will give to WD is comparable to the difference between an Apollo Space Rocket and a Pogo Stick.

What was the infection vector? A browser exploit (so rare these days)? If it was the case than VD could be useful, but if he clicked on photos.jpg.ps1 or cracked GTA V.exe how something like VD could help?

Anti executable is just an annoyance for the average home user, most infections are via fake keygens, cracks and torrents in general and the user is always willing to disable the protection to run it, thats why default-deny doesnt work in the domestic scenario but can be a very effective policy in enterprise usage (the user cant bypass it).

Anti executable, HIPS, default-deny only works in a home scenario usage if the user cant bypass or disable it.

 

[danb]

I respectfully disagree, anyone who knows how to use tools like VoodooShield, Sandboxie and HIPS like software doesnt need them at first place.

IMO Windows Defender is more than enough for everyone, contrary to common sense in security forums, the average joe has never been safer than now.

What is easier to respond to, a UAC prompt or a VS prompt? What is SAFER to respond to, a UAC prompt or a VS prompt? 95%+ of our users are average Joes, and we have a lot of users. I get about 4-5 password reset / account support emails a day, and I have had less than 5 in eight years ask about how to use VS.

When the common denominator is WD, a second layer of some kind is a good idea. In other words, malware authors are going to bypass WD first, since it is the common denominator.

 

[plat]

My longstanding perspective:

It's been a longstanding thing that the comparatives are more like marketing front-ends for the various antivirus vendors. Often the big ones will sport various medals and such on their homepages. Even Microsoft, Robbie.

One conversation in the past involved one of Defender's components being disabled and "tweaked" and then naturally scoring way low. Manipulating the AVs behind the scenes--big time NO NO Someone from a comparatives lab got very huffy and puffy. But you don't do that and then purvey "scientific" data. You can still see this on YouTube sometimes, to forward an agenda. How clean are these studies? Maybe very clean now but we consumers don't know. Money, money to be made.

It's amusing how intricate to simulate the "real world" these studies say they are. But entice me to install Brand X on my machine--not happening. However, for many, the pretty graphs and pie charts are way more "scientific" and therefore credible.

 

[danb]

What was the infection vector? A browser exploit (so rare these days)? If it was the case than VD could be useful, but if he clicked on photos.jpg.ps1 or cracked GTA V.exe how something like VD could help?

Anti executable is just an annoyance for the average home user, most infections are via fake keygens, cracks and torrents in general and the user is always willing to disable the protection to run it, thats why default-deny doesnt work in the domestic scenario but can be a very effective policy in enterprise usage (the user cant bypass it).

Anti executable, HIPS, default-deny only works in a home scenario usage if the user cant bypass or disable it.

It's great that we are all taking Robbie's advice and letting users decide for themselves what security software works best for them .

VS is not a simple antiexecutable. It will detect keygens, cracks and torrents with 3 different file insights.