Advice Request Why I think testing "labs" are useless

[James246]

It came in through Browser a number of years ago but his PC had extra protection of the type that Joe Public would not have available to them, and yes he could not easily disable it.
The point I make is that a Virus Checker even ones that score 100% in tests will not stop everything in the real world

 

[Moonhorse]

You probably could write an entire book of this, how enterprise & home works differently , about protection modules those antivirus solutions do have, about the user, his family, his workingmates , his customers and their habits, the list will never end

On topic: you really summed this well together, well done

 

[Nightwalker]

What is easier to respond to, a UAC prompt or a VS prompt? What is SAFER to respond to, a UAC prompt or a VS prompt? 95%+ of our users are average Joes, and we have a lot of users. I get about 4-5 password reset / account support emails a day, and I have had less than 5 in eight years ask about how to use VS.

When the common denominator is WD, a second layer of some kind is a good idea. In other words, malware authors are going to bypass WD first, since it is the common denominator.

UAC is not a security boundary and default-deny isnt a security solution, it is a security policy, but VD is more than a simple antiexecutable and it deserve its credits.

What I dont agree is the necessity to use something beyond Windows 10 default security, after all the infections in a domestic scenario usage is almost always because of the user disabling or ignoring security solutions warnings and thats something that no security solution can help with it.

 

[bayasdev]

Those days when home users can easily get their PCs infested of PUPs and trojans are gone, Ransomware can be bypassed by having a proper auto-backup strategy nor cloud sync. Now cybercriminals are looking forward to targeted attacks, that don't necessarily need to disrupt the business activity, instead they prefer to remain silent, get sensitive information, steal trade secrets, intellectual property, and propietary source code, making the company to lose market advantage and harm their reputation by giving this confidential data to their competitors, the malware actors can also blackmail the company to not leak this data asking huge sums of money.

 

[Nightwalker]

It came in through Browser a number of years ago but his PC had extra protection of the type that Joe Public would not have available to them, and yes he could not easily disable it.
The point I make is that a Virus Checker even ones that score 100% in tests will not stop everything in the real world

Nowadays you really have to be "very lucky" to be infected via a browser exploit in a Windows 10 up-to-date system , those things now are very expensive and reserved to high value targets.

 

[danb]

UAC is not a security boundary and default-deny isnt a security solution, it is a security policy, but VD is more than a simple antiexecutable and it deserve its credits.

What I dont agree is the necessity to use something beyond Windows 10 default security, after all the infections in a domestic scenario usage is almost always because of the user disabling or ignoring security solutions warnings and thats something that no security solution can help with it.

I agree that UAC is not a security mechanism. I was simply illustrating which one is easier for the average user to handle and understand.

You have to admit, it is quite funny that the same people who compare and contrast home vs enterprise are the same people who harden their systems against enterprise attacks, ie. SMB, RDP, etc. Why does this logic not apply to system hardening?

We are probably way off topic, so maybe we could start another thread if you guys want to discuss this further.

 

[RoboMan]

I love these respectful and serious discussions!

I want to give my insight about WD since many comments are about it. I have to disagree with they common say that WD is trash, since it's not. It's a FREE ANTIVIRUS and has as many modules as a paid suite. Of course, it has a long way to become a solid, "almost bullet proof" software, but so far it's the BEST free alternative. I dare you list at least 3 free antiviruses than include the modules WD offers (PUP protection, Controlled Access Folder, anti tamper, ransomware protection, encrypted files recovery via OneDrive, anti-exploit, vulnerability protection, parental control). It has a suite level of protection, although we can agree it still needs some work to polish its protection; not meaning it's not a good software! IMO the best free alternative in the market.

 

[blackice]

I love these respectful and serious discussions!

I want to give my insight about WD since many comments are about it. I have to disagree with they common say that WD is trash, since it's not. It's a FREE ANTIVIRUS and has as many modules as a paid suite. Of course, it has a long way to become a solid, "almost bullet proof" software, but so far it's the BEST free alternative. I dare you list at least 3 free antiviruses than include the modules WD offers (PUP protection, Controlled Access Folder, anti tamper, ransomware protection, encrypted files recovery via OneDrive, anti-exploit, vulnerability protection, parental control). It has a suite level of protection, although we can agree it still needs some work to polish its protection; not meaning it's not a good software! IMO the best free alternative in the market.

If they would make Controlled Folder Access more user friendly then it would be hands down the best option for most anyone.

 

[blackice]

I think that the testing labs and TPSC are most useful for identifying only what is completely ineffective. There are some security software that we all know just aren’t worth the effort or money because they can’t even identify 50% of known malware in big lab tests. Those are not what you want as a last line of defense. It can also give an indication of a solution is going downhill. That’s about it.

Also, pretty much anyone here on MT knows at least a little about security and is probably fine with an adblocker and WD. Most of it is debate for fun. Only once in the last year did I hear of a member here getting malware, and it sounded like a particularly nasty drive by.

 

[danb]

I love these respectful and serious discussions!

I want to give my insight about WD since many comments are about it. I have to disagree with they common say that WD is trash, since it's not. It's a FREE ANTIVIRUS and has as many modules as a paid suite. Of course, it has a long way to become a solid, "almost bullet proof" software, but so far it's the BEST free alternative. I dare you list at least 3 free antiviruses than include the modules WD offers (PUP protection, Controlled Access Folder, anti tamper, ransomware protection, encrypted files recovery via OneDrive, anti-exploit, vulnerability protection, parental control). It has a suite level of protection, although we can agree it still needs some work to polish its protection; not meaning it's not a good software! IMO the best free alternative in the market.

I love respectful and serious discussions as well. I just do not want to get into trouble by going off topic .

Years ago WD was not all that great, but now it is a solid contender. In fact, WD plus one other tiny layer of protection is really all you need to be safe. But relying on WD alone is dangerous since that is the first layer that all malware authors must penetrate.

I have actually been running CFA, and it was worked out great for me. I have had a couple of unnecessary blocks, but they were all things I did not use anyway, so I just left them blocked.

 

[nikcave]

you're making a really relevant point and you're right here. looked through the thread and understood that I've never paid enough attention to this issue .___.

 

[Arequire]

I'd go further than useless, I'd label them as unconsciously deceptive. Not because I think the labs are corrupt or something just as ridiculously conspiratorial, but because they lack transparency in regards to their testing parameters, and don't adequately communicate that whatever results are achieved by the participating products, said results are only relevant within those exact parameters.
That last point is why I label them deceptive; a user lacking the appropriate knowledge is likely to interpret the results as gospel both within and outside of the testing parameters, and unknowingly gain a false sense of security in doing so.

anyone who knows how to use tools like VoodooShield, Sandboxie and HIPS like software doesnt need them at first place.

I suppose I'd fall into this category as VS is the only protection I use.
It's not that I think my system would suddenly fall to infection if I just used a traditional antivirus, it's that I find default-allow to be an idiotic way of handling security.

 

[struppigel]

Regardless of how meaningful those test results are; testing labs themselves are useful. They force improvements in protection via competition. Without those tests, a user would only be left with marketing. The products with best marketing would win, protection wouldn't count at all to sell stuff. No one would be driven to improve protection too much.

But: Regardless how much effort testing labs put into it, they cannot re-create reality. It's not ethically and practically possible to throw entirely new malware against AVs. Malware has to come from some source. Creating new malware is not ethically right. The labs cannot be perfect and can only test a part of what really counts for user protection. AV companies are indirectly forced to tailor their products so that they compete well in the testing lab. Meaning: only things that are tested, are improved upon, whereas other things that also help for protection may not be as good as they could be. Or if they are improved, they will not be reflected in the test results, thus, not create revenue. Furthermore, if most AVs are similarly good, the labs still need to engineer their tests in a way that makes them distinguishable. So they may indeed put focus on minor differences or not that important stuff in order to determine the winners and loosers.

I have no good solution for this. It's just the way it is. Things aren't perfect and we can only make the best of it.

 

[Vitali Ortzi]

Regardless of how meaningful those test results are; testing labs themselves are useful. They force improvements in protection via competition. Without those tests, a user would only be left with marketing. The products with best marketing would win, protection wouldn't count at all to sell stuff. No one would be driven to improve protection too much.

But: Regardless how much effort testing labs put into it, they cannot re-create reality. It's not ethically and practically possible to throw entirely new malware against AVs. Malware has to come from some source. Creating new malware is not ethically right. The labs cannot be perfect and can only test a part of what really counts for user protection. AV companies are indirectly forced to tailor their products so that they compete well in the testing lab. Meaning: only things that are tested, are improved upon, whereas other things that also help for protection may not be as good as they could be. Or if they are improved, they will not be reflected in the test results, thus, not create revenue. Furthermore, if most AVs are similarly good, the labs still need to engineer their tests in a way that makes them distinguishable. So they may indeed put focus on minor differences or not that important stuff in order to determine the winners and loosers.

I have no good solution for this. It's just the way it is. Things aren't perfect and we can only make the best of it.

So true

 

[Vitali Ortzi]

Off topic any more recent exploit test
Comparable to this https://www.mrg-effitas.com/wp-content/uploads/2018/05/MRG_Exploit_Protection.pdf ?

 

[Nightwalker]

I suppose I'd fall into this category as VS is the only protection I use.
It's not that I think my system would suddenly fall to infection if I just used a traditional antivirus, it's that I find default-allow to be an idiotic way of handling security.

That is exactly what I think default-deny is for me as a home user, idiotic and annoying; I dont need a tool to babysit me and stop what I want to run in my PC.

If I want and need to run "suspicious file that has some potential utility.exe" what I want is a good antivirus with a great behavior blocker to save me if the file is not what I was looking for.

Default-deny is much cheaper and lower resource usage friendly than antivirus solutions, but antivirus can make a definitive diagnostic in files and thats why the market for the latter is strong and alive while we dont have standalone HIPS anymore and solutions like VS has a small niche market.

 

[Digmor Crusher]

Sophos did really well in that test Vitali, read below to see why, taken from page 4 of test:

Another reason to put little faith in these tests other than for entertainment purposes only.

 

[Arequire]

That is exactly what I think default-deny is for me as a home user, idiotic and annoying; I dont need a tool to babysit me and stop what I want to run in my PC.

If I want and need to run "suspicious file that has some potential utility.exe" what I want is a good antivirus with a great behavior blocker to save me if the file is not what I was looking for.

Default-deny is much cheaper and lower resource usage friendly than antivirus solutions, but antivirus can make a definitive diagnostic in files and thats why the market for the latter is strong and alive while we dont have standalone HIPS anymore and solutions like VS has a small niche market.

I just dislike the idea that anything that isn't definitively identified as malicious is allowed to run unmitigated on my system. Sure there's behaviour blockers, but from all the testing I've seen I consider them unreliable at best. Missing even a single sample after it executes could cause a serious headache depending on what the malware actually does, and I have absolutely zero faith in behaviour blockers to work as intended.

I'm not sure about your system, but one thing that may be different between us is that my system is almost wholly static. Besides updates, I haven't installed a single thing since February, and the last prompt I received was back in early March. So I completely understand why it'd be annoying to use default-deny if you're installing new applications often, but I very rarely even notice it's there the majority of the time.

 

[MacDefender]

My thoughts on the matter of user protection:

Scenario	Solution
User downloads trustworthy apps from the Windows Store or digitally signed software, respects SmartScreen and Google Safe Browsing, and does not change default settings to execute scripts/macros embedded in documents or PDFs.	Windows Defender is perfectly fine and probably won't even trigger. It's extremely unlikely to even encounter malware if you do this.
User is slightly off the beaten path, sometimes downloads software or documents from places not vetted	Windows Defender or most lab tested AVs are probably fine. Most likely the malware encountered this way are not zero-days and have been incorporated in most AVs for detection
User partakes heavily in piracy, greyware, hacktools, or other sort of habits that routinely exposes them to malware	High end AV suite with excellent zero day signatures or layered approach (e.g. BitDefender's suite, Kaspersky, F-Secure, ESET, etc). WD is likely also acceptable but in my opinion it is not AS good as some of the options above at detecting trojans in this context of greyware.
User as part of his job HAS to execute code or macro documents that are delivered by potentially untrusted parties (for example, if you are a reviewer for an app publishing house, or you're a data scientist who exchanges scripts and other packages, or your business involves exchanging Microsoft Office documents with legit macros)	High end AV suite with sophisticated behavior blocking (Kaspersky, Emsisoft, F-Secure, etc) or some sort of HIPS system. Even with all of this you still may be at some risk.
User is expected to be directly targeted by custom-tailored attacks, is intentionally executing malware for the purpose of analysis, etc, and does many of the things mentioned in the previous row above.	Good luck. No out of the box tool is going to provide meaningful protection here. User better use most of the techniques that Malware Hub testers use to contain/isolate their environment, plus host based infection detection, plus something at the network level to monitor for indicators-of-compromise, etc.
User literally gets an AV lab's 10,000 samples, and clicks them one by one on their computer and does not want to be infected.	Well for you, AV tests are perfectly accurate, only choose something with 100% test scores. But why would you do that?

If you fall into the first two categories, which I consider "the average user", it is really unlikely that you even encounter malware in the first place, much less something that defeats Windows Defender and SmartScreen reputation.

It's primarily when the user intentionally has to defeat some of those layers of security to do what they do, that they would benefit from looking elsewhere for improved protection.

With that said, IMO improved protection is NOT the only reason to look at a third party AV suite. Some of them provide improved visibility into the system (like Kaspersky's Application Activity / Network Monitor), or provide pseudo sandboxing tools like Application Control that can be used as a sandboxing tool, etc, so it could be soft features that lead to selecting a different tool.

I do agree with @Robbie's original point though that something with 93% protection isn't necessarily worse than something with 99% protection at protecting you. It's all about what the 6% difference is, and what is the chance that you would end up encountering something like that?


I'm on a computer 14 hours a day and honestly I have never encountered malware without actively going out and looking for trouble.

 

[Tutman]

This is my opinion, and the fact I'm dropping it here, in the general security discussion section, is with hope we can establish a serious discussion about this subject; about why I think testing labs are useless.

When, in real-life, average users WILL NEVER EVER face 20,000 threats in one day, and Windows Defender will still have a 100% protection rate against the 2 or 3 possible malware they face online each month.

This is true BUT on the other hand, you feel better knowing that in the wild you may only get 2-3 possible viruses but also knowing your AV product can withstand against 20,000 in a lab test!