Should you trust Test-labs?

Do you trust Test-labs?

  • Yes

    Votes: 3 10.3%
  • No

    Votes: 20 69.0%
  • Other (post in thread)

    Votes: 6 20.7%

  • Total voters
    29
D

Deleted member 178

Thread author
(This is my personal opinion, i am not in the business so i may be wrong; feel free to think different.)

Lately lot of users (especially on this forum) strongly rely on AV test labs (like AV-Comparative, AV-Test, Matousec, etc...) to choose their security solution, that should be a good behavior but the problem is that they mislead you. why?

Basically, Testing organizations use malware samples found in the wild, install the product in default setting in its own system, then froze the signature update by disabling the internet connection, then launch the malwares against the product to test its proactivity & detection following a distinct procedure.

what you -users- must know:

A- non-paranoid view

1- Product Features:

Each products use different ways to protect you:

- Real-time scanner Aka the "engine" using signature and heuristics, unfortunately for test labs , each vendors has its own delay before adding the latest malwares signatures, so when they froze the updates, a vendor may not add yet some sigs so it will failed the test. For this point it is quite normal, but what you should know is that some vendors uses only the recent known most propagated malwares, if a lab decide to use old samples (retired from the database because no more considered as a threat) or those active in some regions only, the product may fail.

- Proactive features aka Behavior Blocker/HIPS/Web Filter/Cloud, etc...
again, some products don't use any of them and are still compared with the ones that have it...major failure, you can't compare uncomparable things. products should be tested separately depending their features.

- Settings, each products is set out of the box with what we use to call "default settings" , most labs test them with default settings; it is again a failure since some products are set almost at maximum by default when others don't. To be meaningful test-labs should set every tested product with minimal or maximal setting.

Testing Labs need to be able to configure multiple products to provide comparable protection in each of them to offer legitimate results. Since no products offer the same/similar levels of protection, comparing them is quite useless.

2- Malwares samples

tests-labs use hundreds of malwares in a row, real world users will never encounter that sums of malwares in their whole life, and no products is supposed to be targeted with that sums of malwares.

B- Paranoid View

Of course, i have no proof of what i will write below (even if we know some of this facts), but it does not mean it is not possible :D

1- Business

Security is a big business that generate lot of money, either for cyber-criminals than for security vendors, since Labs' results will influence potential customers for the best rated products, how -us, simple users- could we trust those labs, even if they said they are "independent" (words are just words). Money is money and corruption is everywhere...even some labs are known to give pre-results to some vendors so they can update their products in exchange for some "contributions" then the official result are publicly released if it satisfies the vendor...


Final Note:

I will just give this advice, don't rely "only" on Testing Organizations, take them as "one piece of information"; try a product for 2-3 weeks then decide if you like it enough to buy a license.



Thanks
 

McLovin

Level 76
Verified
Honorary Member
Malware Hunter
Apr 17, 2011
9,222
Everything you see on the internet when it comes to test you should "take with a grain of salt". Not everything is 100% correct.
 
  • Like
Reactions: lobo001

Littlebits

Retired Staff
May 3, 2011
3,893
I trust no test-labs, none of them will not paint a complete picture on how any product will protect a single user. The best example of this is all of the test-labs gives Microsoft Security Essentials/ Windows Defender on Windows 8 a very bad score. But I have several customers that never get infection with MSE which proves it does an excellent job protecting them. We all know that there isn't any AV that will detect everything, the user doesn't learn how to safely download files and browse the web, they will get infected no matter what AV they use.

VirusBulletin is the best of testing labs because they only use malware samples which have been reported to have infections in the wild.

Others will use malware samples which are rare and sometimes have no reported infections at all. Most users will never be exposed to most of the malware samples so who cares if your AV doesn't detect it?

Even VirusBulletin has its faults as well which keep it from completely reliable.

Test-labs are no way to choose what is the best AV for yourself, even the AV's with the highest detection rate can miss simple malware that the other AV's with lower detection rate can detect.

I look at testing-labs as commercials for AV products, nothing more.

Thanks.:D
 
  • Like
Reactions: lobo001

tapoo

Level 4
Verified
Jan 21, 2012
639
i just checks their reviews for just curiosity, just to know how they performing in their test, nothing else
 

Coffeeman

Level 3
Verified
Nov 24, 2012
202
IMO this test results should be viewed with a little skepticism because these days some are paid by some to bring their products in the front. I am not saying that these are not somehow reliable but they do not really reflect real world use of some antimalware products.
 

Valentin N

Level 2
Feb 25, 2011
1,314
the best is to ask, watch some review see what tests labs says and most importantly to test it yourself and build your own opinion.
 

Littlebits

Retired Staff
May 3, 2011
3,893
Watch out for the video reviews that purposely download and try to execute malware samples from various sources, they are no better than test-labs. If you are looking for samples that will go undetected they are not hard to find. Most end users don't purposely download malware because these kind of reviews are irrelevant since no AV will detect everything.

Thanks.:D
 

Spirit

Level 2
May 17, 2012
1,832
I check only one website and trust them too and that is av comparitivies.

I check their result for av detection rate as well av performance test.

The detection is not only the point when I look after av,I check av features,user friendly,lighness and prevention capabilities.

Thanks
 

Valentin N

Level 2
Feb 25, 2011
1,314
Littlebits said:
Watch out for the video reviews that purposely download and try to execute malware samples from various sources, they are no better than test-labs. If you are looking for samples that will go undetected they are not hard to find. Most end users don't purposely download malware because these kind of reviews are irrelevant since no AV will detect everything.

Thanks.:D

testing malware is one part of testing a product. other is to see if you like how it interacts with you and so on.

watching reviews that for instance languy99 does, give insight how the product is confronting a malware and what you can expect from the certain product, so reviews can be useful.

How do you build your opinion about an AV or IS?
 
D

Deleted member 178

Thread author
Valentin N said:
How do you build your opinion about an AV or IS?

for me:

1- Features, by this i meant must have all the principle layers covered.
2- Efficiency, all features must be useful and efficient to a certain degree
3- Lightness on system.
4- GUI, must be ergonomic, all options easy to be accessed , an ugly and non-ergonomic GUI, will push me away from it.
 
P

Plexx

Thread author
Littlebits said:
Watch out for the video reviews that purposely download and try to execute malware samples from various sources, they are no better than test-labs. If you are looking for samples that will go undetected they are not hard to find. Most end users don't purposely download malware because these kind of reviews are irrelevant since no AV will detect everything.

Thanks.:D

I will have to disagree with you on this one.

Video reviews do give a general idea to the user who is watching how a product performs.

Such purpose allows users who do not have the capabilities or facilities to do their own testing, in order for them to draw their own conclusions and up to date, I do not recall any test pointing a gun at the user saying: You should use this vendor's solution.

Saying that these reviews are irrelevant is in a way lack of respect for the users who contribute their time to show the user how a solution performs, whether you agree or not with the core mechanics and infections etc.
 
I

illumination

Thread author
Biozfear said:
Littlebits said:
Watch out for the video reviews that purposely download and try to execute malware samples from various sources, they are no better than test-labs. If you are looking for samples that will go undetected they are not hard to find. Most end users don't purposely download malware because these kind of reviews are irrelevant since no AV will detect everything.

Thanks.:D

I will have to disagree with you on this one.

Video reviews do give a general idea to the user who is watching how a product performs.

Such purpose allows users who do not have the capabilities or facilities to do their own testing, in order for them to draw their own conclusions and up to date, I do not recall any test pointing a gun at the user saying: You should use this vendor's solution.

Saying that these reviews are irrelevant is in a way lack of respect for the users who contribute their time to show the user how a solution performs, whether you agree or not with the core mechanics and infections etc.

I have to agree with you on this, if someone is looking for a quick glimpse of how a product works, a tour through the settings,ect these videos provide such answers, especially for those with out the knowledge to test or capabilities to do so.
 
N

Nige_40

Thread author
I don't trust them eva, They can make a bigger problem, than there ever is.
 

Littlebits

Retired Staff
May 3, 2011
3,893
Biozfear said:
Littlebits said:
Watch out for the video reviews that purposely download and try to execute malware samples from various sources, they are no better than test-labs. If you are looking for samples that will go undetected they are not hard to find. Most end users don't purposely download malware because these kind of reviews are irrelevant since no AV will detect everything.

Thanks.:D

I will have to disagree with you on this one.

Video reviews do give a general idea to the user who is watching how a product performs.

Such purpose allows users who do not have the capabilities or facilities to do their own testing, in order for them to draw their own conclusions and up to date, I do not recall any test pointing a gun at the user saying: You should use this vendor's solution.

Saying that these reviews are irrelevant is in a way lack of respect for the users who contribute their time to show the user how a solution performs, whether you agree or not with the core mechanics and infections etc.

Unfortunately video reviews are the interpretation of the reviewer, the end user may get a totally different experience. You can show someone how to ride a bike, but until they ride that bike on their own, they will not learn how to ride it. Products perform differently on different systems, each user will have their own interpretation of each product based on their own experience with it. Majority of video reviews actually mislead users instead of helping them. I don't say this in a disrespectful way, it is simply the truth. Anyone with a video recorder can make a video review whether they have any qualifications or not. Most reviewers are amateurs with no professional qualifications to speak of.

As a user watching these video reviews how are we to know for sure if the reviewer has any qualifications or isn't bias towards certain products?

Without knowing these answers, video reviews are basically just entertainment. You can't depend on them to base any facts.

Thanks.:D
 

McLovin

Level 76
Verified
Honorary Member
Malware Hunter
Apr 17, 2011
9,222
Littlebits said:
Unfortunately video reviews are the interpretation of the reviewer, the end user may get a totally different experience. You can show someone how to ride a bike, but until they ride that bike on their own, they will not learn how to ride it. Products perform differently on different systems, each user will have their own interpretation of each product based on their own experience with it. Majority of video reviews actually mislead users instead of helping them. I don't say this in a disrespectful way, it is simply the truth. Anyone with a video recorder can make a video review whether they have any qualifications or not. Most reviewers are amateurs with no professional qualifications to speak of.

As a user watching these video reviews how are we to know for sure if the reviewer has any qualifications or isn't bias towards certain products?

Without knowing these answers, video reviews are basically just entertainment. You can't depend on them to base any facts.

Thanks.:D

Also gives people when watching these reviews an idea on how it performs, and if let's say Avast performed great against 10 malicious links and MSE didn't you see the user going to go and try out Avast first before MSE. So once the user has done a fair trial of Avast themselves, they can then say if they like it or not. Also it goes if a person has been using a product for such a long time, i.e me I've been using Trend for years and you see people here on this forum ask about it and see if it's worth trying, and some say it is and some say it's not once they've tried.

Personally video reviews can be good and bad, the way they do them can be good like, Languy, Matt, and so on, but you get other reviews that review every single program update, which the world doesn't need every single update that comes out.

Just my two cents worth too. :)
 

Exterminator

Community Manager
Verified
Staff Member
Well-known
Oct 23, 2012
12,527
I never take any of the test lab results into consideration when choosing to try an AV or Security Suite product.I dont watch video reviews either.After all I dont know the reviewer so as Littlebits stated,how can I trust their review as an unbiased fair interpretation.I will watch some of the video tests on here.Getting to know some of the members on here I feel I can trust their input but in the end I trust myself and how it performs on my system.
 
D

Deleted member 178

Thread author
i know for example that Biozfear or McLovin videos are neutral , just showing pros & cons of a products, but i know too that some fanboys do the videos of their loved product with especially chosen samples which they will be sure their AV will detect and to be more realistic they put one or two malwares the Av will miss.

Also some reviewers doesn't know how the product really behave, Webroot SA is the best example, many claims it is poor and protect nothing, without understanding how it works.
 

McLovin

Level 76
Verified
Honorary Member
Malware Hunter
Apr 17, 2011
9,222
Umbra Corp. said:
i know for example that Biozfear or McLovin videos are neutral , just showing pros & cons of a products, but i know too that some fanboys do the videos of their loved product with especially chosen samples which they will be sure their AV will detect and to be more realistic they put one or two malwares the Av will miss.

Also some reviewers doesn't know how the product really behave, Webroot SA is the best example, many claims it is poor and protect nothing without understanding how it works.

Thanks. ;) Exactly what I mean, I know that I'm some what of a fan boy for Trend Micro, but I keep an open mind when it comes to the tests that it does and to see what other ones are out there that are better than Trend.
 

Exterminator

Community Manager
Verified
Staff Member
Well-known
Oct 23, 2012
12,527
Umbra Corp. said:
i know for example that Biozfear or McLovin videos are neutral , just showing pros & cons of a products


That is an example of 2 of the members here I can trust

Edit:OOooooppps....I mean an example of 3....+Umbra
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top