Do you trust AV-testing labs?

  • Yes, but i take with a grain of salt

    Votes: 52 66.7%
  • Yes, i choose AV depending on those results

    Votes: 4 5.1%
  • No, i certainly do not trust them

    Votes: 22 28.2%
  • Total voters
    78

RoboMan

Level 27
Content Creator
Verified
In the past few days, i have personally contacted popular security companies, faking to be a worried user of their product, claiming their software performed low on independent test labs. This was intended to force techinal support give their opinion on these kind of labs: should a regular user trust them or not? Lets see what they had to say, and sorry for taking a bit of your time technicians!

The actual used text on most cases was the following:

"I'm worried about the continued use of your product since recent tests i've seen online, mostly from independent test labs like AV-Comparatives or AV-Tests. Some of them had your software score low. Should i be worried about it?"

When required, fake name and telephone numbers were used.


---------------------------------------------

BITDEFENDER



Pretty confident about their product, BitDefender wouldn't even trust such tests who flagged their product as mediocre even existed

---------------------------------------------

BULLGUARD




Dafi brought up to the table a pretty interesting point of view

---------------------------------------------

EMSISOFT




On Emsisoft, David claimed testing method is often implemented on a wrong way, and highlighted their product is always flagged as one of the best worldwide. (sorry Umbra!)

---------------------------------------------

ESET




Marcos, from ESET, went straight to the point and suggested to simply not take a look onto these tests.

---------------------------------------------

F-SECURE





At F-Secure recommended to not rely on these tests as well.

---------------------------------------------

KASPERSKY





Jonathan at Kaspersky claimed AV-testing labs are not a valid opinion and have no basis.

---------------------------------------------

MCAFEE






McAfee representative, with its old fashioned chat window style (needs an update!) was extremely sure there was nothing to worry since McAfee offers the best protection ever possible (confident!)

---------------------------------------------

VIPRE



Support team at Vipre claims their product is top class and we should only worry about tests that say they are great lol

---------------------------------------------

All other not mentioned but popular software is not here because either they declined my request for not being an active paid user or never answered to my support ticket.

There were some interesting responses. What do you think?
 
D

Deleted member 65228

It would be in the best interest for any vendor to claim they don't agree with the testing by a vendor when someone asks them about low performance/bad results, for business obviously. Whether they were all truthful or not is another story. And it can also be in the interest to agree they are reliable if they know they performed well

It isn't unknown that the tests should all be taken with a grain of salt, no matter who is behind the testing phase. New malware is released each day of each month of each year and it is impossible for any vendor to keep up with every single one in reality - they all have their own intelligence and/or may share some with other vendors as we already know from SDK licensing and usage.

IMO The people behind companies like VirusBulletin don't sit in their chairs playing Candy Crush all day but instead try to test the products in a professional manner I'd imagine. Some vendors do well, others may not... The next day, the vendor that performed worst could potentially perform best. At the end of the day whether someone thinks they test properly or not is all opinionated.

This was interesting to read thanks for sharing! How long did you spend doing this btw? The waiting times must have been painful :)

I remember once I wanted to speak to Norton on Live Chat and had to wait 30m
 

In2an3_PpG

Level 17
Content Creator
Verified
One thing i learned is that if i need support from Bullguard, I'm going to ask for Dafi. :)

Dafi in my opinion gave you the best answer.

McAfee and Vipre. Ha what a joke. We will protect you don't worry. Were the best cant you see by the test. Nothing to worry about. :LOL:

David from Emsisoft's answer was pretty interesting about them disabling parts of the application.
 

RoboMan

Level 27
Content Creator
Verified
This was interesting to read thanks for sharing! How long did you spend doing this btw? The waiting times must have been painful :)

I remember once I wanted to speak to Norton on Live Chat and had to wait 30m
For most chats, i had to wait around 3-10 minutes to get my request accepted, and with the waiting time between responses I spent around 15-20 minutes with each vendor. As for e-mail support, VIPRE answered 3 days after the ticket was created, and Emsisoft literally answered 10 minutes after i submitted the ticket.
 

_CyberGhosT_

Level 53
Content Creator
Trusted
Verified
I voted no, because of the way that question blankets all "Do you trust AV testing Labs"
As a rule, no I don't, I used to frequent AV Comparitives, but even them not so much anymore.
Barring new entries or major updates the top 2 or 3 will remain there, I don't need to be told monthly
how effective my solution is, if their dependability changes, trust me the users will know. The best solution
will make mistakes we all know this. For me the solution lies in being active on the forums of the solution of
your choice, if there are errors or issues to address AV labs will be the last to know, and if you picked a good
solution and are on their forums their users will be notified and often the first to know ;)
EDIT: Minus -1 for the over use of the word "solution" :sick: lol
 
Last edited:
5

509322

In the past few days, i have personally contacted popular security companies, faking to be a worried user of their product, claiming their software performed low on independent test labs. This was intended to force techinal support give their opinion on these kind of labs: should a regular user trust them or not? Lets see what they had to say, and sorry for taking a bit of your time technicians!

The actual used text on most cases was the following:

"I'm worried about the continued use of your product since recent tests i've seen online, mostly from independent test labs like AV-Comparatives or AV-Tests. Some of them had your software score low. Should i be worried about it?"

When required, fake name and telephone numbers were used.


---------------------------------------------

BITDEFENDER



Pretty confident about their product, BitDefender wouldn't even trust such tests who flagged their product as mediocre even existed

---------------------------------------------

BULLGUARD




Dafi brought up to the table a pretty interesting point of view

---------------------------------------------

EMSISOFT




On Emsisoft, David claimed testing method is often implemented on a wrong way, and highlighted their product is always flagged as one of the best worldwide. (sorry Umbra!)

---------------------------------------------

ESET




Marcos, from ESET, went straight to the point and suggested to simply not take a look onto these tests.

---------------------------------------------

F-SECURE





At F-Secure recommended to not rely on these tests as well.

---------------------------------------------

KASPERSKY





Jonathan at Kaspersky claimed AV-testing labs are not a valid opinion and have no basis.

---------------------------------------------

MCAFEE






McAfee representative, with its old fashioned chat window style (needs an update!) was extremely sure there was nothing to worry since McAfee offers the best protection ever possible (confident!)

---------------------------------------------

VIPRE



Support team at Vipre claims their product is top class and we should only worry about tests that say they are great lol

---------------------------------------------

All other not mentioned but popular software is not here because either they declined my request for not being an active paid user or never answered to my support ticket.

There were some interesting responses. What do you think?
You were communicating mostly with low level technical support staff that almost certainly know little about AV test methodology, AV testing labs, and the actual test reports that you are referencing. In other words, they probably are not qualified to provide an answer - and definitely not going to address any specific reports.

David Biggar from Emsisoft gave the best advice - long term AV test lab result trends.
 
Last edited by a moderator:

insanity

Level 5
I find interesting that while the companies dismissed the validity of AV labs tests when allegedly confronted with poor scores, they are often the same that boast about how their product performed well in some test, usually writing a blog post or showing the AV lab certificate on their sale website.
 
Last edited:

klaken

Level 3
Well I use the comparisons to get an idea. nothing more.

It seems bad to me that companies say do not believe in those shares my software is the best (especially mcafee where e seen low performance).

Here more companies were missing.
I wonder what he would say comodo. sure they say that this test is of detection since their software is based on avoiding infections ... Personally I do not see that they are very worried in the detection although they have improved a lot ..
 
D

Deleted member 65228

I wonder what he would say comodo. sure they say that this test is of detection since their software is based on avoiding infections ... Personally I do not see that they are very worried in the detection although they have improved a lot ..
They have the automatic sandbox. I doubt they literally care about anything else. I once saw on the comodo forum in 2016 a lot of people complaining about something, and all the responses I saw were the same "Use the sandbox", etc. Some dude called Melih, pretty sure he's a CEO of comodo
 

Azure

Level 23
Content Creator
Verified
While it does seem you went to a lot of trouble to do this. I hope you don't get in trouble with them for posting this.

For example, in ESET's case, you agreed to their 'full term and conditions'.


One of the things I'm more interested in those test is a product consistency. If a product is able to maintain a high score throughout not just one but many organizations, then it could be worth using.
 

Slyguy

Level 40
You were communicating mostly with low level technical support staff that almost certainly know little about AV test methodology, AV testing labs, and the actual test reports that you are referencing. In other words, they probably are not qualified to provide an answer - and definitely not going to address any specific reports.

David Biggar from Emsisoft gave the best advice - long term AV test lab result trends.
I was wondering when someone would mention the low level tech thing. These aren't even techs, most likely helpdesk. Except in the case of Dave Biggar of course.

Anyway, I don't even read AV tests or quote them, and it would be very rare for me to even mention them. Most of them are either total rubbish, or should be 'lightly' used in deciding on a product. Unfortunately some people take them for the gospel and make all of their decisions based on these days. Long term results are more valid as Dave says.. I had a friend drop ESET because it was 'scoring lower' in the last few tests according to him.. <yawn>

Do what I do. Setup a fresh laptop, non-VM, put it on a DMZ and 'let it go'.. Use the auto-clicking/auto-browsing tool I use that hits 1.5 million clicks a week. Then check on the machine every weekend and see how it is going. Maybe get creative, let an 8 year old use it, download random trash, surf risky sites.. Spice it up.. Eventually you'll figure out the strongest products, and they generally AREN'T the top scoring ones on those fake AV evaluations and staged tests. :unsure:
 

HarborFront

Level 45
Content Creator
Verified
I remember seeing tests done here by MT members and their results published for the AV products....but I don't seem to see them for quite some time now

Such tests by MT members can be used as a comparison against those by AV labs to either concur or dispute their results

Anyone knows what happen to these MT testings?

:rolleyes:
 
Last edited:
5

509322

I was wondering when someone would mention the low level tech thing. These aren't even techs, most likely helpdesk. Except in the case of Dave Biggar of course.

Anyway, I don't even read AV tests or quote them, and it would be very rare for me to even mention them. Most of them are either total rubbish, or should be 'lightly' used in deciding on a product. Unfortunately some people take them for the gospel and make all of their decisions based on these days. Long term results are more valid as Dave says.. I had a friend drop ESET because it was 'scoring lower' in the last few tests according to him.. <yawn>

Do what I do. Setup a fresh laptop, non-VM, put it on a DMZ and 'let it go'.. Use the auto-clicking/auto-browsing tool I use that hits 1.5 million clicks a week. Then check on the machine every weekend and see how it is going. Maybe get creative, let an 8 year old use it, download random trash, surf risky sites.. Spice it up.. Eventually you'll figure out the strongest products, and they generally AREN'T the top scoring ones on those fake AV evaluations and staged tests. :unsure:
It perplexes me that so many people actually put so much weight upon AV-Comparatives and AV-Test results. At least AVLab.pl and MRG Effitas knock the softs about in ways to give better indications (some will debate that) of what a user can expect when faced with real-world upon execution nasties.

Entry point technical support is never going to respond appropriately to queries about AV test lab results. At least Emsisoft had the sense to pass the question straight to the Technical Support Manager, who then answered the questions.
 
Last edited by a moderator:

Nightwalker

Level 14
Content Creator
Verified
Please take no offence, but what you did is a BIG waste of time (yours and from AV company's staff).
These guys dont make the decisions, they dont make the marketing material and they dont do the R&D, so it is obvious that they need to give a politely and professionally answer without too deep in it and I dont know if it is legal to post these answers out there.

In most AV companies you almost will never talk or actually know the guys that make the R&D of those products (Emsisoft is a exception), so I dont know what you were trying to accomplish there.