Antivirus companies opinions on testing labs

Do you trust AV-testing labs?

  • Yes, but i take with a grain of salt

    Votes: 54 67.5%
  • Yes, i choose AV depending on those results

    Votes: 4 5.0%
  • No, i certainly do not trust them

    Votes: 22 27.5%

  • Total voters
    80

RoboMan

Level 34
Thread author
Verified
Top Poster
Content Creator
Well-known
Jun 24, 2016
2,399
In the past few days, i have personally contacted popular security companies, faking to be a worried user of their product, claiming their software performed low on independent test labs. This was intended to force techinal support give their opinion on these kind of labs: should a regular user trust them or not? Lets see what they had to say, and sorry for taking a bit of your time technicians!

The actual used text on most cases was the following:

"I'm worried about the continued use of your product since recent tests i've seen online, mostly from independent test labs like AV-Comparatives or AV-Tests. Some of them had your software score low. Should i be worried about it?"

When required, fake name and telephone numbers were used.


---------------------------------------------

BITDEFENDER

PrV61zE.png


Pretty confident about their product, BitDefender wouldn't even trust such tests who flagged their product as mediocre even existed

---------------------------------------------

BULLGUARD


jAQ46BC.png


Dafi brought up to the table a pretty interesting point of view

---------------------------------------------

EMSISOFT


yF3LGwn.png


On Emsisoft, David claimed testing method is often implemented on a wrong way, and highlighted their product is always flagged as one of the best worldwide. (sorry Umbra!)

---------------------------------------------

ESET


OeAymN1.png


Marcos, from ESET, went straight to the point and suggested to simply not take a look onto these tests.

---------------------------------------------

F-SECURE



Gu0R8ZC.png


At F-Secure recommended to not rely on these tests as well.

---------------------------------------------

KASPERSKY



MAeBK58.png


Jonathan at Kaspersky claimed AV-testing labs are not a valid opinion and have no basis.

---------------------------------------------

MCAFEE


EjEyuqt.png


wRxWQHz.png


McAfee representative, with its old fashioned chat window style (needs an update!) was extremely sure there was nothing to worry since McAfee offers the best protection ever possible (confident!)

---------------------------------------------

VIPRE

Zh6ydsp.png


Support team at Vipre claims their product is top class and we should only worry about tests that say they are great lol

---------------------------------------------

All other not mentioned but popular software is not here because either they declined my request for not being an active paid user or never answered to my support ticket.

There were some interesting responses. What do you think?
 
D

Deleted member 65228

It would be in the best interest for any vendor to claim they don't agree with the testing by a vendor when someone asks them about low performance/bad results, for business obviously. Whether they were all truthful or not is another story. And it can also be in the interest to agree they are reliable if they know they performed well

It isn't unknown that the tests should all be taken with a grain of salt, no matter who is behind the testing phase. New malware is released each day of each month of each year and it is impossible for any vendor to keep up with every single one in reality - they all have their own intelligence and/or may share some with other vendors as we already know from SDK licensing and usage.

IMO The people behind companies like VirusBulletin don't sit in their chairs playing Candy Crush all day but instead try to test the products in a professional manner I'd imagine. Some vendors do well, others may not... The next day, the vendor that performed worst could potentially perform best. At the end of the day whether someone thinks they test properly or not is all opinionated.

This was interesting to read thanks for sharing! How long did you spend doing this btw? The waiting times must have been painful :)

I remember once I wanted to speak to Norton on Live Chat and had to wait 30m
 

In2an3_PpG

Level 18
Verified
Top Poster
Content Creator
Well-known
Nov 15, 2016
867
One thing i learned is that if i need support from Bullguard, I'm going to ask for Dafi. :)

Dafi in my opinion gave you the best answer.

McAfee and Vipre. Ha what a joke. We will protect you don't worry. Were the best cant you see by the test. Nothing to worry about. :LOL:

David from Emsisoft's answer was pretty interesting about them disabling parts of the application.
 

RoboMan

Level 34
Thread author
Verified
Top Poster
Content Creator
Well-known
Jun 24, 2016
2,399
This was interesting to read thanks for sharing! How long did you spend doing this btw? The waiting times must have been painful :)

I remember once I wanted to speak to Norton on Live Chat and had to wait 30m

For most chats, i had to wait around 3-10 minutes to get my request accepted, and with the waiting time between responses I spent around 15-20 minutes with each vendor. As for e-mail support, VIPRE answered 3 days after the ticket was created, and Emsisoft literally answered 10 minutes after i submitted the ticket.
 

_CyberGhosT_

Level 53
Verified
Honorary Member
Top Poster
Content Creator
Well-known
Aug 2, 2015
4,286
I voted no, because of the way that question blankets all "Do you trust AV testing Labs"
As a rule, no I don't, I used to frequent AV Comparitives, but even them not so much anymore.
Barring new entries or major updates the top 2 or 3 will remain there, I don't need to be told monthly
how effective my solution is, if their dependability changes, trust me the users will know. The best solution
will make mistakes we all know this. For me the solution lies in being active on the forums of the solution of
your choice, if there are errors or issues to address AV labs will be the last to know, and if you picked a good
solution and are on their forums their users will be notified and often the first to know ;)
EDIT: Minus -1 for the over use of the word "solution" :sick: lol
 
Last edited:
5

509322

In the past few days, i have personally contacted popular security companies, faking to be a worried user of their product, claiming their software performed low on independent test labs. This was intended to force techinal support give their opinion on these kind of labs: should a regular user trust them or not? Lets see what they had to say, and sorry for taking a bit of your time technicians!

The actual used text on most cases was the following:

"I'm worried about the continued use of your product since recent tests i've seen online, mostly from independent test labs like AV-Comparatives or AV-Tests. Some of them had your software score low. Should i be worried about it?"

When required, fake name and telephone numbers were used.


---------------------------------------------

BITDEFENDER

PrV61zE.png


Pretty confident about their product, BitDefender wouldn't even trust such tests who flagged their product as mediocre even existed

---------------------------------------------

BULLGUARD


jAQ46BC.png


Dafi brought up to the table a pretty interesting point of view

---------------------------------------------

EMSISOFT


yF3LGwn.png


On Emsisoft, David claimed testing method is often implemented on a wrong way, and highlighted their product is always flagged as one of the best worldwide. (sorry Umbra!)

---------------------------------------------

ESET


OeAymN1.png


Marcos, from ESET, went straight to the point and suggested to simply not take a look onto these tests.

---------------------------------------------

F-SECURE



Gu0R8ZC.png


At F-Secure recommended to not rely on these tests as well.

---------------------------------------------

KASPERSKY



MAeBK58.png


Jonathan at Kaspersky claimed AV-testing labs are not a valid opinion and have no basis.

---------------------------------------------

MCAFEE


EjEyuqt.png


wRxWQHz.png


McAfee representative, with its old fashioned chat window style (needs an update!) was extremely sure there was nothing to worry since McAfee offers the best protection ever possible (confident!)

---------------------------------------------

VIPRE

Zh6ydsp.png


Support team at Vipre claims their product is top class and we should only worry about tests that say they are great lol

---------------------------------------------

All other not mentioned but popular software is not here because either they declined my request for not being an active paid user or never answered to my support ticket.

There were some interesting responses. What do you think?

You were communicating mostly with low level technical support staff that almost certainly know little about AV test methodology, AV testing labs, and the actual test reports that you are referencing. In other words, they probably are not qualified to provide an answer - and definitely not going to address any specific reports.

David Biggar from Emsisoft gave the best advice - long term AV test lab result trends.
 
Last edited by a moderator:

insanity

Level 5
Verified
Oct 9, 2016
216
I find interesting that while the companies dismissed the validity of AV labs tests when allegedly confronted with poor scores, they are often the same that boast about how their product performed well in some test, usually writing a blog post or showing the AV lab certificate on their sale website.
 
Last edited:

klaken

Level 3
Verified
Well-known
Oct 11, 2014
112
Well I use the comparisons to get an idea. nothing more.

It seems bad to me that companies say do not believe in those shares my software is the best (especially mcafee where e seen low performance).

Here more companies were missing.
I wonder what he would say comodo. sure they say that this test is of detection since their software is based on avoiding infections ... Personally I do not see that they are very worried in the detection although they have improved a lot ..
 

SHvFl

Level 35
Verified
Honorary Member
Top Poster
Content Creator
Well-known
Nov 19, 2014
2,342
They mean almost nothing to me because in practise the few % change will not make a difference for my usage. Maybe people who are risk should be more worried but then again nothing will save them anw.
 
D

Deleted member 65228

I wonder what he would say comodo. sure they say that this test is of detection since their software is based on avoiding infections ... Personally I do not see that they are very worried in the detection although they have improved a lot ..
They have the automatic sandbox. I doubt they literally care about anything else. I once saw on the comodo forum in 2016 a lot of people complaining about something, and all the responses I saw were the same "Use the sandbox", etc. Some dude called Melih, pretty sure he's a CEO of comodo
 

Azure

Level 28
Verified
Top Poster
Content Creator
Oct 23, 2014
1,712
While it does seem you went to a lot of trouble to do this. I hope you don't get in trouble with them for posting this.

For example, in ESET's case, you agreed to their 'full term and conditions'.


One of the things I'm more interested in those test is a product consistency. If a product is able to maintain a high score throughout not just one but many organizations, then it could be worth using.
 
F

ForgottenSeer 58943

You were communicating mostly with low level technical support staff that almost certainly know little about AV test methodology, AV testing labs, and the actual test reports that you are referencing. In other words, they probably are not qualified to provide an answer - and definitely not going to address any specific reports.

David Biggar from Emsisoft gave the best advice - long term AV test lab result trends.

I was wondering when someone would mention the low level tech thing. These aren't even techs, most likely helpdesk. Except in the case of Dave Biggar of course.

Anyway, I don't even read AV tests or quote them, and it would be very rare for me to even mention them. Most of them are either total rubbish, or should be 'lightly' used in deciding on a product. Unfortunately some people take them for the gospel and make all of their decisions based on these days. Long term results are more valid as Dave says.. I had a friend drop ESET because it was 'scoring lower' in the last few tests according to him.. <yawn>

Do what I do. Setup a fresh laptop, non-VM, put it on a DMZ and 'let it go'.. Use the auto-clicking/auto-browsing tool I use that hits 1.5 million clicks a week. Then check on the machine every weekend and see how it is going. Maybe get creative, let an 8 year old use it, download random trash, surf risky sites.. Spice it up.. Eventually you'll figure out the strongest products, and they generally AREN'T the top scoring ones on those fake AV evaluations and staged tests. :unsure:
 

HarborFront

Level 71
Verified
Top Poster
Content Creator
Oct 9, 2016
6,014
I remember seeing tests done here by MT members and their results published for the AV products....but I don't seem to see them for quite some time now

Such tests by MT members can be used as a comparison against those by AV labs to either concur or dispute their results

Anyone knows what happen to these MT testings?

:rolleyes:
 
Last edited:
5

509322

I was wondering when someone would mention the low level tech thing. These aren't even techs, most likely helpdesk. Except in the case of Dave Biggar of course.

Anyway, I don't even read AV tests or quote them, and it would be very rare for me to even mention them. Most of them are either total rubbish, or should be 'lightly' used in deciding on a product. Unfortunately some people take them for the gospel and make all of their decisions based on these days. Long term results are more valid as Dave says.. I had a friend drop ESET because it was 'scoring lower' in the last few tests according to him.. <yawn>

Do what I do. Setup a fresh laptop, non-VM, put it on a DMZ and 'let it go'.. Use the auto-clicking/auto-browsing tool I use that hits 1.5 million clicks a week. Then check on the machine every weekend and see how it is going. Maybe get creative, let an 8 year old use it, download random trash, surf risky sites.. Spice it up.. Eventually you'll figure out the strongest products, and they generally AREN'T the top scoring ones on those fake AV evaluations and staged tests. :unsure:

It perplexes me that so many people actually put so much weight upon AV-Comparatives and AV-Test results. At least AVLab.pl and MRG Effitas knock the softs about in ways to give better indications (some will debate that) of what a user can expect when faced with real-world upon execution nasties.

Entry point technical support is never going to respond appropriately to queries about AV test lab results. At least Emsisoft had the sense to pass the question straight to the Technical Support Manager, who then answered the questions.
 
Last edited by a moderator:

Nightwalker

Level 24
Verified
Honorary Member
Top Poster
Content Creator
Well-known
May 26, 2014
1,339
Please take no offence, but what you did is a BIG waste of time (yours and from AV company's staff).
These guys dont make the decisions, they dont make the marketing material and they dont do the R&D, so it is obvious that they need to give a politely and professionally answer without too deep in it and I dont know if it is legal to post these answers out there.

In most AV companies you almost will never talk or actually know the guys that make the R&D of those products (Emsisoft is a exception), so I dont know what you were trying to accomplish there.
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top