AV‐Comparatives, AV‐TEST and Virus Bulletin comment on allegations of inappropriate behavior

Status
Not open for further replies.

BoraMurdar

Community Manager
Thread author
Verified
Staff Member
Well-known
Aug 30, 2012
6,598
Today, three of the world’s most renowned and trusted security testing bodies, AV‐Comparatives, AV‐ TEST and Virus Bulletin, stand united to censure security vendor Qihoo 360 after finding the firm submitted products for comparative and certification testing which behaved significantly differently from those made available to its users and customers. The three testing bodies will revoke all certifications and rankings awarded to the company’s products so far this year, and going forward will insist on more open and fair dealings to ensure users are provided with the most accurate information possible. Investigations by the three labs found that all products submitted for testing by Qihoo had one of the product’s four available engines, provided by Bitdefender, enabled by default, while a second, Qihoo’s own QVM engine, was never enabled. This included versions posted to ostensibly public sections of the company’s websites. By contrast, as far as can be determined, all versions made generally available to users in Qihoo’s main market regions had the Bitdefender engine disabled and the QVM engine active. According to all test data this would provide a considerably lower level of protection and a higher likelihood of false positives. Options are provided in the product to adjust these settings, but as the majority of users leave settings unchanged, most tests insist on using the default product settings to best represent real‐world usage. As part of the investigation into Qihoo 360, counter‐accusations were levelled by the company against two fellow Chinese security firms, Baidu and Tencent. Analysis of products submitted for testing by these companies turned up some unexpected flags within their products, marked with the names of several test labs and implying some difference in product behavior depending on the environment they were run in – similar flags were also found in Qihoo products. However, no evidence could be found that this gave any significant advantage to either product, and in some cases it even seemed to put them at a disadvantage. Both firms were able to provide good reasons for including these flags in their products. On requesting an explanation from Qihoo 360 for their actions, the firm confirmed that some settings had been adjusted for testing, including enabling detection of types of files such as keygens and cracked software, and directing cloud lookups to servers located closer to the test labs. After several requests for specific information on the use of third‐party engines, it was eventually confirmed that the engine configuration submitted for testing differed from that available by default to users.

Read more here
http://www.av-test.org/fileadmin/pdf/VB-AVC-AVT-press-release.pdf
 
D

Deleted member 21043

I think everyone knew it was Qihoo before AV-C/AV-TEST/Virus Bulletin told us... I mean, I highly doubt a trusted vendor like ESET, Emsisoft or BitDefender would even consider thinking about submitting changed versions of the product to the testing labs.
 

Enju

Level 9
Verified
Well-known
Jul 16, 2014
443
I think everyone knew it was Qihoo before AV-C/AV-TEST/Virus Bulletin told us... I mean, I highly doubt a trusted vendor like ESET, Emsisoft or BitDefender would even consider thinking about submitting changed versions of the product to the testing labs.
It's kinda sad that people are still trusting a company like this. Not only are they lying, but they are disabling their own "engine" and are just using Bitdefender instead. o_O
 

Atlas147

Level 30
Verified
Honorary Member
Top Poster
Content Creator
Well-known
Jul 28, 2014
1,990
this is something very interesting, in the past there were a few members who tested the bitdefender engine on the 360 TS and showed that they have a lower protection than the actual bitdefender product itself. Some suggested that this might be because qihoo doesn't include full signatures in their product. But with this accusation I think they just might have the full signatures just that they don't release it to the public, instead they use it for their tests.

I think one of the main reason is because using QVM engine would allow for cloud analysis, giving them some data to sell to other companies.
 

BoraMurdar

Community Manager
Thread author
Verified
Staff Member
Well-known
Aug 30, 2012
6,598
this is something very interesting, in the past there were a few members who tested the bitdefender engine on the 360 TS and showed that they have a lower protection than the actual bitdefender product itself. Some suggested that this might be because qihoo doesn't include full signatures in their product. But with this accusation I think they just might have the full signatures just that they don't release it to the public, instead they use it for their tests.

I think one of the main reason is because using QVM engine would allow for cloud analysis, giving them some data to sell to other companies.
I read somewhere that Bitdefender doesn't provide their latest engine to every partner. Also there is a probability of using their engine with complete signatures, without complete signatures and latest, non-latest combinations. I will not mention additional protection layers found in Bitdefender's Antivirus. Anyway, I was always telling people to use Antivirus engine from original vendor, as those hybrid mumbo-jumbo is always suspicious.
More doesn't always mean better.
 

FreddyFreeloader

Level 32
Verified
Top Poster
Well-known
Jul 23, 2013
2,115
No surprise these "labs" were caught totally unaware what has been going on. This also gives some credence to those of you who have been saying, "it's all about the money." I never trusted these "labs" and if you look at the results of our own testers at MalwareHub, you'll note they are quite different than the results the "labs" post. A pox on all them.
 

woomera

Level 7
Verified
Jan 15, 2012
594
lol when the first day this news came out i thought the word "modified" means something else!
according to the finding they only changed the default settings right? so whats the big fuss is about? dont get it...
anyone that is reading this forum and come to places like this on the web never uses a software on default setting or ... am i the only one... omg the abyss...
dont kid yourself. this is a free AV with no ad/popup and uses 2 of the most powerful AV engines on the market available to users! for free i say JAMES! :throatcleanin:

this doesnt mean that what they did was right, but for those of us here... it changes nothing. anytime i install this for someone i immediately enable BD and Avira engine and go through all the settings.
 
R

Ramona

I don't care about this, I will still use 360 Total Security. They have the best detection, yes, they have false positive, overall it is a good product. No, I am not a fangirl :mad:
 
D

Deleted member 21043

lol when the first day this news came out i thought the word "modified" means something else!
according to the finding they only changed the default settings right? so whats the big fuss is about? dont get it...
anyone that is reading this forum and come to places like this on the web never uses a software on default setting or ... am i the only one... omg the abyss...
dont kid yourself. this is a free AV with no ad/popup and uses 2 of the most powerful AV engines on the market available to users! for free i say JAMES! :throatcleanin:

this doesnt mean that what they did was right, but for those of us here... it changes nothing. anytime i install this for someone i immediately enable BD and Avira engine and go through all the settings.
I understand what you say to the full extent, however the real point here is that Qihoo did something they were not supposed to do and therefore have been "cheating".

ESET don't change the HIPS to Smart Mode to get extra detections for the copy the lab testers receive, do they?

The fact that Qihoo tweaked the settings for a better detection rate is not only misleading to the average user since they may have it on default settings for less False Positive detections, but also unfair on all the other trustworthy vendors who coply with the rules.

If they had have not tweaked the product and had have complied with the rules, then maybe another fair vendor would have received a better rating than them.

It's only fair to the companies who test the product (who have been misleaded by the company in terms of trust) to take the actions they have/will take out due to this incident.

Whether it's a good product or not, they cheated. Cheating is bad and unethical.
 

FireShootSK

Level 17
Verified
Feb 17, 2015
824
Facebook post by AV-TEST (Last update: 20 min ago) https://www.facebook.com/avtestorg/posts/952718894780899

*** UPDATE ***

Unfortunately we have to post an update to our current findings.
So far we checked the possible manipulation of our protection tests.
This is what is being reflected in the posted statement.

We have now started to evaluate the possible manipulation of our performance testing.
We have found strong evidence that another company, not Qihoo, is optimizing their product to do well in our
performance test by excluding certain files and processes from checking.
This is based on filenames and process names and can pose a security risk as well!
We will check with AV-Comparatives and VB100 to verify our findings and will let you know as soon as we have the final data.

*** UPDATE ***
 

aztony

Level 9
Verified
Oct 15, 2013
501
Unfortunately we have to post an update to our current findings. So far we checked the possible manipulation of our protection tests. This is what is being reflected in the posted statement.

We have now started to evaluate the possible manipulation of our performance testing. We have found strong evidence that another company, not Qihoo, is optimizing their product to do well in our performance test by excluding certain files and processes from checking. This is based on filenames and process names and can pose a security risk as well! We will check with AV-Comparatives and VB100 to verify our findings and will let you know as soon as we have the final data.
https://www.facebook.com/avtestorg
 
Status
Not open for further replies.

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top