AV-Comparatives Heuristic / Behaviour Test 2014

[Fabian Wosar]

I wonder why everyone is so hostile against AV-Comparatives

Because vendors pay for these tests, which is an instant deal breaker for a lot of people. What most people forget to mention though is, that every vendor pays the same amount. Given that AV-C is a non-profit organization, they may also be obliged to provide publicly accessible financial reports. But I am not that familiar with Austrian NPO laws to be honest .

but easily puts his trust in YouTube testers of all people, who can't even differentiate between a legitimate official Chinese / Japanese installer of Baidu Antivirus and actual malware, or can't tell a toolbar from a rootkit, just because HitmanPro said so.

I have to admit, Youtube reviews are a guilty pleasure of mine. Sure, they contain a ##### ton of crap. But I just like watching them to see how products react. That being said, I do have some gripes. The biggest issue is that there is no feedback loop. The test VMs are more often than not instantly reset. So if I ask one of those reviewers about some information to figure out what may have went wrong on their system they can't give me anything. They don't keep the samples they downloaded and executed around either. With some luck I can copy the URL they used to download the sample from the video and pray the sample hasn't changed yet, but that does not help when the sample came from a sample pack. When you manage to get some information, you are often met with a huge amount of hostility. I am not sure why that is. Maybe people are just bitter due to constant abuse in the Youtube comments. Personally I would be excited if a vendor of a product I just made a review of contacted me to ask for my help. But a lot of reviewers don't share that sentiment. They just want to move on as quickly as possible. This is especially frustrating if it later turns out that they screwed up their review, making false claims in their recap.

 

[WinXPert]

I think members on these forums, including Malware1 analyse the files. I can't say if they do, though.

I do if I have the time and if the sample is a new one that is prevalent in my locale. Most of these are malwares written by IT students near our place

and I don't just analyze them, I also provide manual removal instructions posted on one of our local forums. Been doing it since 2009

 

[Deleted member 21043]

I do if I have the time and if the sample is a new one that is prevalent in my locale. Most of these are malwares written by IT students near our place

The IT students could probably learn enough to start learning how to make a Antivirus scanner (get file directories, the MD5/SHA1 hash from the file, compare it to a database) *that at the least, instead of making malware. Would that not be better?

The above was a basic example ^^

 

[BoraMurdar]

As someone who has access to the Bitdefender SDKs I can assure you that is 100% false. Differences exist mostly due to the fact that vendors may choose to use a different mix of technologies. We for example only use the actual Bitdefender engine and signatures. On top of that we use our own behavior blocker, our own additional scan engine and our own URL blocker.

The way you choose to present results to the user also have an impact on these tests. Whenever you leave the decision about a detection to the user, you get only half the points, even if the dialog urged the user to Block and Quarantine it. That is one of the reasons why we changed the behavior of Emsisoft Anti-Malware in version 9.0 where it makes a lot more decisions on its own instead of asking the user for confirmation. Once AV-C starts testing version 9.0 the large yellow part of our graph should become significantly smaller.

Last but not least you can always just have bad luck. We missed 39 samples for example. 30 of those 39 samples belonged to the same malware family (Caphaw). We already had an update for our behavior blocker out at the time of the test to cover that family properly, but the test was done with an installation that was frozen on March 7th when the update wasn't available yet. But that's just life. ##### happens .

Last time I checked Bitdefender was selling his SDK of its Antivirus from 2011 version. Never newer. But I will need to check it again, maybe they changed their motto

There are a lot of independent AV-Testing companies but whoever pays to win in these tests we can just speculate and guess that it actually happens. No one here actually visited them. One is for sure, no one here can conduct even approximate accuracy, "time-stop" systems, large collection of malware samples and malware diversity like they do in AV-Comparatives, AV-Test and others...

 

[Deleted member 21043]

Last time I checked Bitdefender was selling his SDK of its Antivirus from 2011 version. Never newer. But I will need to check it again, maybe they change their motto

There are a lot of independent AV-Testing companies but whoever pays to win in these tests we can just speculate and guess that it actually happens. No one here actually visited them. One is for sure, no one here can conduct even approximate accuracy, "time-stop" systems, large collection of malware samples and malware diversity like they do in AV-Comparatives, AV-Test and others...

Yes, but even if they are selling a older version you could contact them and try to make a deal for their newer one. If the price is right, they'll accept...

As someone who has access to the Bitdefender SDKs I can assure you that is 100% false. Differences exist mostly due to the fact that vendors may choose to use a different mix of technologies. We for example only use the actual Bitdefender engine and signatures. On top of that we use our own behavior blocker, our own additional scan engine and our own URL blocker.

The way you choose to present results to the user also have an impact on these tests. Whenever you leave the decision about a detection to the user, you get only half the points, even if the dialog urged the user to Block and Quarantine it. That is one of the reasons why we changed the behavior of Emsisoft Anti-Malware in version 9.0 where it makes a lot more decisions on its own instead of asking the user for confirmation. Once AV-C starts testing version 9.0 the large yellow part of our graph should become significantly smaller.

Last but not least you can always just have bad luck. We missed 39 samples for example. 30 of those 39 samples belonged to the same malware family (Caphaw). We already had an update for our behavior blocker out at the time of the test to cover that family properly, but the test was done with an installation that was frozen on March 7th when the update wasn't available yet. But that's just life. ##### happens .

Considering you are someone with access to the Bitdefender SDK, are you able to tell me if you pay monthly/annually or if it's a one-time fee for it?

 

[BoraMurdar]

Yes, but even if they are selling a older version you could contact them and try to make a deal for their newer one. If the price is right, they'll accept...

If you make a cure for cancer would you sell the prescription to other company? Or if you don't have any money you will probably sell yourself to some pharmaceutic company.
There is no logic of making an engine and sell it to everyone.
I have posted a war room topic about "what differs one bitdefender based product from another one" and its not just additional tools like url filter, behavior blocker, intrusion prevention system, kernel block and others... Engine itself differs. (well it was, I'll need to make some phone calls )

 

[Fabian Wosar]

Considering you are someone with access to the Bitdefender SDK, are you able to tell me if you pay monthly/annually or if it's a one-time fee for it?

NDA, sorry.

Engine itself differs. (well it was, I'll need to make some phone calls )

Engine contains of bdcore.dll as well as the signature files. Feel free to compare those across BD partners and BD. They will be the same with some delay caused by server syncs.

 

[Nightwalker]

I just need to say that Fabian Wosar is a awesome guy, it is so nice to have someone like him here

 

[Deleted member 21043]

I just need to say that Fabian Wosar is a awesome guy, it is so nice to have someone like him here

Earlier when I saw his name on the forums, I thought it was some guy messing with us and having a joke, pretending to be him. Then I thought and realized it really is him, so now I'm pretty excited and amazed he's here on MalwareTips.

 

[kev777]

thank you fabian, for being on MalwareTips

 

[marg]

Yes... Thanks Fabian!

 

[CapeBuffalo]

The countries best known for hacking are the ones with the "best" antivirus companies (romania and russia)
i prefer real world test or test done here which tells another story though rather than close sourced reviews as i believe bias and money does alot of talking (not offence to fair testers)
e.g. IGN a reputable or the largest most well known gaming review site gives some questionable good score to some crappy big franchise games e.g. call of duty ghost

 

[Deleted member 21043]

The countries best known for hacking are the ones with the "best" antivirus companies (romania and russia)
i prefer real world test or test done here which tells another story though rather than close sourced reviews as i believe bias and money does alot of talking (not offence to fair testers)
e.g. IGN a reputable or the largest most well known gaming review site gives some questionable good score to some crappy big franchise games e.g. call of duty ghost

You can't exactly call Call of Duty: Ghosts a crap game when a good majority of gamers play it, especially on PS3/4 and Xbox 360/One. In fact, I would say Call of Duty is the most heard game title I hear people mention (in real life, that is). And, if you look on YouTube, a majority of the recorders do Call of Duty gaming.

Anyway, it's going a bit off topic now so we should get things back to the topic and end the discussion about IGN game reviewing and if call of duty ghosts is a crap game or not.

 

[Littlebits]

I just wished Microsoft would drop out of AV testing and destroy their baseline.
More vendors are discovering the flaws of AV testing that is why so many have already dropped out.

Enjoy!!

 

[Fabian Wosar]

I just wished Microsoft would drop out of AV testing and destroy their baseline.

I am pretty sure Microsoft doesn't participate officially. They are just included because that is what the OS ships with.