AV-Comparatives Heuristic / Behaviour Test 2014

[Ink]

Why other vendors are not included...

"This test is currently an optional part of our public main test-series, that is to say, manufacturers can decide at the beginning of the year whether they want their respective products to be included in the test."

And why it's posted now, in July...

"The report is delivered several months later due to the large amount of work required, deeper analysis, preparation and dynamic execution of the retrospective test-set."

 

[Deleted member 178]

all those test labs are eyes-powder...

the best test is to put an AV on your system for a month or two and then see how it performs.

 

[WinXPert]

yup. I use samples that are prevalent in my locale and tests them on different AVs. At least mine are objective because I'm not getting paid to make one perform better than the other

 

[Tony Cole]

Are you all saying what I think your saying - the AV industry is corrupt

 

[Deleted member 178]

Are you all saying what I think your saying - the AV industry is corrupt

business is corruption in essence

 

[WinXPert]

and looks like some are corrupting the AV business with rigged testings

 

[Wax]

Doesn't Emsisoft use the Bitdefender engine? Why is there such a massive difference in detection?

 

[Deleted member 21043]

Doesn't Emsisoft use the Bitdefender engine? Why is there such a massive difference in detection?

Wow, your right as well. Emsisoft use Bitdefender and their own engine, so they should have more or less the SAME detection results. That may be evidence these results aren't very reliable/accurate.

 

[BoraMurdar]

Doesn't Emsisoft use the Bitdefender engine? Why is there such a massive difference in detection?

2 reasons minimum :
Bitdefender never gives his latest engine to other vendors.
Bitdefender maybe relies on his behavior blocker to detect malicious running process and shut it off automatically. Emsisoft difference is about asking the user what to do.
And yet, if those met the requirements, it still doesn't mean that this test is 100% accurate or true in the first place...

 

[Paul.R]

Bora, Emsisoft i think as the latest BD engine.

 

[Kate_L]

AV-Comparatives, AV-Test ... are untrusted to me. If anyone wanna see what is real detection, should see the scan results from malware packs in the Virus Exchange

 

[Petrovic]

 

[Fabian Wosar]

Bitdefender never gives his latest engine to other vendors.

As someone who has access to the Bitdefender SDKs I can assure you that is 100% false. Differences exist mostly due to the fact that vendors may choose to use a different mix of technologies. We for example only use the actual Bitdefender engine and signatures. On top of that we use our own behavior blocker, our own additional scan engine and our own URL blocker.

The way you choose to present results to the user also have an impact on these tests. Whenever you leave the decision about a detection to the user, you get only half the points, even if the dialog urged the user to Block and Quarantine it. That is one of the reasons why we changed the behavior of Emsisoft Anti-Malware in version 9.0 where it makes a lot more decisions on its own instead of asking the user for confirmation. Once AV-C starts testing version 9.0 the large yellow part of our graph should become significantly smaller.

Last but not least you can always just have bad luck. We missed 39 samples for example. 30 of those 39 samples belonged to the same malware family (Caphaw). We already had an update for our behavior blocker out at the time of the test to cover that family properly, but the test was done with an installation that was frozen on March 7th when the update wasn't available yet. But that's just life. ##### happens .

 

[FleischmannTV]

I wonder why everyone is so hostile against AV-Comparatives but easily puts his trust in YouTube testers of all people, who can't even differentiate between a legitimate official Chinese / Japanese installer of Baidu Antivirus and actual malware, or can't tell a toolbar from a rootkit, just because HitmanPro said so.

Then of course you can look at how products fare against malware packs, but then you have to ask yourself who verified the contents of these packs? Is it real malware, potentially unwanted programs or even legitimate software that ended up by mistake in the pack. What if a vendor watches the forum, downloads the pack and blindly classifies all of its content as Generic.Malware or Trojan.Generic (even .txt files)? Is their product then better than others?

I wholeheartedly agree that AV-Comparatives and consorts are flawed but they are still the best resources of comparative testing. YouTube tests on the other hand are refined satire and serve no other purpose than getting a glance at the products' gui and toast notifications, and the occasional laugh of course. If anything they are an unintentionally funny manifestation of slander.

 

[Deleted member 21043]

I wonder why everyone is so hostile against AV-Comparatives but easily puts his trust in YouTube testers of all people, who can't even differentiate between a legitimate official Chinese / Japanese installer of Baidu Antivirus and actual malware, or can't tell a toolbar from a rootkit, just because HitmanPro said so.

Then of course you can look at how products fare against malware packs, but then you have to ask yourself who verified the contents of these packs? Is it real malware, potentially unwanted programs or even legitimate software that ended up by mistake in the pack. What if a vendor watches the forum, downloads the pack and blindly classifies all of its content as Generic.Malware or Trojan.Generic (even .txt files)? Is their product then better than others?

I wholeheartedly agree that AV-Comparatives and consorts are flawed but they are still the best resources of comparative testing. YouTube tests on the other hand are refined satire and serve no other purpose than getting a glance at the products' gui and toast notifications, and the occasional laugh of course. If anything they are an unintentionally funny manifestation of slander.

I think members on these forums, including Malware1 analyse the files. I can't say if they do, though.

 

[Malware1]

An interesting question is, who's providing the samples? The batches must be tailor made to make a select few perform impressively and others do a what we call "epic fail"

The samples are provided by AV companies. That's why these tests are useless.

@kram7750
that depends if I have enough time

 

[Manzai]

Bitdefender n°1

 

[marg]

Samples are provided by AV companies? Oh my! Lions Tigers & Bears are on the loose.

 

[Kate_L]

Samples are provided by AV companies? Oh my! Lions Tigers & Bears are on the loose.

That sounds like "fair" detection rate

 

[Deleted member 21043]

Bitdefender n°1

That's what they say, but when I had Emsisoft Internet Security v9 beta installed, it took out everything I threw at it. I was in a virtual machine with it installed aimlessly throwing random infected/malicious websites from various sources: the samples in virus exchange here on MalwareTips, Malc0de, etc. I also tested avast! and Bitdefender. I use avast! on my system *not on Virtual Machine* because it seems really light on my system and I never feel a system slowdown like I sometimes do with Emsisoft or Bitdefender, but Bitdefender didn't catch as many. avast! caught more or less the same amount as Bitdefender but Emsisoft caught a few more.