Why are Independant lab results unreliable?

[jackuars]

I'm not interested in claims that they grab money for publishing results, but what could be the other reasons that they are unreliable in testing the overall effectiveness of an antivirus software? Can you give me some examples?

Also in what ways are User Reviews at MalwareTips better or worse than the ones' published at Independent Labs?

Time and again this debate comes up in our forums so let's put and end to that?

 

[conceptualclarity]

There's some definite variation in the labs' results, but time and again they show Kaspersky and Bitdefender on top and Norton and ESET pretty close. It's hard for me to see how this really consistent outcome can't have a real basis in truth.

Sorry I'm not qualified to say much beyond that, so I step aside for others.

 

[CMLew]

I can only say this:-
One man's meat is another man's poison

Different people have different views over their reliability. The same goes to MT reviews.

Cheers!

 

[LabZero]

Personally, I think the comparison of the various tests performed by different laboratories, could be judged reliable if they were adhered to the following factors:

1-same malware database
2-same hardware configuration
3-same software configuration

If they were respected these conditions, it would make sense to compare the various tests among them, indicating parameters such as the scan time or data rate.
So, those are the first parameters because surely they are influenced by the diversity of hardware/software config.

Also, important thing for the signatures; information is not given to us about malware used in testing, and it can reasonably be expected that a test can be conducted on a different database with different results.

 

[hjlbx]

Variability in:

• malware sampling\collection methods
• actual malware samples used
• age of samples

has a lot to do with it.

When, where, and how the labs collect samples influences the test set.

Also, signature-based engine detection varies a bit over time - sometimes significantly.

Finally, hardware, operating system and version of AV does influence test results.

The overall trend over time is the best indicator of protection:

1. or 2. Kaspersky
1. or 2. Bitdefender
3. ...
4. ...
5. ...

3, 4, and 5 always seem to be Avira, ESET and Trend Micro.

 

[Rishi]

The reason is quite simple : The tests conducted use malware samples which basically represent a drop in an ocean full of malware.A malware collection used in one test may vary considerably with another lab's collection.Given the uniqueness of the security products, that fluctuation in performance reflects in the tests(for example certain AV may do better in rootkit detection while another surpasses others in worm detection).Also,the Av's user base/number of reported malware/quick update of database/number of false positives is another variable.

Then coming to the real world scenario where a user's hardware parameters(compatibility,old/new,OS updated or not,which softwares he uses),his browsing habits,disaster management(regular backup,virtualization)and human factors(social engineering attacks) all are considered vs world wide web which is the real time library of all kinds of malware and zero day exploits.This scenario best determines the comfort level with various security options.

The test results should be taken with a grain of salt.

 

[Kate_L]

The "real world" test is the best (here, the most accurate is; how often does a average user get infected with X product installed).
The malware tips reviews are ok most of the time.
Independent tests, well, I don't trust them because I test AVs daily and I get other results.

 

[Kiwimike]

The most accurate tests I've seen by independant labs is AV-Comparatives, who were recently qualified by Eicar.

But the reason some tests can be innacurate is because of the wide range of factors including:

• False positives - how many false positives does an antivirus get?
• detection of known malware - how many (decenty aged) malware samples get detected?
  
• Performance Impact - how much time does it take to open a program with or without the antivirus installed?
  
• Firewall - what common attack methods are blocked by the firewall on both public and private networks under stressful conditions?
• Behaviour/Hueristics - What can the antivirus block (without manual hips or sandbox) in terms of malware that it does not recongnize?

All of these things contribute to the total rating of an antivirus. I would also include personally anti-exploit tests, whether the firewall and behaviour system allows malware to update through a service started by task shceduler. Whether the antivirus is resiliant to attempts to shut the service down.

After all of these, in my eyes the best antivirus doesn't exist. Because of the simple fact every antivirus I have personally test cannot block exploits without signatures or behaviour analysis, can be shut down, can let malware update in the background, doesn't detect every threat through behaviour analysis and has a firewall barely better than Windows.

I understand that no antivirus is perfect, but having these severe issues is not good and I would consider every antivirus on the market a failure for these reasons. Zero day protection is a backup for incase the signature detection fails therefore it should be expected to live up to its expectation.

 

[jamescv7]

Simply the insufficient methods are raised for the test where errors are prone, traditional methods without experimentation of other concepts to make sure the product performs consistent or not and many others.

Sometimes that paperwork can be easily tampered without any basis + the reputation of a well known organization is bound for such allegations.

Nevertheless real world test that conducts by ordinary users are far better as we can see the overall strong points of then products.

Kaspersky, Avira, Bitdefender and few others may coincidentally consistent due to common test but who knows?

 

[RmG152]

Independent lab results are reliable, but aren't 100% trustable.
You need to take it with care and only as light reference.

Also in what ways are User Reviews at MalwareTips better or worse than the ones' published at Independent Labs?

Typical user video reviews are useless.

I only like @cruelsister cryptoseries, and some specific videos

 

[Kiwimike]

Simply the insufficient methods are raised for the test where errors are prone, traditional methods without experimentation of other concepts to make sure the product performs consistent or not and many others.

Sometimes that paperwork can be easily tampered without any basis + the reputation of a well known organization is bound for such allegations.

Nevertheless real world test that conducts by ordinary users are far better as we can see the overall strong points of then products.

Kaspersky, Avira, Bitdefender and few others may coincidentally consistent due to common test but who knows?

You can see the strong points of a product from an ordernary user test? This makes little sense to me considering they are using malware databases that have a high chance of being on most of the better antivirus' signatures.

While I could be wrong I also think that the malware used from a lot of databases are not as well designed for bypassing antivirus software. Antiviruses should NOT get tested against regular malware. Antivirus software is supposed to protect organisations, businesses, corperations, enterprises. This means antivirus software needs to protect users from targetted attacks and advanced zero day threats and APTs (advanced persistant threats).

This is why security professionals get paid so much, because antiviruses cannot handle the threats out there today.
antivirus software blocks about 20% of the malware out there, considering that the other 80% are more advanced threats. This means that antiviruses need to be tested against more advanced methods.

Antiviruses need to be tested against basic and advanced methods of bypassing and exploiting the system.

• Firewall's should be tested against remote exploitation attempts over both WAN and LAN.
  
• Antivirus software should survive, at a minimum a KILL5 test.
  
• Antivirus software needs to block a keylogger to be somewhat successful in my book
• Antivirus software defense systems MUST past at least 250 in CLT, the current case is most custom firewalls are equal or slightly more or less effective than Windows system itself.
  
• All firewalls Should have ARP poisoning blocked by default which is simply not the case for a lot of firewalls including Comodo's this ARP poisoning attack can cause serious issues with security, and blocking it would block a lot of packet sniffing attempts over LAN.

Firewalls need to block attacks, they are the FIRST line of defense. Which MUST be reliable and secure.

Independent lab results are reliable, but aren't 100% trustable.
You need to take it with care and only as light reference.



Typical user video reviews are useless.

I only like @cruelsister cryptoseries, and some specific videos

In the end, you should use what you trust.
Antiviruses are pretty much equal in terms of detection amoung the good ones, Kaspersky, Bitdefender, Eset, Avast, AVG etc.

I think that all antiviruses are okay, but they aren't.
We need to stop treating them like they are okay, they aren't! Antiviruses suck right now, they can't block everything they need to, and that is a massive drawback to your protection.

Esspecially Symantec, with such a big customer base. They NEED to improve security, same goes for Mcafee and Trend.

 

[Enju]

@Kiwimike

This is why security professionals get paid so much, because antiviruses cannot handle the threats out there today.
antivirus software blocks about 20% of the malware out there, considering that the other 80% are more advanced threats. This means that antiviruses need to be tested against more advanced methods.

Where did you get those statistics from?

Antiviruses need to be tested against basic and advanced methods of bypassing and exploiting the system.

Firewall's should be tested against remote exploitation attempts over both WAN and LAN.

How should they do that? They would need zero-day exploits for this kind of test and those go for quite a bit of money, more that they make every year with their tests.

Antivirus software should survive, at a minimum a KILL5 test.

Killtesting AV software is useless, if it's on the system the protection of your AV is the least of your worries.

Antivirus software needs to block a keylogger to be somewhat successful in my book

There are no absolutes in software, it can and will fail, so based on this every security software can and will fail.

Antivirus software defense systems MUST past at least 250 in CLT, the current case is most custom firewalls are equal or slightly more or less effective than Windows system itself.

Basing the security of a software on an impractical and outdated test?

All firewalls Should have ARP poisoning blocked by default which is simply not the case for a lot of firewalls including Comodo's this ARP poisoning attack can cause serious issues with security, and blocking it would block a lot of packet sniffing attempts over LAN.

Are we talking about home users here or businesses? ARP spoofing protection for home users is unnecessary.

Firewalls need to block attacks, they are the FIRST line of defense. Which MUST be reliable and secure.

Shameless self quote Best Firewall: Kaspersky IS 2016, BitDefender IS, or Eset Smart Security, or Emsisoft IS

I think that all antiviruses are okay, but they aren't.
We need to stop treating them like they are okay, they aren't! Antiviruses suck right now, they can't block everything they need to, and that is a massive drawback to your protection.

They never have and never will block everything, they are okay for what they are supposed to be doing ,blocking known and common stuff, but that's it.

Esspecially Symantec, with such a big customer base. They NEED to improve security, same goes for Mcafee and Trend.

Trashing three of the biggest security companies out there without any reason? Have you even tried their endpoint and/or server protection?

 

[FleischmannTV]

In my opinion it is a matter of ressources. The test labs have less financial ressources and personnel than the big AV companies, so why should they be faster in acquiring and verifying malicious samples than companies which make hundreds of millions - some even more than a billion - dollars each year. I doubt AV-Test, AV-Comparatives and the likes have an annual budget of that size available.

 

[illumination]

None of the security products are exactly the same, in the way the modules work and respond, so using the same method to test them all will be completely inaccurate is the bottom line of why these tests need to be taken with a grain of salt. The tests done in reviews here are done individually "mainly", by members that understand how the product works and will be effected.

 

[Deleted member 178]

You really want test AVs? go get some FUD encrypted zero-minutes exploiys/malwares from some hacker buddies; because as me, they laugh at those tests, i knew some guys that have some exploits that can disable most of the big names security softs...and if you got some kernel exploits everything is done...bye bye security apps.

I rather believes some black/grey-hats than those silly test-labs using thousands of outdated samples

 

[jackuars]

What about those malware's that can identify and bypass systems that are conducting the tests under a virtual machine? Will they defeat these tests?

 

[conceptualclarity]

• All firewalls Should have ARP poisoning blocked by default which is simply not the case for a lot of firewalls including Comodo's this ARP poisoning attack can cause serious issues with security, and blocking it would block a lot of packet sniffing attempts over LAN.

Firewalls need to block attacks, they are the FIRST line of defense. Which MUST be reliable and secure.

Interestingly, that's the first mention of ARP poisoning I've come across on MT. I was wondering when it would come up.

You make good threads, jackuars.

 

[Infamous]

Personally I feel that these tests should be taken with a grain of salt. Both from companies which do these independent tests and standard video reviews you can find on YouTube. The reason for this is simply because each product will have their differences, and even if the product did have a good detection ratio in one test it doesn't mean it'll always have a good detection ratio since there are so many samples out there it's impossible to detect them all (and with many more coming on a daily basis). If one vendor came first on one of the tests, one of the vendors who didn't come first will detect samples which the winner vendor won't have in their database.

If you would like to test a security product, I think you need to learn about how its different features actually work and then take it from there and look at its advantages and disadvantages. For example, one vendor may focus on behavioural analysis whereas another vendor may focus on static analysis (detecting before execution). If this isn't taken into account properly, it can just end up making the product look bad when in actual fact it is great when its strength is being used as an advantage.

You really want test AVs? go get some FUD encrypted zero-minutes exploiys/malwares from some hacker buddies; because as me, they laugh at those tests, i knew some guys that have some exploits that can disable most of the big names security softs...and if you got some kernel exploits everything is done...bye bye security apps.

I rather believes some black/grey-hats than those silly test-labs using thousands of outdated samples

Since @Umbra mentioned kernel exploits, I may as well just note here - I spoke to an AVG engineer a few months back about some code I had which had the ability to overpower the self protection which was in place and terminate it without any issues (and then prevent it from starting back up again). It works best on x64 systems simply because it cannot be fixed "ethically". The same method will work with all the other security products on x64 systems and it works silently. The point I am trying to make by mentioning this is that no product is bulletproof either.

The best thing in my view would be to test out different security software and then decide which one you feel more comfortable using. Because no matter what the independent tests promote as the winners, it doesn't mean that the winning vendors really are the "best" - truth is there is no best security product.

If you test a security product the results may show that the product is really good at protecting the system. If an independent company tests the product the results may show the opposite. The same statement works both ways. No tests will really show if a product is good or bad, because the product will react differently in different situations and will detect different malware samples.

 

[illumination]

i knew some guys that have some exploits that can disable most of the big names security softs...and if you got some kernel exploits everything is done...bye bye security apps.

These "guys" that can disable "most" security suite's, do they do so remotely, or as Admin on the system it is installed?

The second line made me chuckle, and also wonder why we have a security advising forum, as according to that sentence, we are all doomed

Neither one of these problems, would be a problem if the attacker can not make it into the system to begin with..

 

[Infamous]

These "guys" that can disable "most" security suite's, do they do so remotely, or as Admin on the system it is installed?

The second line made me chuckle, and also wonder why we have a security advising forum, as according to that sentence, we are all doomed

Neither one of these problems, would be a problem if the attacker can not make it into the system to begin with..

Well I think you are both right. Because you are right about remotely or as admin, but then if malicious software did indeed get into the system and have correct privileges then its possible for the security to be disabled like Umbra mentioned. That's just how I see it - both of you right.