AV-Comparatives: Real-World Protection Test – August 2017

212eta

Level 9
Verified
Well-known
May 11, 2011
444
The correlation between lab scenario and real-world use is vague, right?
The specific AV-Comparatives Test is called Real-World Protection Test.
These tests evaluate the suites “real-world” protection capabilities with default settings (incl. on-execution protection features).
It is our aim to do these tests rigorously

Real-World Protection Test - AV-Comparatives
Our Real-World Protection Test is currently the most comprehensive and complex test available, using a large number of test cases.
The results are based on the test set of 389 live test cases (malicious URLs found in the field), consisting of working exploits (i.e. drive-by downloads) and URLs pointing directly to malware. Thus, exactly the same infection vectors are used as a typical user would experience in everyday life. The test-cases used cover a wide range of current malicious sites and provide insights into the protection given by the various products (using all their protection features) while surfing the web.
https://www.av-comparatives.org/wp-content/uploads/2017/09/avc_factsheet2017_08.pdf
 
F

ForgottenSeer 58943

Thread author
I don't trust synthetic testing for many reasons..

I don't study their test precautions. But from an IT perspective, a company could probably 'game' test results. For example those AV's must be connected on a network and talk out during the test right? What if a company watched for those MAC addresses, IP addresses, CPU ID codes and other things and 'stroked' their product during the test from remote?

Also, working in the real world, at an MSP with 33K endpoints, we know NO protection is REALLY 100%. It's impossible in my opinion and we make sure clients know that we will do our best but cannot ever guarantee 100% protection... Think about this - have you ever installed a so-called 100% product and found a machine infected some time later? I'm sure most of us have, right? I've seen grossly infected Trend, Bit Defender, Kaspersky and especially Norton infections. In fact I have seen Norton machines in the last few weeks infected and completely subverted with File-Less malware and active botnets.

AV tests are like those warranties when you buy stuff that say 'Guaranteed Refund if it fails!'.. Then you read the fine print and find 50 conditions that have to be met that are impossible to meet so the guarantee is really nonsense.
 
P

plat1098

Thread author
@212eta Yep, I'd read the same thing you posted and unlike MRG, there is nothing explicitly stated, just assumptions you can make about what "default settings" were utilized. It's OK. It's only of interest from past circuses :mad: for equitable test methods. That's as far as I'm taking this, lol. (n):coffee:
 

RoboMan

Level 34
Verified
Top Poster
Content Creator
Well-known
Jun 24, 2016
2,399
I don't trust synthetic testing for many reasons..

I don't study their test precautions. But from an IT perspective, a company could probably 'game' test results. For example those AV's must be connected on a network and talk out during the test right? What if a company watched for those MAC addresses, IP addresses, CPU ID codes and other things and 'stroked' their product during the test from remote?

Also, working in the real world, at an MSP with 33K endpoints, we know NO protection is REALLY 100%. It's impossible in my opinion and we make sure clients know that we will do our best but cannot ever guarantee 100% protection... Think about this - have you ever installed a so-called 100% product and found a machine infected some time later? I'm sure most of us have, right? I've seen grossly infected Trend, Bit Defender, Kaspersky and especially Norton infections. In fact I have seen Norton machines in the last few weeks infected and completely subverted with File-Less malware and active botnets.

AV tests are like those warranties when you buy stuff that say 'Guaranteed Refund if it fails!'.. Then you read the fine print and find 50 conditions that have to be met that are impossible to meet so the guarantee is really nonsense.
Well, to be honest, antivirus reviews and comparatives are just a tool, but they're far from being accurate. There was an interesting article something posted here while ago, where it mentioned the thousands of requisites a test should include to even get close to being accurate. That's why we all take these with a grain of salt. Just as a motivation to push users to test and try softwares on their systems and see how it performs. This is the most close to an accurate test you can get, test by yourself. All systems are different :)
 
F

ForgottenSeer 58943

Thread author
Well, to be honest, antivirus reviews and comparatives are just a tool, but they're far from being accurate. There was an interesting article something posted here while ago, where it mentioned the thousands of requisites a test should include to even get close to being accurate. That's why we all take these with a grain of salt. Just as a motivation to push users to test and try softwares on their systems and see how it performs. This is the most close to an accurate test you can get, test by yourself. All systems are different :)

The best test imo.. Take a laptop, fresh install of Win10, disable WD and Smartscreen. Drop an AV on it you want to test, then put the laptop on a DMZ port on your firewall/UTM then 'be a douche' with it. After hammering it AND exposing it to the world for a few weeks or so, how does it look? After you are done DBAN the drive and start over.

That's how my testing is taking place right now. Trend and Norton lasted under 3 days. Kaspersky lasted many days.. GData lasted almost two weeks. Granted, my network is subjected to attack by advanced, well funded actors, it's still interesting to do. While these aren't clinical level tests, they satisfy my own desire to test a product not for the masses, but to see how well it can be trusted on my own devices. As always, they will remain unpublished and largely undocumented, and only casually mentioned. Currently testing something else, which high hopes as I am running out of stuff I want to test that I would consider using on my own systems.

I have a program to automatically surf the internet, open web pages, click ads and other crap let me know. I have one. Essentially I have a program that acts like a moron, automatically, and clicks everything. :p
 
F

ForgottenSeer 58943

Thread author
Below the belt tactics by Vipre!!!!;)

Vipre is a joke.. I always laugh when we take over a company from a failed one-man-show IT company and they almost always have the cheap GFI remote management crap running. This kind of nonsense makes me actually want to buy Kaspersky.

Keep in mind though, Julian over at Threattrack (Vipre) is commonly seen out lecturing at intelligence and intelligence contractor conventions and counts some of the biggest spooks as his pals. When this guy is keynote speaker at an NSA conference you need to think about installing the product on your systems. Threattrack has been working hard to court US Govt. contracts. So much so, they moved their HQ to Reston VA, actually in the building used in the past for NSA front companies like Sensa.

Crappy tactic by them for sure. Almost as if intelligence assets work there. Oh wait..
 

simmerskool

Level 31
Verified
Top Poster
Well-known
Apr 16, 2017
2,094
ESET is always great with its super low FPs (y)
One day I might give the ESET IS a try
Panda free AV and MS also look good :D.........but now I have Immunet installed

You're usijng immunet as your primary av or with another av. I'm trying immunet 6 with BD_free on my other win7.

EDIT "using"
 
Last edited:
  • Like
Reactions: HarborFront

simmerskool

Level 31
Verified
Top Poster
Well-known
Apr 16, 2017
2,094
Excellent results this month. I keep seeing BitDefender top of the class, but i would like to hear anybody experience on this suite? Can it beat Kaspersky? Or perform similar?

I ran KIS2017 on one win7 and BDIS 2017 on win8.1. both good both seemed light. Those pc are used differently. BD ran on autopilot. KIS with many tweaks. I thought I could do more with KIS tweaks, and KIS was light and trouble-free for me, & KIS seemed lighter but it was also on stronger hardware. I liked them both, I liked KIS a little better, and don't run either on my primary box. ;)
 

Nheo_Linkin

Level 1
Verified
Feb 19, 2017
44
Why is it so different? I don't get it
Agust_2017.png
 

212eta

Level 9
Verified
Well-known
May 11, 2011
444
It's only of interest from past circuses :mad: for equitable test methods.
That's as far as I'm taking this, lol. (n):coffee:
Please, feel Free to present your *OWN* Testing Methods & Results
since the ones by AV-Comparatives do Not satisfy your Standards.

I'm looking forward to reading your work...

[Criticism without offering a better Alternative is Not constructive at all...]
 
P

plat1098

Thread author
Please, feel Free to present your *OWN* Testing Methods & Results
since the ones by AV-Comparatives do Not satisfy your Standards.

I'm looking forward to reading your work...

[Criticism without offering a better Alternative is Not constructive at all...]

My posts had to do with wondering whether Microsoft's user-dependent findings were due to the inclusion of SmartScreen. There was nothing there to "criticize" anything. My comments and their emoticons were referring to a different discussion in a different context at a different time. If my posts "confused" anyone, let this post "unconfuse" you. A lot of interest on my part but nothing negative; sorry this was misinterpreted.

Moving on, OK? Thanks.
 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,040
I use Defender on Windows 10, but one should be cautious with its excellent detection result :
'Blocked' + 'User dependent' = 100%
The 'User dependent' detection is related to SmartScreen. It is OK only if tested executables are downloaded from the Internet by the: Web Browser, One Drive, etc., to the NTFS hard disk. In the real world scenario, the users can run files from other sources too, like: pendrives (FAT32), memory cards, DVDs, ISO images - the files from those sources will be ignored by SmartScreen. There are also problems with executables downloaded from the Internet in the compressed format *.arj, *.7z (and other) or by using download managers (accelerators).
So, in the the real world scenario the 'User dependent' factor for Defender will be lower than published in the AV-Comparatives report.
The maximum 'User dependent' detection is possible, only when using something like forced SmartScreen to run executables from all sources (not only downloaded from the Internet) with the SmartScreen check.
 
Last edited:

russ0408

Level 5
Verified
Well-known
Jul 28, 2013
234
When using Windows Defender I would never use it as a standalone. I usually had Voodooshield and Zemana Premium running with it, to make sure all angles were covered.
 
  • Like
Reactions: tonibalas

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,040
When using Windows Defender I would never use it as a standalone. I usually had Voodooshield and Zemana Premium running with it, to make sure all angles were covered.
Any of them has more false positives, than SmartScreen alone, with the similar detection rate. But anyway, they can save the user in the post-exploitation stage. The user has to decide, if the greater post-exploitation security is so important in Windows 10, as to install two additional realtime security solutions.
Some users, solved this by using Standard User Account + Windows Hardening (and well updated system).
Both solutions have its pros and cons.
 
Last edited:

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top