212eta

Level 9
Verified
The correlation between lab scenario and real-world use is vague, right?
The specific AV-Comparatives Test is called Real-World Protection Test.
These tests evaluate the suites “real-world” protection capabilities with default settings (incl. on-execution protection features).
It is our aim to do these tests rigorously

Real-World Protection Test - AV-Comparatives
Our Real-World Protection Test is currently the most comprehensive and complex test available, using a large number of test cases.
The results are based on the test set of 389 live test cases (malicious URLs found in the field), consisting of working exploits (i.e. drive-by downloads) and URLs pointing directly to malware. Thus, exactly the same infection vectors are used as a typical user would experience in everyday life. The test-cases used cover a wide range of current malicious sites and provide insights into the protection given by the various products (using all their protection features) while surfing the web.
https://www.av-comparatives.org/wp-content/uploads/2017/09/avc_factsheet2017_08.pdf
 

Slyguy

Level 42
Verified
I don't trust synthetic testing for many reasons..

I don't study their test precautions. But from an IT perspective, a company could probably 'game' test results. For example those AV's must be connected on a network and talk out during the test right? What if a company watched for those MAC addresses, IP addresses, CPU ID codes and other things and 'stroked' their product during the test from remote?

Also, working in the real world, at an MSP with 33K endpoints, we know NO protection is REALLY 100%. It's impossible in my opinion and we make sure clients know that we will do our best but cannot ever guarantee 100% protection... Think about this - have you ever installed a so-called 100% product and found a machine infected some time later? I'm sure most of us have, right? I've seen grossly infected Trend, Bit Defender, Kaspersky and especially Norton infections. In fact I have seen Norton machines in the last few weeks infected and completely subverted with File-Less malware and active botnets.

AV tests are like those warranties when you buy stuff that say 'Guaranteed Refund if it fails!'.. Then you read the fine print and find 50 conditions that have to be met that are impossible to meet so the guarantee is really nonsense.
 
P

plat1098

@212eta Yep, I'd read the same thing you posted and unlike MRG, there is nothing explicitly stated, just assumptions you can make about what "default settings" were utilized. It's OK. It's only of interest from past circuses :mad: for equitable test methods. That's as far as I'm taking this, lol. (n):coffee:
 

Robbie

Level 28
Verified
Content Creator
I don't trust synthetic testing for many reasons..

I don't study their test precautions. But from an IT perspective, a company could probably 'game' test results. For example those AV's must be connected on a network and talk out during the test right? What if a company watched for those MAC addresses, IP addresses, CPU ID codes and other things and 'stroked' their product during the test from remote?

Also, working in the real world, at an MSP with 33K endpoints, we know NO protection is REALLY 100%. It's impossible in my opinion and we make sure clients know that we will do our best but cannot ever guarantee 100% protection... Think about this - have you ever installed a so-called 100% product and found a machine infected some time later? I'm sure most of us have, right? I've seen grossly infected Trend, Bit Defender, Kaspersky and especially Norton infections. In fact I have seen Norton machines in the last few weeks infected and completely subverted with File-Less malware and active botnets.

AV tests are like those warranties when you buy stuff that say 'Guaranteed Refund if it fails!'.. Then you read the fine print and find 50 conditions that have to be met that are impossible to meet so the guarantee is really nonsense.
Well, to be honest, antivirus reviews and comparatives are just a tool, but they're far from being accurate. There was an interesting article something posted here while ago, where it mentioned the thousands of requisites a test should include to even get close to being accurate. That's why we all take these with a grain of salt. Just as a motivation to push users to test and try softwares on their systems and see how it performs. This is the most close to an accurate test you can get, test by yourself. All systems are different :)
 

Slyguy

Level 42
Verified
Well, to be honest, antivirus reviews and comparatives are just a tool, but they're far from being accurate. There was an interesting article something posted here while ago, where it mentioned the thousands of requisites a test should include to even get close to being accurate. That's why we all take these with a grain of salt. Just as a motivation to push users to test and try softwares on their systems and see how it performs. This is the most close to an accurate test you can get, test by yourself. All systems are different :)
The best test imo.. Take a laptop, fresh install of Windows 10, disable WD and Smartscreen. Drop an AV on it you want to test, then put the laptop on a DMZ port on your firewall/UTM then 'be a douche' with it. After hammering it AND exposing it to the world for a few weeks or so, how does it look? After you are done DBAN the drive and start over.

That's how my testing is taking place right now. Trend and Norton lasted under 3 days. Kaspersky lasted many days.. GData lasted almost two weeks. Granted, my network is subjected to attack by advanced, well funded actors, it's still interesting to do. While these aren't clinical level tests, they satisfy my own desire to test a product not for the masses, but to see how well it can be trusted on my own devices. As always, they will remain unpublished and largely undocumented, and only casually mentioned. Currently testing something else, which high hopes as I am running out of stuff I want to test that I would consider using on my own systems.

I have a program to automatically surf the internet, open web pages, click ads and other crap let me know. I have one. Essentially I have a program that acts like a moron, automatically, and clicks everything. :p
 

Slyguy

Level 42
Verified
Below the belt tactics by Vipre!!!!;)
Vipre is a joke.. I always laugh when we take over a company from a failed one-man-show IT company and they almost always have the cheap GFI remote management crap running. This kind of nonsense makes me actually want to buy Kaspersky.

Keep in mind though, Julian over at Threattrack (Vipre) is commonly seen out lecturing at intelligence and intelligence contractor conventions and counts some of the biggest spooks as his pals. When this guy is keynote speaker at an NSA conference you need to think about installing the product on your systems. Threattrack has been working hard to court US Govt. contracts. So much so, they moved their HQ to Reston VA, actually in the building used in the past for NSA front companies like Sensa.

Crappy tactic by them for sure. Almost as if intelligence assets work there. Oh wait..
 

simmerskool

Level 7
Verified
Malware Tester
ESET is always great with its super low FPs (y)
One day I might give the ESET IS a try
Panda free AV and MS also look good :D.........but now I have Immunet installed
You're usijng immunet as your primary av or with another av. I'm trying immunet 6 with BD_free on my other Windows 7.

EDIT "using"
 
Last edited:
  • Like
Reactions: HarborFront

simmerskool

Level 7
Verified
Malware Tester
Excellent results this month. I keep seeing BitDefender top of the class, but i would like to hear anybody experience on this suite? Can it beat Kaspersky? Or perform similar?
I ran KIS2017 on one Windows 7 and BDIS 2017 on Windows 8.1. both good both seemed light. Those pc are used differently. BD ran on autopilot. KIS with many tweaks. I thought I could do more with KIS tweaks, and KIS was light and trouble-free for me, & KIS seemed lighter but it was also on stronger hardware. I liked them both, I liked KIS a little better, and don't run either on my primary box. ;)
 

212eta

Level 9
Verified
It's only of interest from past circuses :mad: for equitable test methods.
That's as far as I'm taking this, lol. (n):coffee:
Please, feel Free to present your *OWN* Testing Methods & Results
since the ones by AV-Comparatives do Not satisfy your Standards.

I'm looking forward to reading your work...

[Criticism without offering a better Alternative is Not constructive at all...]
 
P

plat1098

Please, feel Free to present your *OWN* Testing Methods & Results
since the ones by AV-Comparatives do Not satisfy your Standards.

I'm looking forward to reading your work...

[Criticism without offering a better Alternative is Not constructive at all...]
My posts had to do with wondering whether Microsoft's user-dependent findings were due to the inclusion of SmartScreen. There was nothing there to "criticize" anything. My comments and their emoticons were referring to a different discussion in a different context at a different time. If my posts "confused" anyone, let this post "unconfuse" you. A lot of interest on my part but nothing negative; sorry this was misinterpreted.

Moving on, OK? Thanks.
 

Andy Ful

Level 48
Verified
Trusted
Content Creator
I use Defender on Windows 10, but one should be cautious with its excellent detection result :
'Blocked' + 'User dependent' = 100%
The 'User dependent' detection is related to SmartScreen. It is OK only if tested executables are downloaded from the Internet by the: Web Browser, One Drive, etc., to the NTFS hard disk. In the real world scenario, the users can run files from other sources too, like: pendrives (FAT32), memory cards, DVDs, ISO images - the files from those sources will be ignored by SmartScreen. There are also problems with executables downloaded from the Internet in the compressed format *.arj, *.7z (and other) or by using download managers (accelerators).
So, in the the real world scenario the 'User dependent' factor for Defender will be lower than published in the AV-Comparatives report.
The maximum 'User dependent' detection is possible, only when using something like forced SmartScreen to run executables from all sources (not only downloaded from the Internet) with the SmartScreen check.
 
Last edited:

russ0408

Level 4
When using Windows Defender I would never use it as a standalone. I usually had Voodooshield and Zemana Premium running with it, to make sure all angles were covered.
 
  • Like
Reactions: tonibalas

Andy Ful

Level 48
Verified
Trusted
Content Creator
When using Windows Defender I would never use it as a standalone. I usually had Voodooshield and Zemana Premium running with it, to make sure all angles were covered.
Any of them has more false positives, than SmartScreen alone, with the similar detection rate. But anyway, they can save the user in the post-exploitation stage. The user has to decide, if the greater post-exploitation security is so important in Windows 10, as to install two additional realtime security solutions.
Some users, solved this by using Standard User Account + Windows Hardening (and well updated system).
Both solutions have its pros and cons.
 
Last edited: