March-April 2018 test results: More insights into industry AV tests

[Bot]

In a previous post, in the spirit of our commitment to delivering industry-leading protection, customer choice, and transparency on the quality of our solutions, we shared insights and context into the results of AV-TESTs January-February 2018 test cycle. We released a transparency report to help our customers and the broader security community to stay informed and understand independent test results better.

In the continued spirit of these principles, wed like to share Windows Defender AVs scores in the March-April 2018 test. In this new iteration of the transparency report, we continue to investigate the relationship of independent test results and the real-world protection of antivirus solutions. We hope that you find the report insightful.

Below is a summary of the transparency report:

	Protection: Windows Defender AV achieved an overall Protection score of 5.5/6.0, missing 2 out of 5,680 malware samples (0.035% miss rate). With the latest results, Windows Defender AV has achieved 100% on 9 of the 12 most recent tests (combined “Real World” and “Prevalent malware”).
	Usability (false positives):Windows Defender AV maintained its previous score of 5.5/6.0. Based on telemetry, most samples that Windows Defender AV incorrectly classified as malware (false positive) had very low prevalence and are not commonly used in business context. This means that it is unlikely for these false positives to affect enterprise customers.
	Performance: Windows Defender AV maintained its previous score of 5.5/6.0 and continued to outperform the industry in most areas. These results reflect the investments we made in optimizing Windows Defender AV performance for high-frequency actions.

The report aims to help customers evaluate the extent to which test results are reflective of the quality of protection in the real world. At the same time, insights from the report continue to drive further improvements in the intelligent security services that Microsoft provides for customers.

Windows Defender AV and the rest of the built-in security technologies in Windows Defender Advanced Threat Protection work together to create a unified endpoint security platform. In real customer environments, this unified security platform provides intelligent protection, detection, investigation, and response capabilities that are not currently reflected in independent tests. We tested the two malware samples that Windows Defender AV missed in the March-April 2018 test and proved that for both missed samples, at least three other components of Windows Defender ATP would detect or block the malware in a true attack scenario. You can find these details and more in the transparency report.

Download the complete transparency report on March-April 2018 AV-TEST results

 

[slash/]

It was always obvious that AV-TEST doesn't cover every single testing aspect of security software.
The timing of this transparency report seems like Microsoft is trying to defend themselves from WD's overall score dropping 0.5 points.

 

[ForgottenSeer 69673]

Ok so was this a test with Windows 10 Enterprise? I am confused again.

 

[slash/]

Ok so was this a test with Windows 10 Enterprise? I am confused again.

Most likely. Windows Defender ATP requires Windows 10 Enterprise/Educational E5 licensing.

 

[RoboMan]

Article summary: Now we're paying AV-testing companies too, notice us.

 

[illumination]

Article summary: Now we're paying AV-testing companies too, notice us.

Translation: Excellent PR

 

[509322]

Ok so was this a test with Windows 10 Enterprise? I am confused again.

AV-Test tests Windows Defender. The same Windows Defender that is on Home, Pro and Education\Enterprise.

What the report is blatantly attempting to do is to sell the Windows Defender-Windows Advanced Threat Protection (ATP) "stack."

Notice Microsoft states their products are short-changed because the lab tested Windows Defender, "in isolation", without the benefit of their ATP "stack."

Just go ahead and get your hands on all those products that Microsoft is pushing so hard. After spending a lot of money, just try configuring those products using Microsoft's absolutely atrocious, basically nonexistent documentation.

Good luck.

 

[illumination]

It's a test of Windows Defender. The same Windows Defender that is on Home, Pro and Education\Enterprise.

What the report is blatantly attempting to do is to sell the Windows Defender-Windows Advanced Threat Protection (ATP) "stack."

What i just said with 2 words above your post

 

[509322]

What i just said with 2 words above your post

Just more attempted Microsoft scamming. Wholly my own point of view.

I suppose Windows ATP sales aren't doing so good. So sad.

 

[cruelsister]

Unless the Pro AV testing sites divulge the EXACT age of the samples used the results are without any realistic meaning.

Serious Blackhats- those that are actively pushing out malware- will morph these malware files every 12-24 hours to avoid detection from the bulk of AV products. Therefore the results of tests using malware samples a few days old will be significantly different from "Real World" malware that are 24 hours old or less.

I'll be releasing a video tonight to demonstrate what I mean.

 

[ForgottenSeer 69673]

Unless the Pro AV testing sites divulge the EXACT age of the samples used the results are without any realistic meaning.

Serious Blackhats- those that are actively pushing out malware- will morph these malware files every 12-24 hours to avoid detection from the bulk of AV products. Therefore the results of tests using malware samples a few days old will be significantly different from "Real World" malware that are 24 hours old or less.

I'll be releasing a video tonight to demonstrate what I mean.

Cool cool, I will be waiting. but really you should be drinking a few glasses of wine, and watching Netflix and grilling this time of year M. That is what I do and so will watch you video tomorrow. have you selected your choice of music yet.
a good song for one of your best videos would be eve of destruction but it has the words you might have to edit out.
Barry McGuire...

 

[dJim]

sorry but no way WD can be " that good " im just installed avast free and guess what.. if found a virus on recicle bin from long time ago.

 

[upnorth]

LOL @ticklemefeet . I mean that hint/tip was just brilliant as more should follow it. Great song btw.

 

[Andy Ful]

Even when Pro AV testing sites will adopt the EXACT age of the samples (like @cruelsister rightly proposes), the tests will be not very useful for enterprises, because of the targetted attacks.
Many agencies, organizations, enterprises, etc. use unpatched, older Windows versions and vulnerable routers. So, the danger of attacks via the exploit kits is very probable. Furthermore, in the targetted attacks, the malware will be often a true 0-day (not detected by any AV). Such malware samples will be beyond the definition of the sample age as the time period between the moment of the first detection (by some AV in the world) and the moment of the test.
In fact, one should also assume in the test that the enterprise network can be compromised by a true 0-day malware, so it is important to measure how resistant it is after infection and how quickly the malware can be neutralized.
Making the test that could properly include the above would be an extremely difficult task.

 

[cruelsister]

Andy- My compliments on a most excellent post. Yes indeed, proper testing would be a difficult task. All malware would have to be stuff that is being actively pushed out (less than 24 hours old) and once the samples are collected the testing MUST BE DONE ON ALL TESTED PRODUCTS SIMULTANEOUSLY. But as this would be not only a lot of work as well as being a Pain in the Ass we get the crap that is shoveled in front of us whenever a Pro (and sadly amateur) Test comes out.

The fact that an accurate test is difficult to accomplish does not make a flawed test valid. Any Pro or amateur video that will give a traditional AV product anything over 50% detection should be discounted immediately as they are obviously using older samples, and who knows how many older morphed versions of the same stuff.

As to Enterprise testing, yes, targeted attacks certainly occur. But these malware are typically not something that will need some unpatched vulnerability or anything arcane but instead will utilize malware that "Whatever Security solution" can not distinguish from being legitimate. Thus the multiple breaches in Banks and Retail where the malware is nothing but a simple (Kinda simple anyway) script. You may notice that when telling what sort of malware the Pros site used in an Enterprise test the words "worm" or "python" never come up...

 

[Snickers102]

Any Pro or amateur video that will give a traditional AV product anything over 50% detection should be discounted immediately as they are obviously using older samples, and who knows how many older morphed versions of the same stuff.

How did you create this 50% breakpoint? Have you done any tests? Or just slapping numbers around?

As to Enterprise testing, yes, targeted attacks certainly occur. But these malware are typically not something that will need some unpatched vulnerability or anything arcane but instead will utilize malware that "Whatever Security solution" can not distinguish from being legitimate. Thus the multiple breaches in Banks and Retail where the malware is nothing but a simple (Kinda simple anyway) script. You may notice that when telling what sort of malware the Pros site used in an Enterprise test the words "worm" or "python" never come up...

Again, I don't take anything anyone says for granted, and so should others, so could you specify where do you have that information from and what evidence is it based on? It sounds hard to believe that AVs in 2018 will fail to detect something so simple as a "kinda simple" script, it's almost as if anyone can quickly write a batch file and he can suddenly "hack" any enterprise he wants, just doesn't make sense

 

[cruelsister]

Or just slapping numbers around?

I NEVER just pull data out of my (shapely) behind. For any truly new malware there will be a First Detection by someone (normally Kaspersky or Qihoo) and then the others will follow; some fast, some not so fast. I'm sure that there will be a few well respected members here that can conform this- or else just watch my last video for an example.

so could you specify where do you have that information from and what evidence is it based on?

I worked for a time at a Company that did post-breach analysis and Remediation (a peasant on a few, the Primary on others) and so have forst hand knowledge. Although I will not be more specific, if you are actually interested ask Professor Google about the Home Depot and Target beaches.

anyone can quickly write a batch file and he can suddenly "hack" any enterprise he wants, just doesn't make sense

I agree- it does not make sense. Please see some of the 3 years of my videos...

 

[illumination]

I'm sure that there will be a few well respected members here that can conform this- or else just watch my last video for an example.

I will be your huckleberry....

Pro or amateur testing samples are pre packed, some facilities collect samples all year long and then utilize them, most sample uploading and analysis sites that others use run them through automated sandbox malware analysis and generally they are run through sites like virus total, so these samples at this point, are already on the radar and detection growing. You can physically see this by visiting sites most testers frequent. You may also read the disclaimer at the testing facilities on how they collect and test "if this is even available as some are just vague with details.

Finding completely undetected samples is just not a simple thing, and would require much time to collect the amount of samples both professionals and amateurs claim to have done.

 

[Snickers102]

I agree- it does not make sense. Please see some of the 3 years of my videos...

I will, when I have more time! You're on my "to watch" list. Along with my queue of "to read" articles. Only a few thousand of them remaining And they become more and more by the day, because there's so much interesting stuff to read/watch and so little time. But, you just won a PRIORITY INCREASE!!! I will now move your channel higher on the queue, so I'll get to it faster

I'm sure that when I do get to your videos, I will miss them when I watch all of them, cuz there will be none left to watch (and they look interesting), so have you thought of creating more videos? You've done only 2 in the last 3 months, what's going on?

 

[Andy Ful]

I agree with @cruelsister. The malware has not to be very complicated to infect the Enterprise network. All the attacker needs is finding a weak point. It can be a simple CHM file with the embedded script, etc.