AV-Comparatives AV-Comparatives Real-World protection July-August 2019

[Andy Ful]

Te latest AV-Comparatives Real-World Protection Test (Jul-Aug 2019):

Test chart:
https://www.av-comparatives.org/com...chart_month=Jul-Aug&chart_sort=1&chart_zoom=2

Test document in PDF:
Real-World Protection Test Jul-Aug 2019 – Factsheet | AV-Comparatives

***********************************************************************************
False positives

The false-alarm test in the Whole-Product Dynamic “Real-World” Protection Test consists of two parts: wrongly blocked domains (while browsing) and wrongly blocked files (while downloading/installing).
https://www.av-comparatives.org/real-world-protection-test-methodology/
***********************************************************************************

From AV-Comparatives Real-World test, one cannot conclude how many false positives can produce the AV engine while downloading/executing the files.
This is tested separately for AV-Comparatives Malware Protection tests in False Alarm tests, for example: False Alarm Test March 2019 | AV-Comparatives

Current Malware Protection test and False Alarm test will be probably published soon.

 

[SeriousHoax]

The results are based on the test set of 352 live test cases (malicious URLs found in the field), consisting of working exploits (i.e. drive-by downloads) and URLs pointing directly to malware.

What does it mean actually? AVs that fail to block the malicious url but blocked the sample locally is considered a pass too, right?

 

[shmu26]

From AV-Comparatives Real-World test, one cannot conclude how many false positives can produce the AV engine while downloading/executing the files.

This is because the file FPs are grouped together with the url FPs, correct?
I looked at the False Alarm Test from March that you linked us to, and I see that Windows Defender is the "few FPs" category, along with Bitdefender and Kaspersky.

 

[Andy Ful]

This is because the file FPs are grouped together with the url FPs, correct?
...

That is right.

What does it mean actually? AVs that fail to block the malicious url but blocked the sample locally is considered a pass too, right?

Yes.

 

[low L!fe]

Correct Analysis
1- BitDefender 100% 0 False positive
2-Kaspersky 99.1% 0 False Positive
3-McAfee 99.7% 3 False Positive (More False Positive)
4-Symantec 100% 4 False Positive ( More more False positive)
5-Windows Defender More More More More False Positive

 

[shmu26]

Windows Defender More More More More False Positive

That's the way it looks, but that's not the way it really is. Please see @Andy Ful's comments in the very first post -- the FPs are not from files. They are from websites that are being blocked.

 

[Andy Ful]

Correct Analysis
1- BitDefender 100% 0 False positive
2-Kaspersky 99.1% 0 False Positive
3-McAfee 99.7% 3 False Positive (More False Positive)
4-Symantec 100% 4 False Positive ( More more False positive)
5-Windows Defender More More More More False Positive

You cannot say that your analysis is correct, without presenting the correct/accepted criteria.
If not, then anyone could post "his correct analysis" which would be very different from yours.
Your criteria do not rely on data (from this test) based on the number of blocked samples and the number of false positives (where is Avira 100% with 1 false positive, or K7, Total Defense, Vipre 99.7 with 1 false positive?).

 

[ForgottenSeer 58943]

I'd take 'real world' with any of these tests with a;

 

[oldschool]

I'd take 'real world' with any of these tests with a;

View attachment 224905

Only if it's Himalayan salt!

 

[Burrito]

I'd take 'real world' with any of these tests with a;

View attachment 224905

I think that may be a high-quality uncut diamond.

Sly..... I don't know if I'd put that much 'value' in the test.

I'm just sayin'...

 

[93803123]

False positives are highly over-rated as a negative.

I'd take 'real world' with any of these tests with a;

View attachment 224905

Looks like a giant crystal meth rock.

Compare

 

[Venustus]

I just can't resist

 

[Cortex]

It's took a long time but I really have given up taking these tests seriously, as to FP's with the none savvy (everyone I seem to know) they are as troublesome as malware is ~

 

[Andrew3000]

Bitdefender 100%

 

[93803123]

Bitdefender 100%
View attachment 224929

This is a fair symbolic representation of the experience with software generally.

 

[Andy Ful]

It's took a long time but I really have given up taking these tests seriously, as to FP's with the none savvy (everyone I seem to know) they are as troublesome as malware is ~

The URL false positives are usually less troublesome, especially those with low prevalence.

 

[mlnevese]

The URL false positives are usually less troublesome, especially those with low prevalence.

The only problem is when the URL false positive stops you from doing your work. Kaspersky used to report many government sites I need for my work as "potential data loss". They eventually solved it but for a time I had to whitelist those sites and some would get blocked even when whitelisted.

 

[blackice]

The URL false positives are usually less troublesome, especially those with low prevalence.

I always push back on the railing on M$ for false positives. Anecdotally it’s always people who dabble in esoteric software that most users would never even know existed. Even a lot of advanced users aren’t using tons of tools with low prevalence, which is where most FPs come from.

Now your discussion of how FPs are done in the testing and contrasting URL vs Files really shines a light on how misguided this “concern” is for most users.

 

[conceptualclarity]

False positives are highly over-rated as a negative.

I disagree. If a security product does a lot of false positives I find it a strong disincentive to scan with it, because i think "I just don't have time now to deal with all those false positives." It's especially frustrating when a false-positive-prone program shows me registry keys with only numbers, nothing to indicate what the keys actually have to do with.

But as for an anti-virus product, I flat out am not going to use it if makes a habit of going after harmless things on my computer.

 

[93803123]

I disagree. If a security product does a lot of false positives I find it a strong disincentive to scan with it, because i think "I just don't have time now to deal with all those false positives." It's especially frustrating when a false-positive-prone program shows me registry keys with only numbers, nothing to indicate what the keys actually have to do with.

But as for an anti-virus product, I flat out am not going to use it if makes a habit of going after harmless things on my computer.

You shouldn't need to run a scan more than once. And that's immediately after installing it on a known, clean-state system. Afterwards, you are going to have to have faith in your choice that the real-time protection will protect you. Scanning a system more than once per month is a waste of your time and your system resources.

Very few users have major issues with false positives. A small percentage have annoyances with false positives or, more likely, the publisher rates a program as PUA\PUP.