AV-Comparatives AV-Comparatives Real-World protection July-August 2019

Cortex

Level 26
Verified
Top Poster
Well-known
Aug 4, 2016
1,465
The URL false positives are usually less troublesome, especially those with low prevalence.(y)
As I said it tends to be those who know little about security whose PC's I maintain, & or relatives (though I'm trying to move away from doing this) who have problems with FP's, to them its a malicious site or malware, they don't have the knowledge or experience to differentiate between the two - I don't have an issue with this. :):)
Edited to make some sense ~
 
Last edited:

TairikuOkami

Level 35
Verified
Top Poster
Content Creator
Well-known
May 13, 2017
2,452
It's especially frustrating
Ditto. When I test AV or clean install Windows, WD removes many of my files and customized shortcuts, it is extremely annoying. :mad:

But as for an anti-virus product, I flat out am not going to use it if makes a habit of going after harmless things on my computer.
Exactly, like when I install Kaspersky and try to check processes via Process Hacker (not running as admin) and whoops, it is gone. :cautious:
 

mlnevese

Level 26
Verified
Top Poster
Well-known
May 3, 2015
1,531
False positives is not a problem to most enthusiasts that frequent forums like this. If it happens they know how to deal with it. BUT most people are not enthusiasts. In a situation like mine a lawyer in my country who was using Kaspersky back when it blocked government sites would be unable to access any of the internal Court systems and would be unable to work with that computer. Most people don't have any idea what black list and white lists are or how to manage them.

In other words a consumer level product must have as few FP as possible. It also should be as automatic as possible. If it's configurable so that a more experienced user can have a higher degree of control then it's good for both the experienced and the inexperienced user.

A business protection product on the other hand is meant to be configured by a specialist and should give as much control to that professional as possible.
 
Last edited:

Andy Ful

From Hard_Configurator Tools
Thread author
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,040
In a situation like mine a lawyer in my country who was using Kaspersky back when it blocked government sites would be unable to access any of the internal Court systems and would be unable to work with that computer. Most people don't have any idea what black list and white lists are or how to manage them.
...
This situation is very rare. Usually, AVs blocks URLs (false positives) when the website hosted something malicious (hacked website in the past) or hosted something that was wrongly recognized as malicious. I am curious how Windows Defender will score in the upcoming False Alarm test. The total number of false positives (URLs + files) in the current real-world test is rather high, but there is no information about samples' prevalence.
 

The Rectifier

New Member
Sep 12, 2019
3
1569175462909.png


Real-World Protection Test Jul-Aug 2019 – Factsheet | AV-Comparatives

I am particualry worried about ESET's performance in the latest AV-Comparatives. It consistenly falls behind the other AVs.
 

jetman

Level 10
Verified
Well-known
Jun 6, 2017
470
Symantec/Norton with a strong performance again.

I'd like to see what difference it would make if they tested Kaspersky with the Trusted Application Mode enabled. This is a feature that most applications do not offer.
 
9

93803123

I am particualry worried about ESET's performance in the latest AV-Comparatives. It consistenly falls behind the other AVs.

Please explain what the practical difference between 0.8 and 1.7 % are ? That is 3/352 and 6/352 test cases. What is there to worry about between those two values ?

See. Just what I said. People measure by the observed failures, and not the successes. ESET was successful in 346 out of 352 test cases.
 

The Rectifier

New Member
Sep 12, 2019
3
Please explain what the practical difference between 0.8 and 1.7 % are ? That is 3/352 and 6/352 test cases. What is there to worry about between those two values ?

See. Just what I said. People measure by the observed failures, and not the successes. ESET was successful in 346 out of 352 test cases.

I see your point. You are right the difference is very small. The thing that bothers me is not the value of the difference. It is the consistency that this difference occurs. Other vendors seem to maintain a 0% compromised in all these tests for many months. For example Bitdefender and Microsoft maintain a 0% compromised since February. It has been a while since ESET has scored scored a 0% compromised in a real world protection
 

roger_m

Level 41
Verified
Top Poster
Content Creator
Dec 4, 2014
3,014
I see your point. You are right the difference is very small. The thing that bothers me is not the value of the difference. It is the consistency that this difference occurs. Other vendors seem to maintain a 0% compromised in all these tests for many months. For example Bitdefender and Microsoft maintain a 0% compromised since February. It has been a while since ESET has scored scored a 0% compromised in a real world protection
ESET has excellent signatures and is almost always one of the first antiviruses to add signatures for new threats. With its default settings, the proactive protection could be better, but this can be tweaked to work better.
With Robbie's custom settings, it would score better.
 

Nightwalker

Level 24
Verified
Honorary Member
Top Poster
Content Creator
Well-known
May 26, 2014
1,339
ESET has excellent signatures and is almost always one of the first antiviruses to add signatures for new threats. With its default settings, the proactive protection could be better, but this can be tweaked to work better.
With Robbie's custom settings, it would score better.

Those settings arent that good and wont change too much, if one wants to tweak ESET, it is much better to use those HIPS rules crafted by its staff:


About ESET signatures, yes it has good signatures, but what is really great is its advanced heuristics (a form of zero day protection), most zero day malware are detected by ESET using it and some people think erroneous think that it was a signature detection.

I think there is no product with better heuristics and generic family malware detection than ESET (DNA Detection), it has always been its strong point after all.
 
Last edited:
9

93803123

I see your point. You are right the difference is very small. The thing that bothers me is not the value of the difference. It is the consistency that this difference occurs. Other vendors seem to maintain a 0% compromised in all these tests for many months. For example Bitdefender and Microsoft maintain a 0% compromised since February. It has been a while since ESET has scored scored a 0% compromised in a real world protection

I understand. A consistent record of 0 % compromise can be reassuring. Yet I wouldn't put a whole lot of faith in that kind of consistency.

Perhaps ESET is being honest while Bitdefender and Microsoft are gaming the test ? Or it could be any of a myriad of circumstances that affect these av lab test outcomes, yet are never revealed to the report readers.
 
9

93803123

Well also while some vendors get 100% on mainstream published AV tests, nobody aced the malware sample packs in the malware hub.

0% in those tests clearly doesn’t mean you’re safe from compromise, so as others have said, test results have to be taken with a grain of salt.

Consistent 0 % compromise result in a 350 sample av lab test cannot be linearly extrapolated to 0 % clean in a 1000 sample test or 2000 or 4000 or 8000 and so on. Statistics and reality do not work that way, but people make that extrapolation in their minds every time they look at such tests. As the sample size, or system usage increases, the probability of a compromise increases. The shape of that compromise probability curve is not linear or smooth. It is complex, increasing and decreasing over time.

If one is an indiscriminate downloader, then a PUP or PUA infection is almost certain. A more serious infection, it is difficult to quantify the probability, but it is less.
 
Last edited by a moderator:

mlnevese

Level 26
Verified
Top Poster
Well-known
May 3, 2015
1,531
My personal experience is that I haven't seem a single malware in my online activity for years. Saw some scam sites though. Otherwise the only recent live malware I saw was in an infected pendrive one of my clients brought to my office. It had a backdoor of some kind and was blocked and erased by Kaspersky
 
9

93803123

My personal experience is that I haven't seem a single malware in my online activity for years. Saw some scam sites though. Otherwise the only recent live malware I saw was in an infected pendrive one of my clients brought to my office. It had a backdoor of some kind and was blocked and erased by Kaspersky

The incidence of serious infection is low.

One should be confident in their choices if one has taken sound steps to protect their local system. There's no need to obsess about an infection.
 

mlnevese

Level 26
Verified
Top Poster
Well-known
May 3, 2015
1,531
I think being a “prolific downloader” is the main way a home user encounters malware. In my recent surveys, basically 100% of the dozens of pirated software I downloaded contained legitimate malware. Unfortunately unlike the “old days” it no longer is blatantly obvious when what you downloaded is bogus. These days they will either bury the real software beneath layers of fake setup.exe files, or worse, I saw a Windows Server installer ISO where the Setup.exe was repacked to drop some malware before unpacking and running the real installer.
Long story short the most likely way you’ll get malware these days as a home user is by partaking in downloading questionable stuff online. Drive by exploits do exist but they’re not at all common.
I can imagine in a business environment, phishing emails and people bringing malware onto the network is far more prevalent.

That's my point. If you stay legal the chance of seeing malware is minimal. If you don't navigate through the dark side of the web the chance of infection is minimal. The problem is when the user downloads every pirated software they can find and don't even check if the video they just downloaded is actually a video...
 

notabot

Level 15
Verified
Oct 31, 2018
703
I think being a “prolific downloader” is the main way a home user encounters malware. In my recent surveys, basically 100% of the dozens of pirated software I downloaded contained legitimate malware. Unfortunately unlike the “old days” it no longer is blatantly obvious when what you downloaded is bogus. These days they will either bury the real software beneath layers of fake setup.exe files, or worse, I saw a Windows Server installer ISO where the Setup.exe was repacked to drop some malware before unpacking and running the real installer.
Long story short the most likely way you’ll get malware these days as a home user is by partaking in downloading questionable stuff online. Drive by exploits do exist but they’re not at all common.
I can imagine in a business environment, phishing emails and people bringing malware onto the network is far more prevalent.

Just to say - Recently I opened and scanned a laptop from my student years (~15 yrs old). Back in the day I did download from torrents but only from reputable torrent creators, from memory tpb had a special color for their icons. I was using 1-2 AVs as a student (one was just static) and they never reported anything. I had forgotten about the files but using a modern AV flagged *most* of them with malware.
15 years of progress in signatures and ML static detection revealed that yesterday’s “reputable” torrent creators were in fact seeding malware and getting a clear pass from 2 AVs wasn’t sufficient.
Being the paranoid that I am I was running these from within a VM but the moral is clear pirated stuff is extremely dangerous and I found it worrying that at the time this was the AAA stuff that was the “golden standard”.

Only use software you’ve bought, if it’s too expensive use a big & reputable open source alternative
 

The Rectifier

New Member
Sep 12, 2019
3
I’m in full agreement. But I think part of the appeal of having a suite of fancy anti malware tools is for those who choose to (or have to) handle high risk content, how do we keep ourselves safe?

I will criticize 95% of AV software for having a confusing stand on piracy tools. Most of them will flag things like AutoKMS and other activators and even keygens as PUA or HackTools. But at the same time I’ve seen some (esp BitDefender and Windows Defender) incorrectly flag them as trojans or malware. To make matters worse I’ve also found a fake KMS activator (which just phones home to a botnet domain and downloads a script to run) marked as “just” a KMS tool. In the end I don’t have good advice in terms of running AV scanners on pirated software as the mechanism for determining if it is legitimate. Hopefully anyone playing that game possesses the tools and background knowledge to analyze their own binaries.

EDIT: would also like to point out that BB’s aren’t a magic bullet here either. A lot of piracy tools do things that inherently look suspicious — temporarily installing root certificates, setting themselves to run in early boot, patching binaries on the system and disabling Windows integrity checking, etc. Most BB’s rightfully mark these actions as suspicious but that’s not helpful if you’re trying to determine whether it’s acting as a piracy tool or root kit.


That's exactly what I was experiencing! I solved my problem by using ESET. It has excellent categorization regarding cracking tools etc and in my experience it flags only the truly malicious ones.
 

Andy Ful

From Hard_Configurator Tools
Thread author
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,040
...
False positives
...

Current Malware Protection test and False Alarm test will be probably published soon.

Here are the results of False Alarm test from September 2019 report:

AV-ComparativesFAtestSep2019.png


The results do not take account of sample prevalence. For example, Trend Micro had twice more false positives than Avast. But from the test, it follows that there will be ten times (or more) people who will encounter the false positives with Avast, as compared to Trend Micro.
 
Last edited:

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top