App Review The Fallacy of Professional AV Tests

[Andy Ful]

There is one important thing to be considered when going "beyond Earth". The KIS spaceship is known to be less stable than WD spaceship - hard choice.:emoji_pray:

 

[Deleted Member 3a5v73x]

The point of the tests is to test which AV is the best, we all know that for old malware all the AVs can detect it, it's with zero-day malware where they struggle, otherwise there is little difference

There are still somewhere months and even years undetected malware.

 

[illumination]

There is one important thing to be considered when going "beyond Earth". The KIS spaceship is known to be less stable than WD spaceship - hard choice.:emoji_pray:

Wise words in a out of this world analogy.

 

[Andy Ful]

...
And smartscreen is pretty good along with cloud check

Yes, SmartScreen is probably the best application reputation service available. But, it cannot protect you against malicious files like: payloads (even with EXE extension), scripts, scriptlets, documents, etc.
'Block at first sight' (cloud check) is also good but not as good as SmartScreen.

 

[Andy Ful]

By the analogy - for the home users, the difference between the two tested AVs is usually like the difference between Aspirin and Paracetamol for fighting the headache. Aspirin may be a better choice for some people because it seems to protect also against heart attack. On the other side, Paracetamol seems to be less invasive to the stomach.

 

[Burrito]

There was once a french testing site, testing fresh samples daily, ESET had the top score 50-60%, the rest was around 30-40%. No wonder, it is gone.

I'd forgotten about that... until you just mentioned it. Anybody know of anything else like that 'out there?'

It is a never-ending war, some battles are won, some lost. What is the point of fighting then?

Because it's fun and interesting. Sometimes the war is the point. Sometimes it's the path, not the peak. Many of us could dump the process and just adopt a solution (images, Chromebook, Linux, Deep Freeze...) and never come back to MalwareTips again. But what fun would that be? Let the war rage on....

 

[Handsome Recluse]

Like I said, Chromebook is perfect for the average person who doesn't use a system for anything other than surfing the net, watching movies, and the occasional office applications. For them, it is perfect and provides very high security.

For my own personal use, I have an Acer Chromebook for Work 14. And I love it. No problems. No hassles. Very long battery life. Light.

But sooner or later, once enough people start using Chromebook, it will be heavily targeted too. Just like Windows. And that time won't be too far off considering that Chromebook popularity and use is growing at a high rate.

Probably not gonna be prevalent on third world countries because they're always behind and they have too poor internet to use it.

 

[cruelsister]

You can make millions upon millions upon millions of bypass videos and nothing is going to change.

Sadly I must totally agree with you. If I wasn't such an arrogant self-important Bitch I would have given up a long time ago. The Truth is always much harder to take than Deception, so folks tend to ignore it (reminds me of a line from the movie "Inherit The Wind'- "I don't like to think about things that I don't like to think about"). One prefers a 99% detection rate for their security solution given by an "Authority" instead of the actual Real World 20% or less.

This attitude plays into the hands of the Blackhats (and makes Ophelia drool).

 

[Snickers102]

One prefers a 99% detection rate for their security solution given by an "Authority" instead of the actual Real World 20% or less.

The vast majority of home users won't get infected by a targeted NSA zero-day, they'll download a .jpg.exe file and run it or something stupid like that, normal AVs are perfectly fine for them, even if they're waaaaaay inferior to something like appguard (cough cough), how often do you think the average user encounters malware that is less than a few days old?

 

[Windows_Security]

That would assume AVs literally have a 0% detection rate on zero-day malware, which is not the case. It's not even that low, judging by the malware testing hub that we have here in MalwareTips. What you said implies as if zero-day protection is so bad that an AV is basically all about the signatures, and thus time-until-zero-day-malware-is-added-to-signatures is the only thing that matters in an AV, which is not the case I think.

You quoted from a post which started with

When a typical blacklist antivirus provides protection against a new sample of malware (99% an adopted version of existing malware), somewhere in the world some poor PC user was the first victim in 40% of the cases (assuming static analysis, machine learning, code emulation, behavioral analysis, HIPS et cetera provide protection for other 60%).

Did you overlook that I posted that I guessed at least 60% of the new samples (or zero days) are blocked by AV's? I also mentioned that the real infection risk is lower because samples are shared and the chance of being a first victim is much lower (as the remaining 40% might suggest). Is it a typical case of black Tuesday short memory fall out as a result of enjoying a good weekend?

 

[Windows_Security]

Statistically, the majority of malware attacks are from old malware (not counting phishing and stuff), not brand new zero-day or close-to-zero-day which the AV may not have picked up yet, so AVs makes sense for the average user. Besides, there are free AVs that are pretty good relative to some paid AVs

Well my analogy of the lion got lost in translation. You start to disagree while we (at least that is how I read it) both argue that the chance of an average user being the first victim of a brand new malware first victim risk, is much lower than the fail rate of an AV against such a zero day (chance of encounter x risk of infection).

I will blame myself and not argue about TTD and kilowatt analogy. I consider them both a secret of profession. No AV will tell its protection rate in TTD context in public, like no professional cyclist will tell is kilowatt in public. Forget about that analogy, fact is that Cruel Sister makes a valid point. But I think it won't going to happen. A 99% protection rate without TTD context looks way better than a 40% protection of zero day, 60% protection of 1 day old and 80% of two day old and 99,99% of three day old samples.