App Review The Fallacy of Professional AV Tests

[cruelsister]

Content source

https://youtu.be/G3Kg_PAjr0s

 

[ForgottenSeer 58943]

I assume most people already knew this. One can take any existing piece of malware and modify it slightly so it evades detection and it becomes a new threat, active and functional, until discovered, analyzed and signature updates are pushed. That's pretty much common knowledge and is showing the fallacy of signature based systems for the most part.

This is similar to my claim that blacklisting websites is a losing battle. Some people install 14 different extensions in Chrome 'hoping' to catch every possible malware site as soon as possible, but they'll always be a step behind. Fortinet is very good with categorizing new web threats, but even Fortinet is always a step behind.

This is why I am advocating, researching, and working on teams looking to evolve technology to the next level. These technologies are coming, eventually. Gryhon is the first 'home' system to exploit these new systems at any measurable level. Gryphon is quite nice in it's ML/AI IPS system in that it 'watches' your devices, then uses Bayes' theorem to develop a statistical history of how the device functions 'out of the box'. Bayes' theorem creates the normalcy metric and any variance of that causes the device to be quarantined as compromised and notify the owner (you) via alerts on the mobile app. As this system evolves, I believe it's one of the better technologies going forward.

IMO Fortinet is behind the curve on this as they still use mostly traditional methods and rely more on a security fabric for increasing awareness rather than prevention itself. Keep an eye out, a lot of firms are working to address the very problem CS posted about. It's only a matter of time.

 

[illumination]

If i could give you a thousand likes for this demonstration, i certainly would. An excellent example that shoots straight to the truth of the matter, something that has been mentioned often, but either overlooked/ignored.

 

[Kuttz]

The top 6 or 5 anti virus companies that always score 100% detection rate won't allow Pro AV test sites to test with true Zero Day malware samples. If Pro-AV testers do that 100%s come down to 3, 2, 3.5, 0, 0, 1.5 etc

 

[Deleted member 178]

It is why i kept saying AV labs are just marketing tools. Only idiots fight over it.
During AV-C last year survey, when i was working for Emsisoft, i asked them to publicly publish on their reports details of the samples used, guess what was the answer: "we don't publicly disclose samples, however vendors can freely ask us the samples" ....
basically, the public is purposely kept in ignorance so they can't really evaluate a product efficiency, while vendors have free pass to adjust it for better results.

 

[509322]

You can make millions upon millions upon millions of bypass videos and nothing is going to change. It hasn't changed one bit from when you first started and it will be the same far-far into the future - until mankind is gone.

I assume most people already knew this. One can take any existing piece of malware and modify it slightly so it evades detection and it becomes a new threat, active and functional, until discovered, analyzed and signature updates are pushed. That's pretty much common knowledge and is showing the fallacy of signature based systems for the most part.

This is similar to my claim that blacklisting websites is a losing battle. Some people install 14 different extensions in Chrome 'hoping' to catch every possible malware site as soon as possible, but they'll always be a step behind. Fortinet is very good with categorizing new web threats, but even Fortinet is always a step behind.

This is why I am advocating, researching, and working on teams looking to evolve technology to the next level. These technologies are coming, eventually. Gryhon is the first 'home' system to exploit these new systems at any measurable level. Gryphon is quite nice in it's ML/AI IPS system in that it 'watches' your devices, then uses Bayes' theorem to develop a statistical history of how the device functions 'out of the box'. Bayes' theorem creates the normalcy metric and any variance of that causes the device to be quarantined as compromised and notify the owner (you) via alerts on the mobile app. As this system evolves, I believe it's one of the better technologies going forward.

IMO Fortinet is behind the curve on this as they still use mostly traditional methods and rely more on a security fabric for increasing awareness rather than prevention itself. Keep an eye out, a lot of firms are working to address the very problem CS posted about. It's only a matter of time.

No, they don't know.

The failure is the expectation that technology will solve the malware and attack problems, when it is the technology that inherently and insidiously creates the difficult-to-solve security problems in the first place. IT technology + typical humans = guaranteed mayhem and calamity.

Unless you take an operating system and start disabling the stuff on it that makes it inherently vulnerable, it remains vulnerable no matter what security soft(s) are installed.

Lock down the system and one ends up with a quite secure system - without the hassles that some people actively claim it will cause.

Many things shipped with operating systems and software are not a good thing. Just because some publisher includes or implements something that meets limited customer demand, doesn't make it good. It only makes it a good idea for that publisher's bottom line without any prioritization of the best security interests of its users. Microsoft... hint, hint.

So the best security solution for those that are not inclined to be security enthusiasts it is best practice to get the hell off of Windows entirely and use Chromebook.

Rules based solutions are always behind the 8 Ball. Hell, some top security soft publishers only implemented ransomware protection just barely a year-and-half ago... almost 30 years after the first appearance of a ransom based attack.

Rules based solutions are typically implemented based upon the incidence of attacks - which mostly accounts for the long delay in their implementation.

 

[ForgottenSeer 58943]

So the best security solution for those that are not inclined to be security enthusiasts it is best practice to get the hell off of Windows entirely and use Chromebook.

100% agreement here. As you know, I sold off all of our Windows Notebooks and purchased a case of Chromebooks. It's been 100% smooth sailing with Chromebooks. No chance of infection, I powerwash them once a month just for the heck of it since it only takes 9 seconds. Even my riskiest users doing the most risky things, in dangerous locations haven't been able to screw up my Chromebooks.

I feel so confident with Chromebooks I don't even bother with any security extensions or even monitoring them. But after weeks when I check on them, it's the same as usual.. Totally clean, running fast, batteries that last forever. Love it! Anyone that doesn't NEED Windows should quite simply stop using it.

My home is 75% Chromebook, 20% Debian/FreeBSD/Linux, and 5% Windows now. Good times. With the security theater largely behind me I've actually softened my stance and have been working to improve internet speed and flexibility, reducing hassles of bad page blocks, and generally focusing on usability.

I keep a notebook handy for important things. Really important things never see an electronic device anymore. But if they did, they'd see a Chromebook. My windows boxes just play games for an 8 hour period, then the router/UTM shuts them down and blocks their internet. I do a Win10 reset on them every 3 months and call it a day.

 

[Sunshine-boy]

But with Chromebook, I can't play league of legend, I can't use Fx audio enhancer(its important for me), I can't use Adobe illustrator and splash player premium(i don't like other players )!
Chromebook is also expensive! I can get free update when I'm using windows but with Chromebook, there is always a life cycle ( 5 years I think)! so you pay a lot of money and after 5 years you have to buy a new one! Chromebook is not for third world countries! maybe its good for ppl like ForgottenSeer 58943 who want to pays a lot of money.

 

[TairikuOkami]

One can take any existing piece of malware and modify it slightly so it evades detection and it becomes a new threat, active and functional, until discovered, analyzed and signature updates are pushed. That's pretty much common knowledge and is showing the fallacy of signature based systems for the most part.

Yes, that is why I gave up on AVs long time ago. It is an never ending war, some battles are won, some lost. What is the point of fighting then?

This is similar to my claim that blacklisting websites is a losing battle. Some people install 14 different extensions in Chrome 'hoping' to catch every possible malware site as soon as possible, but they'll always be a step behind.

True, but chances of being infected by the fresh sample are unlikely, it takes days to spread. K9 blocks all links on MalCode/PhishTank.

Lock down the system and one ends up with a quite secure system - without the hassles that some people actively claim it will cause.

The hassle takes a few mins, but people want everything to be served to them. It is easier to just install AV and be done with it.

The top 6 or 5 anti virus companies that always score 100% detection rate won't allow Pro AV test sites to test with true Zero Day malware samples.

There was once a french testing site, testing fresh samples daily, ESET had the top score 50-60%, the rest was around 30-40%. No wonder, it is gone.

 

[upnorth]

@cruelsister What " Pro " testing sites are you talking about?

 

[509322]

But with Chromebook, I can't play league of legend, I can't use Fx audio enhancer(its important for me), I can't use Adobe illustrator and splash player premium(i don't like other players )!
Chromebook is also expensive! I can get free update when I'm using windows but with Chromebook, there is always a life cycle ( 5 years I think)! so you pay a lot of money and after 5 years you have to buy a new one! Chromebook is not for third world countries! maybe its good for ppl like ForgottenSeer 58943 who want to pays a lot of money.

Like I said, Chromebook is perfect for the average person who doesn't use a system for anything other than surfing the net, watching movies, and the occasional office applications. For them, it is perfect and provides very high security.

For my own personal use, I have an Acer Chromebook for Work 14. And I love it. No problems. No hassles. Very long battery life. Light.

But sooner or later, once enough people start using Chromebook, it will be heavily targeted too. Just like Windows. And that time won't be too far off considering that Chromebook popularity and use is growing at a high rate.

 

[509322]

The hassle takes a few mins, but people want everything to be served to them. It is easier to just install AV and be done with it.

Yeah, well... until people are educated so that their expectations and wants are realistic, nothing will change.

People want it all or think they can achieve perfect security by spending $0 or $5 on a soft, install it, and then forget it.

 

[509322]

Yes, that is why I gave up on AVs long time ago. It is an never ending war, some battles are won, some lost. What is the point of fighting then?

It's a war that cannot be won. And the security industry has been losing the war since day 1. Malware and malicious attack growth is accelerating at a far faster rate than the industry can keep up with.

 

[Nestor]

This is really true, but on the other hand, AV testing must simulate real life I think, so maybe it will be more possible to come through old malwares than zero day in life of every moderate user.

 

[Windows_Security]

When a typical blacklist antivirus provides protection against a new sample of malware (99% an adopted version of existing malware), somewhere in the world some poor PC user was the first victim in 40% of the cases (assuming static analysis, machine learning, code emulation, behavioral analysis, HIPS et cetera provide protection for other 60%).

Better to fall back to whitelisting, Would that sample have passed Windows Defender Smart screen or Avast in hardened aggressive mode, I wonder.

To survive an attack of a LION, you don't have to outrun the lion. You only have to be faster than one other person. AV companies share new detection samples. Even with blacklisting a high first victim risk (chance of surviving a lion attack) does not automatically translate to a high infection risk (chance of encountering a lion in real life). Why bother to pay for an AV at all?

At the moment the tour de France is running. The power (in kilowatt) what a cyclist can deliver is kept secret (otherwise a cyclist would know at what level of effort/strain he could break a competitor). TimeToDetection performance is the kilowatt secret of the AV-industry. Asking for TTD (although valid) is like fighting Windmill's . It is not going to happen Dona Cruel Sister

 

[Snickers102]

But sooner or later, once enough people start using Chromebook, it will be heavily targeted too. Just like Windows. And that time won't be too far off considering that Chromebook popularity and use is growing at a high rate.

Then the next chromebook will come around, and so on

It's a war that cannot be won. And the security industry has been losing the war since day 1. Malware and malicious attack growth is accelerating at a far faster rate than the industry can keep up with.

Perhaps appguard should step up their marketing, give em some fire Lockdown

Yes, that is why I gave up on AVs long time ago. It is an never ending war, some battles are won, some lost. What is the point of fighting then?

Give up on AVs, break entire windows, worth?

The hassle takes a few mins, but people want everything to be served to them. It is easier to just install AV and be done with it.

So it's easier to just strip windows naked?

This is really true, but on the other hand, AV testing must simulate real life I think, so maybe it will be more possible to come through old malwares than zero day in life of every moderate user.

The point of the tests is to test which AV is the best, we all know that for old malware all the AVs can detect it, it's with zero-day malware where they struggle, otherwise there is little difference

Better to fall back to whitelisting, Would that sample have passed Windows Defender Smart screen or Avast in hardened aggressive mode, I wonder.

People would whitelist at least some of the stuff, downloaded some image ending with .jpg.exe? "That's a legit image, stupid smart screen hahaha", whereas AV doesn't ask people "is this legit stuff or not?", it says "This stuff is bad" and user is like "Aah, good find, guess I'm not gonna open it". Ofc there's always the person that'll allow it anyway, but that's the outlier

To survive an attack of a LION, you don't have to outrun the lion. You only have to be faster than one other person. AV companies share new detection samples. Even with blacklisting a high first victim risk (chance of surviving a lion attack) does not automatically translate to a high infection risk (chance of encountering a lion in real life). Why bother to pay for an AV at all?

Statistically, the majority of malware attacks are from old malware (not counting phishing and stuff), not brand new zero-day or close-to-zero-day which the AV may not have picked up yet, so AVs makes sense for the average user. Besides, there are free AVs that are pretty good relative to some paid AVs

At the moment the tour de France is running. The power (in kilowatt) what a cyclist can deliver is kept secret (otherwise a cyclist would know at what level of effort/strain he could break a competitor). TimeToDetection performance is the kilowatt secret of the AV-industry. Asking for TTD (although valid) is like fighting Windmill's . It is not going to happen Dona Cruel Sister

That would assume AVs literally have a 0% detection rate on zero-day malware, which is not the case. It's not even that low, judging by the malware testing hub that we have here in MalwareTips. What you said implies as if zero-day protection is so bad that an AV is basically all about the signatures, and thus time-until-zero-day-malware-is-added-to-signatures is the only thing that matters in an AV, which is not the case I think. Also, your cyclist analogy is bad. Just because you can see how fast the other cyclist is (in this case in terms of kilowatts) doesn't mean you can input enough kilowatts to beat him. It's not like you suddenly become stronger by learning how much stronger the other cyclist is compared to you, you still have to train to obtain that strength

 

[Andy Ful]

All known popular malware tests are more faith than science. There are many random error sources in malware test procedures, so the testers cannot even calculate the statistical errors. So, when you see 100% detection score for AV1 and 98% for AV2, then it is possible that after extending the tests to all malware samples available for the tested time period, the real result would be 99% for AV1 and 99.5% for AV2.
.
Furthermore, you do not have any information about how significant is the 2% difference between 100% and 98% detection scores. It can be like the difference in the probability of having a headache or blahs once a year (both quite probable). But, it can also be like the difference in the probability to be hit by a thunderbolt or meteorite (both quite improbable).
.
Another problem is that most AV tests do not measure how quickly the concrete AV can protect customers after the first customer was infected (post infection protection). The first infected is like a guinea pig - this is also the well known and effective protection used by the animal herds.
.
For me, all real-world tests hardly can indicate that there is a significant difference in the protection of home users when using popular AVs. I am not saying that on Windows Home ed., Windows Defender is a better AV than Kaspersky IS. I am only saying that KIS can better protect the home users against "thunderbolts and meteorites". So, if someone likes the trips "beyond Earth", then KIS would be probably the better choice.

 

[Snickers102]

So, if someone likes the trips "beyond Earth", then KIS would be probably the better choice.

Based on what?

 

[Andy Ful]

Based on what?

Based on some additional KIS features, like for example, Application Control.

 

[Snickers102]

Based on some additional KIS features, like for example, Application Control.

Still can't compare with appguard, not even close (hire me Lockdown, am I doing this right?)

And smartscreen is pretty good along with cloud check