Do you really understand AV test results?

[509322]

What is the informative value of consumer tests in non-real world conditions? Every car vendor publishes fuel consumption. Every consumer knows these values will never be met in real life (commuter driving) conditions. Every smart phone vendor publishes battery life. Every consumer knows that you have to charge your phone earlier. What is the rational of your critism against real world test conditions? In what paralel (non real) world are you planning to use an antivirus?

Of course real-world. That's not what I was getting at. There was no criticism of real-world conditions.

You tell me what samples they used. Their description is as vague as and ill-defined as anything. The samples could be a couple of years old. LOL. Nobody outside the lab really knows. And sampling varies from lab to lab, test to test.

What does "representative" mean. It could mean anything. If you take a critical look at that the supplemental notes and methodology, you start to ask a lot of hard questions. But there are no answers.

The labs aren't exactly going to spell it out or hadn't anyone figured that out yet ?

The point I was making is that the labs do not test true "zero-day" malware. They are using old malware. Because old malware is "representative" malware. Day-to-day malware is run-of-the-mill malware that has been around for days, weeks, months and years. To me, that is "representative" malware.

And you can plainly see that the AVs are still missing "representative" (old) malware - even with the Ai\machine learning and other gizmos.

When you have the CTO of Emsisoft openly state that detection suxx, then you know it is endgame for the protection model. People know this rationally, but keep using antivirus. EDIT: Let me restate what I recall Fabian saying before I get falsely accused of something. What I remember that he said here or on Wilders was "yeah, signatures suck."

That is simple, any result of the standard 0-day test can be incorrect for AVs which use post-infection signatures on the fly.

If a system is infected, it is too little, too late. Post-infection signatures are great for the guy who isn't infected, but not the guy who is infected - Ebola just blew out his eyeballs.

 

[Windows_Security]

Sorry I have to disagree again (did not you noticed that I am only counter arguing for fun).

AV-test labs don't use real world conditions. I don't have a problem with their honeypots picking up old malware or their untransparent criteria of determing the representative real world set. The real problem with their current testing process, is the fact that they test only once a momnth. Most malware is only active for a few days. So their once in a month testing cycle does not represent real world conditions.

Sophos bought Invincea for over 100 million dollar. AVast took over AVG for 1.4 billion dollar. So it is safe to say that the AV-companies have much deeper pockets than the testlabs. This means that it is shocking that the testlabs are able to find new malware samples which the AV-vendors have not found (or exchanged) themselves.

Of to work now , it is snowing in the Netherlands now

 

[509322]

Sorry I have to disagree again (did not you noticed that I am only counter arguing for fun).

AV-test labs don't use real world conditions. I don't have a problem with their honeypots picking up old malware or their untransparent criteria of determing the representative real world set. The real problem with their current testing process, is the fact that they test only once a momnth. Most malware is only active for a few days. So their once in a month testing cycle does not represent real world conditions.

Sophos bought Invincea for over 100 million dollar. AVast took over AVG for 1.4 billion dollar. So it is safe to say that the AV-companies have much deeper pockets than the testlabs. This means that it is shocking that the testlabs are able to find new malware samples which the AV-vendors have not found (or exchanged) themselves.

It's all about money.

Based upon everything that I have observed, it's all a scam. The vast majority of the representative malware is days, weeks, and older. Some will say that is too harsh. Whatever. It all comes down to how does the industry establish standards to which highly disparate products can be fairly tested and compared ? Well, there is an absolute scale way of doing it, but none of the publishers is going to 1) be willing to pay for such testing and 2) willing to have the results published if they don't show their product in a good light.

Like most things in life it comes down to money - or those jockeying for their piece of it. And that is what AV test labs and their testing industry are all about - helping security soft publishers jockey for a piece of the cash pie. The AV test lab industry is seriously and fundamentally flawed - and I'm not talking about corruption. I am referring to the standards to which everyone will agree I mentioned in the first paragraph. It's a debate that has raged forever. There are those of us that understand that fact, and all the rest that do not.

 

[Prorootect]

Lockdown wrote(my thoughts!):

"It's all about money.
Based upon everything that I have observed, it's all a scam. The vast majority of the representative malware is days, weeks, and older. Some will say that is too harsh. Whatever. It all comes down to how does the industry establish standards to which highly disparate products can be fairly tested and compared ? Well, there is an absolute scale way of doing it, but none of the publishers is going to 1) be willing to pay for such testing and 2) willing to have the results published if they don't show their product in a good light.

Like most things in life it comes down to money - or those jockeying for their piece of it. And that is what AV test labs and their testing industry are all about - helping security soft publishers jockey for a piece of the cash pie. The AV test lab industry is seriously and fundamentally flawed - and I'm not talking about corruption. I am referring to the standards to which everyone will agree I mentioned in the first paragraph. It's a debate that has raged forever. There are those of us that understand that fact, and all the rest that do not."

_____________________

- Then just have to change 1 byte, so that the AV working with database doesn't recognize the signature anymore..."yeah signatures suck". It's bad idea to work with signature database, outdated, so why bother with this kind of obsolete?
Why should we deal with this kind of obsolescence?
Good idea is to work without signatures, as does PCHunter, PowerTool or similar tools. Look onPCHunter "Setting" tab, or ithurricane PowerTool "Configuration" button on the bottom of GUI... Here you have manual settings that are effective in the wrong situation. Efficient, without signatures.
You must surely know these tools, yet there are not many comments about, why?

 

[509322]

You must surely know these tools, yet there are not many comments about, why?

I know IT Hurricane's PowerTool. Super Security Geek Tool. Not actively developed. No documentation. No learning resources. Same as GMER and the others. Requires too much knowledge and a user can destroy their system with it.

Average Joe will pop an artery in da brain trying to figure it out. He's got much more important priorities like watching WWF and gaming. Learning about IT security is not a statistically "typical" person priority.

 

[Deleted member 65228]

Then just have to change 1 byte, so that the AV working with database doesn't recognize the signature anymore..."yeah signatures suck". It's bad idea to work with signature database, outdated, so why bother with this kind of obsolete?
Why should we deal with this kind of obsolescence?

This is for checksum hash signatures. Generic signatures can be bad if the malware author is experienced enough to work out what section of code is causing the flag due to the byte pattern/s used in scanning however a good memory scanner can be great against packing techniques for on-execution flagging.

As for utilities like PC Hunter, they are not for everyday use. They do not only list the bad and requires a trained eye to comprehend the results. "My AV registers a kernel-mode callback? OMG KERNEL MODE ROOTKIT!" - see.

 

[Prorootect]

Opcode: "As for utilities like PC Hunter, they are not for everyday use."

- the contrary I see: click on "Minimize to tray" button in PowerTool, that's this

"They do not only list the bad and requires a trained eye to comprehend the results."
- sure

 

[Prorootect]

Quote from another thread:

Safe download link on Softpedia.com: PowerTool Download

'A rather small application in stature but powerful enough to detect kernel alterations that occur due to the presence of various malware components

PowerTool is a security tool developed to offer you a simple means of keeping your computer clean of rootkit viruses and fixing kernel structure modifications.

Rootkit viruses act on an administrative privileges and hide inside certain processes or applications. They can exploit certain system vulnerabilities that might lead to loss of personal information or even system override.

PowerTool scans and analyzes files at kernel level which means that the scans get as thorough as possible. The application displays a comprehensive interface with a tabbed structure which makes it very easy to use and navigate.

It’s separated into sections such as ‘System’, ‘Process’, ‘Kernel Module’, ‘Application’, ‘Registry’, ‘Services’ and ‘Startup’ which means that it covers all the crucial system file types that may be subjected to infection. The application doesn't just detect problems, it can also fix them. With it you are able to repair Registry Editor and Task Manager problems and various typical errors. ...'

-----------------
Lockdown wrote: ,Not actively developed' - hmm


Runs on all Windows:
runs on:
Windows XP • Windows Vista 64 bit • Windows 7 • Windows 7 64 bit • Windows 8 • Windows 8 64 bit • Windows 2003 • Windows 2008 • Windows 2008 64 bit • Windows 10 • Windows 10 64 bit

-----------------
Some other thoughts from Softpedia:

'Since malware components are able to alter the kernels structure of your system, the application allows you to lock certain privileges. For example, you can have PowerTool forbid the creation of processes and threads, disable registry editing and deny creating of any files.

The application also provides features that allow you to locate the scanned files on your computer, verify their signatures and dump memory, and even upload them online to be scanned for malicious content. As far as Startup goes, the application offers you a full list of all the process that run and enables you to backup a selected file or delete it.
PowerTool comes with a self-protect feature...'

 

[Andy Ful]

If a system is infected, it is too little, too late. Post-infection signatures are great for the guy who isn't infected, but not the guy who is infected - Ebola just blew out his eyeballs.

Yes, it looks like trying to save people, after killing them.
It is not the good solution for Enterprises/Companies (targetted attacks, big local networks).
It can work, as a rescue solution when other security fails.
It is a 'guinea pig' kind of solution, to gather 0-day signatures on masses, and give the better protection for Enterprises.
It can by useful only when AI can create post-infection signatures in some minutes.
It can compensate the lack of 0-day signatures.
There is no proof that it works as it could work in theory, because the standard tests cannot measure such protection.

 

[509322]

Yes, it looks like trying to save people, after killing them.
It is not the good solution for Enterprises/Companies (targetted attacks, big local networks).
It can work, as a rescue solution when other security fails.
It is a 'guinea pig' kind of solution, to gather 0-day signatures on masses, and give the better protection for Enterprises.
It can by useful only when AI can create post-infection signatures in some minutes.
It can compensate the lack of 0-day signatures.
There is no proof that it works as it could work in theory, because the standard tests cannot measure such protection.

The only way to keep people from infecting themselves is to lock them out of their PCs and devices. If it doesn't come pre-installed, then guess what, you're beat ! Even Microsoft knows this and is experimenting with 10S. In the future, it will be one of the layered answers.

Today, people think being locked out of the OS as a personal insult and an attack upon their fundamental right to do as they see fit - even if it means inflicting self-harm. "We're locked out ! We have no choices to install our self-inflicted 3rd-party garbage ! We have rights ! We've been robbed of our choices ! We've been cheated !"

In the future, attitudes will change. "Time and pressure..."

 

[Prorootect]

The only way to keep people from infecting themselves is to lock them out of their PCs and devices. If it doesn't come pre-installed, then guess what, you're beat ! Even Microsoft knows this and is experimenting with 10S. In the future, it will be one of the layered answers.

Today, people think being locked out of the OS as a personal insult and an attack upon their fundamental right to do as they see fit - even if it means inflicting self-harm. "We're locked out ! We have no choices to install our self-inflicted 3rd-party garbage ! We have rights ! We've been robbed of our choices ! We've been cheated !"

Better is to lock down certain Windows privileges...

'to lock certain privileges. For example, you can have PowerTool forbid the creation of processes and threads, disable registry editing and deny creating of any files.'

 

[509322]

Better is to lock down certain Windows privileges...

'to lock certain privileges. For example, you can have PowerTool forbid the creation of processes and threads, disable registry editing and deny creating of any files.'

The best way to protect users from themselves is to lock them out of the OS along with a tight OS. Yes, I agree.

 

[Deleted member 65228]

@Prorootect Yes, all these rootkit scanning tools as suggested are not appropriate for a normal average user to make use of. If a trained malware removal expert requests it then that is fair enough, and scanners like Malwarebytes Anti-Rootkit are fine.

If you use utilities like PC Hunter, GMER, etc. You need to have knowledge on rootkits. Preferrably hands-on experience with actually developing them (in the sense of testing purposes of course - or at least doing extensive research).

KiSystemServiceTable has been abused to hook NtOpenProcess and the responsible driver is from AVG. "OMG my system is infected and AVG is compromised" could be an initial thought.

Google search history five minutes later
"What is a driver"
"what is a hook"
"what is a callback"
"i have a bootkit from clicking a cat image in chrome help me"
....

It just is not practical for a normal environment. Run one of those utilities yourself, do you personally understand what everything means, regardless of you being on a security forum? I doubt it. Don't take that as an insult, I am just being realistic.

 

[Andy Ful]

When we talk about AV signatures/protection, it seems that everyone here is right in some way.
There are many good ways to do something, but usually, only a few that are optimal.
For example, the 0-day run-time protection of Avast CyberCapture (lock the file up to 2 hours + cloud analysis) can be great for protection and detection (good test results), but is hardly convenient. Many users will simply turn it off. More optimal from the global point of view is adopting 'mini CyberCapture' (lock the file only on some seconds + cloud analysis) and creating post-infection signatures for the malware that passes 'mini CyberCapture' ('guinea pig' solution). It will not shine probably in detection tests, but can be the optimal protection for masses + some bonuses for Enterprises. Maybe the best solution, would be adding some rollback feature like in Kaspersky System Watcher + KSN.
.
As for AV testing, it is a complicated problem. It resembles me a Miss Beauty World contests. They are not so helpful, when you are looking for a wife.

 

[509322]

There are many good ways to do something, but usually, only a few that are optimal.

For average Joe, detection by signature is the optimal protection. That is why there is still antivirus in this day and age. Or at least that is the general consensus, belief, tradition, or whatever one wishes to call it. I think that false hope is exactly what it is - an anachronistic scam of epic proportions and utter nonsense that people keep on buying into, but that is just me.

We keep using gasoline automobiles, despite everybody knowing full well that they are bad because their emissions are contributing to increasing climate temperatures. Nobody has come up with anything most people will accept over the fossil fuel automobile and nobody is willing to pay for nor implement the alternatives. Same general concept in IT security.

 

[Deleted member 65228]

For example, the 0-day run-time protection of Avast CyberCapture (lock the file up to 2 hours + cloud analysis) can be great for protection and detection (good test results), but is hardly convenient.

The DeepScreen feature (or if they have since re-named it/shuffled things around) was really flawed in the past and it likely still is even today. It would run the sample in a "virtualised" state for X amount of seconds to determine whether it believed it was malicious or clean based on the behavior on-execution. Sounds great!

Reality? Wait 10-20 seconds before executing any "intriguing" code. All you literally must do is sleep for around 10-20 seconds - there is absolutely nothing complex behind doing this, it is a one liner. The feature will then determine it as clean and allow it to actively run on the main Host environment, no questions asked. If you ask me, that work-around is far too simplistic. The idea behind it was good but it wasn't very effective in my opinion based on hands-on testing with it.

I recon this is why they pushed further and proceeded with the Behaviour Shield... Because their current implementations of dynamic protection at the time weren't performing as well as they had planned probably.

 

[509322]

When we talk about AV signatures/protection, it seems that everyone here is right in some way.
There are many good ways to do something, but usually, only a few that are optimal.
For example, the 0-day run-time protection of Avast CyberCapture (lock the file up to 2 hours + cloud analysis) can be great for protection and detection (good test results), but is hardly convenient. Many users will simply turn it off. More optimal from the global point of view is adopting 'mini CyberCapture' (lock the file only on some seconds + cloud analysis) and creating post-infection signatures for the malware that passes 'mini CyberCapture' ('guinea pig' solution). It will not shine probably in detection tests, but can be the optimal protection for masses + some bonuses for Enterprises. Maybe the best solution, would be adding some rollback feature like in Kaspersky System Watcher + KSN.
.
As for AV testing, it is a complicated problem. It resembles me a Miss Beauty World contests. They are not so helpful, when you are looking for a wife.

Hell, Webroot has been doing that sort of thing for years (over a decade) with their file monitoring and their cloud Ai\machine learning and still can't manage to keep systems clean. Their guidance is that it will take up to 4 hours for a system to be rolled back. In testing I have seen systems at hour +144 not rolled back. You report that kind of stuff and I don't know where the report goes - the abyss - but a reply never, ever, comes back.

I don't know how many times, before I went to work for Blue Ridge Networks, I carefully gathered files, logs, hashes, and whatever else was requested, documented it and submitted the stuff. All I know, in the end, is that detection by hash was implemented.

 

[509322]

The things we are discussing here - like Ai and Machine Learning, were the promises that were made 20 years ago. Security fundamentals have always worked. Stick to the basics. The basics will take you further into the future than anything else. 1 + 1 = 2. If you block it from executing in the first place, then you do not have deal with any unpleasant consequences afterwards. Ai, Machine Learning, behavioral analysis, etc - they require you to execute something. If they don't catch it, you're beat ! You're much better off blocking by default. Default deny will save you much pain. So plainly obvious and common sense, you have to wonder why so very few people actually adhere to it.

 

[Andy Ful]

Hell, Webroot has been doing that sort of thing for years (over a decade) with their file monitoring and their cloud Ai\machine learning and still can't manage to keep systems clean. Their guidance is that it will take up to 4 hours for a system to be rolled back. In testing I have seen systems at hour +144 not rolled back. You report that kind of stuff and I don't know where the report goes - the abyss - but a reply never, ever, comes back.

I don't know how many times, before I went to work for Blue Ridge Networks, I carefully gathered files, logs, hashes, and whatever else was requested, documented it and submitted the stuff. All I know, in the end, is that detection by hash was implemented.

It is a real hell, because even the Windows native rollback (from System Restore Point) sucks sometimes (even when not infected).
I read somewhere that Kaspersky rollback solution works, but I read this also for Webroot. The reality is more cruel.

 

[harlan4096]

KL System Watcher is not a "panacea", sometimes misses changes and can't block encrypting files... even if They are constantly improving it: Updatable modules beta-testing

I send them from time to time via pm some reports (collected traces for malware that bypass SW)...