- Oct 2, 2020
- 455
Results for Emsisoft are also expected. By default it doesn't scan files until they're executed so there is no L2 protection when in default config.
Modern protection without signatures – comparison test on real threats (Advanced In The Wild Malware Test)
- Thread starter Adrian Ścibor
- Start date
- Apr 13, 2013
- 3,224
I did notice that their database has indeed improved against PE executable malware, but for whatever reason it remains a horror versus scriptors and this is not good at all.
- May 26, 2014
- 1,339
With pentest mode enabled it is somewhat equal or better than some "tradicional" antivirus solutions, but of course it cant hold a candle to Cruel Comodo, WiseVector StopX or Kaspersky System Watcher.
- Apr 9, 2018
- 220
Unfournatelly, a Community doesn't have access to their malware sample hash, however we can read the AV-C and AV-T's methodolgy and you have right.
I cannot agree with you.
Level 3 is responsible for non-signature protection, because at this level, the sample is always running. Before that, malware has a chance to be detected, because it is in the system for about 60 seconds.
Furthermore, "Level 3" is marked in the our testing database with the correct detection technology due to the IoC e.g. Comodo IS:
Comodo Internet Security:
| ANTIVIRUS INDICATORS | DESCRIPTION |
| C:\ProgramData\Comodo\Cis\Quarantine\* | Malware was quarantined |
| C:\VTRoot\* | Virus was run in a sandbox |
| C:\ProgramData\Comodo\Firewall Pro\cislogs.sdb | Information on an event in a firewall |
| HKLM\SYSTEM\VritualRoot\* | Information on running in a sandbox |
The key is the "Quarantine path" (Level 2) and (as Level 3 - non-quarantine event) C:\VTRoot, HKLM\SYSTEM\VritualRoot\* in Sysmon logs view. To capture such of indicators you have to create manually a XML configuration and import to Sysmon. Very important is set the Sysmon's driver altitude at lower than AVs drivers. Please read the updated methodology, how to to that: Methods of carrying out automatic tests - AVLab.pl
So, Level 2 is when the malware is captured to Quarantine (before running!): The system level, i.e. a virus has been downloaded, but it hasn’t been allowed to run.
The Level 3 is after when the signatures can be used: we can see it on the detection logs of e.g. firewalls (cislogs.sdb) or auto-sandbox (C:\VTRoot\*, HKLM\SYSTEM\VritualRoot\*). These two IoC shows that malware was blocked by modern protection.
Very similar to another security product with different IoC.
Indeed, we can be more transparent (thanks for the idea!) and add an all IoC to the Table Results.
I hope, everything is clearer now
Take a look at the March 1 base dump: Comodo stopped Level 2 malware in quarantine.
And another screen - malware is stopped by aut-sandbox (Level 3).
Attachments
Last edited:
- Apr 13, 2013
- 3,224
Far too many of the popular tests mindlessly chose quantity over quality by testing various AM applications against vast numbers of malware samples (many of which could be essentially redundant) without actually running the samples in a test system (where VM naive systems are preferred, of course). Although a sugnature only based tests are amusing, they are hardly real world.
It is very nice to see a test that understands that the true measure of the effectiveness of anti-malware applications does not reside in signature based (dumb) detection. but instead will also emphasize the ability to prevent infections based on mechanism of action.
Dziękuję bardzo!
I was just wondering if most of the applications tested used AI?
LOLBINS
LOLBINS
Last edited by a moderator:
- Apr 5, 2014
- 6,008
Modern Protection without signatures ... Remember me using Sandboxie & Shadow Defender
I still use SDModern Protection without signatures ... Remember me using Sandboxie & Shadow Defender
- Dec 23, 2014
- 8,592
I did not intend to contradict your post but only make it slightly clearer. I did not say that non-signature protection is excluded from Level 3. I only said that "it is not generally true that Level 3 of this test represents modern protection without any signatures or shows the true effectiveness of protection against fresh/unknown malware." I had in mind that there can still be many signature-based detections in the cloud backend at Level 3 (so I mentioned that you had probably in mind the offline signatures).
- Some AVs can delay the file execution at Level 3 and check the file against the cloud backend (fast signatures, behavior-based detections, detonation in the sandbox, etc). So, in this case, the file is not really executed at Level 3. The analysis and detection are made at the pre-execution stage.
- An example can be Microsoft Defender when the BAFS feature was not triggered before execution. That is what exactly happens in this test. Other examples: Avast CyberCapture, Avast Hardened Mode, Norton Insight, Windows SmartScreen for Explorer, some Defender's ASR rules, Emsisoft Behavior Blocker reputation check, Comodo Firewall File Lookup (before sandboxing), etc.
- From point 1 it also follows that Level 3 can often include detections of already known malware and the file is not obliged to run at this level.
- Some AVs can also analyze and detect the file after it has been executed. Such post-execution features can be triggered by the telemetry sent to the cloud backend (like in the case of Microsoft Defender). Also at this stage, fast signatures can be used.
I intentionally did not use the words "modern protection", because the protection used by Emsisoft and Comodo is rather old (but still very effective for fresh samples).
Of course, Level 3 includes also many behavior-based detections, sandboxing, etc., so it also represents in a great deal the non-signature protection. But to be sure if the detection/block was signature dependent or not, one has to always take a closer look at the AV particular detection.
Last edited:
- Dec 23, 2014
- 8,592
If the intention of this test was to show which AV has got the best protection against threats blocked at Level 3, then one can make the conclusions as follows:
Edit.
The task would be much simpler when testing all products on the same footing, by forcing AVs to use only Level 3. Of course, such tests could not be reliable as a measure of overall protection.
- Comodo, Emsisoft, and Malwarebytes are among the top products. This does not exclude any of the other AVs except Defender. We already know that Defender blocked 78% of threats at Level 3 and no more. Other AVs could in theory block more on Level 3 with disabled web protection.
- One cannot say much about the protection against threats blocked at Level 3 provided by other AVs, except that for Defender and Avast this protection is good (only Comodo, Emsisoft, and Malwarebytes blocked more at Level 3, for sure).
- One cannot exclude the possibility that the protection of Defender at Level 3 might be better compared to Avira, F-Secure, and SecureAPlus (they did not block more at Level 3).
- The fact that Defender missed 22% of threats is not conclusive here, because these threats might be blocked on Level 1 and 2 by other AVs (so irrelevant for Level 3 blocks). Even if Defender would use web protection (like SmartScreen) or BAFS then these threats would be blocked at Level 1 or 2, so the protection at Level 3 would not change for Defender, too.
Edit.
The task would be much simpler when testing all products on the same footing, by forcing AVs to use only Level 3. Of course, such tests could not be reliable as a measure of overall protection.
Last edited:
I would also add that both F-Secure DeepGuard and Kaspersky System Watcher have signatures too. They are runtime behavior blockers who can have signature updates pertaining to the runtime behavior, like a HIPS with downloadable rules.I did not intend to contradict your post but only make it slightly clearer. I did not say that non-signature protection is excluded from Level 3. I only said that "it is not generally true that Level 3 of this test represents modern protection without any signatures or shows the true effectiveness of protection against fresh/unknown malware." I had in mind that there can still be many signature-based detections in the cloud backend at Level 3 (so I mentioned that you had probably in mind the offline signatures).
So I would use other words to describe this level: Level 3 represents the protection against fresh malware.
- Some AVs can delay the file execution at Level 3 and check the file against the cloud backend (fast signatures, behavior-based detections, detonation in the sandbox, etc). So, in this case, the file is not really executed at Level 3. The analysis and detection are made at the pre-execution stage.
- An example can be Microsoft Defender when the BAFS feature was not triggered before execution. That is what exactly happens in this test. Other examples: Avast CyberCapture, Avast Hardened Mode, Norton Insight, Windows SmartScreen for Explorer, some Defender's ASR rules, Emsisoft Behavior Blocker reputation check, Comodo Firewall File Lookup (before sandboxing), etc.
- From point 1 it also follows that Level 3 can often include detections of already known malware and the file is not obliged to run at this level.
- Some AVs can also analyze and detect the file after it has been executed. Such post-execution features can be triggered by the telemetry sent to the cloud backend (like in the case of Microsoft Defender). Also at this stage, fast signatures can be used.
I intentionally did not use the words "modern protection", because the protection used by Emsisoft and Comodo is rather old (but still very effective for fresh samples).
Of course, Level 3 includes also many behavior-based detections, sandboxing, etc., so it also represents in a great deal the non-signature protection. But to be sure if the detection/block was signature dependent or not, one has to always take a closer look at the AV particular detection.
- Apr 5, 2021
- 621
What blocking mechanisms are typically being used in Level 3 protection? Is it sandboxing, outbound firewall control, scripting protection, HIPS, or maybe a combination of these, or something else? I'm guessing with Comodo products it's sandboxing of the payload and/or reputation signature analysis?
Last edited:
- Dec 23, 2014
- 8,592
That can depend on the AV. Anyway, at Level 3 the AVs usually trigger the strongest features because the threat is going to be executed (even if the execution is delayed). Some AVs use only pre-execution features, others can use also post-execution features based on the telemetry of running processes. But, the strong pre-execution features can be triggered also at Levels 1 or 2, if the AV protection is closely integrated with the web browser. An example of such protection can be Defender's BAFS, which works best with Chromium browsers. It can sometimes trigger at Level 1 or 2, even stronger pre-execution protection than at Level 3.
Unfortunately, this test is prepared in a special way that forces Defender to use only Level 3, which mimics the scenario when the already infected system downloads/drops/execute payloads.
Among the non-signature features triggered at Level 3 are HIPS, Local ASR, Local AI, AMSI, Local behavior-based detections, Local sandboxing, and their cloud counterparts (including Deep learning). Some AVs can also trigger reputation-based features or provide Network protection for the executed processes.
Examples of behavior blocking (related to Defender but probably also common for other AVs):
- Credential dumping from LSASS
- Cross-process injection
- Process hollowing
- User Account Control bypass
- Tampering with antivirus (such as disabling it or adding the malware as exclusion)
- Contacting Command and Control (C&C) to download payloads
- Coin mining
- Boot record modification
- Pass-the-hash attacks
- Installation of root certificate
- Exploitation attempt for various vulnerabilities
Last edited:
- Apr 5, 2021
- 621
So many more than I knew of. Thanks!
Yeah, traditional examples for consumer AVs include “behavior blocking” (monitoring Win32 API calls made by a process), as well as less exciting approaches like inspecting memory contents or network activity using a signature scanner.
The latter, in particular, was seen quite a bit for ESET in the Malware Hub. A lot of malware is broken into two stages…. The first stage is a highly obfuscated and difficult to detect stub that’s meant to fetch the real payload and help it evade detection. The second stage tends to be recycled from existing malware and contains a lot more complex code that’s difficult to hide from signature scanner. Though ESET detected almost everything upon static scan (level 2 or earlier), some of the detections were at runtime either thanks to the network scanning or looking at temporary files during execution.
The latter, in particular, was seen quite a bit for ESET in the Malware Hub. A lot of malware is broken into two stages…. The first stage is a highly obfuscated and difficult to detect stub that’s meant to fetch the real payload and help it evade detection. The second stage tends to be recycled from existing malware and contains a lot more complex code that’s difficult to hide from signature scanner. Though ESET detected almost everything upon static scan (level 2 or earlier), some of the detections were at runtime either thanks to the network scanning or looking at temporary files during execution.
- Apr 9, 2018
- 220
As Andy replied to you - it depends on AV. In our tests, Level 3 is the most dangerous, because it is already running malware. That's why proactive protection and dynamic analysis in the cloud is in the game, e.g. for Avira (by the way, their infrastructure is based on a modified Cuckoo Sandbox, if anyone is interested - I think they have it described deep in the documents somewhere, or they had it in the past). Moreover the Cuckoo Sandbox is used in VoodooShield and in some project by CERT Poland: Strengthening our malware analysis capabilities
Eset SS is included in March edition advanced in the wild malware test, therefore please follow me on this brilliant forum
Results will be available in mid April.
- Apr 5, 2021
- 621
Last edited:
- Apr 13, 2013
- 3,224
Cloud detection implies signature based detection which was not the point of the test, and it is refreshing to view a testing scenario that releases us from the shackles of the typical scan and compare AV philosophy.
- Apr 5, 2021
- 621
Okay that makes sense, thanks. It's just that I didn't see anywhere in the AVLabs test page how, exactly, the individual products were configured with regards to which of their individual settings were enabled or disabled. Maybe that's all listed somewhere, and I'm too blind to see them
EDIT
I even tried finding more information on individual product setup here: Methods of carrying out automatic tests - AVLab.pl
but so far I haven't been able to find anything that clearly illustrates how each tested product was configured for this particular test.
Last edited:
- Apr 9, 2018
- 220
Default settings. In most cases it means enabled cloud protection, if exist.
*EDIT*
Default settings > however we chose silent mode and/or move threat to quarantinne automatically without user confirmation.
Similar threads
