AVLab.pl Modern protection without signatures – comparison test on real threats (Advanced In The Wild Malware Test)

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Nightwalker

Level 24
Verified
Honorary Member
Top poster
Content Creator
Well-known
May 26, 2014
1,314
I did notice that their database has indeed improved against PE executable malware, but for whatever reason it remains a horror versus scriptors and this is not good at all.

With pentest mode enabled it is somewhat equal or better than some "tradicional" antivirus solutions, but of course it cant hold a candle to Cruel Comodo, WiseVector StopX or Kaspersky System Watcher.
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
117
The difference between this test and Malware Protection tests made by AV-Test or AV-Comparatives is that the samples are (on average) a few days old in AVLab tests and a few weeks old in AV-Test/AV-Comparatives tests. Also, the Malware Protection tests are used as a kind of reference (subsidiary) tests to the Real-World tests.
Unfournatelly, a Community doesn't have access to their malware sample hash, however we can read the AV-C and AV-T's methodolgy and you have right.

Anyway, it is not generally true that Level 3 of this test represents modern protection without any signatures or shows the true effectiveness of protection against fresh/unknown malware. I think that the author had in mind only the local offline signatures. Also, the true effectiveness of protection is narrowed here only to EXE payloads (scripting methods are skipped).

I cannot agree with you.

Level 3 is responsible for non-signature protection, because at this level, the sample is always running. Before that, malware has a chance to be detected, because it is in the system for about 60 seconds.

Furthermore, "Level 3" is marked in the our testing database with the correct detection technology due to the IoC e.g. Comodo IS:

Comodo Internet Security:

ANTIVIRUS INDICATORSDESCRIPTION
C:\ProgramData\Comodo\Cis\Quarantine\*Malware was quarantined
C:\VTRoot\*Virus was run in a sandbox
C:\ProgramData\Comodo\Firewall Pro\cislogs.sdbInformation on an event in a firewall
HKLM\SYSTEM\VritualRoot\*Information on running in a sandbox

The key is the "Quarantine path" (Level 2) and (as Level 3 - non-quarantine event) C:\VTRoot, HKLM\SYSTEM\VritualRoot\* in Sysmon logs view. To capture such of indicators you have to create manually a XML configuration and import to Sysmon. Very important is set the Sysmon's driver altitude at lower than AVs drivers. Please read the updated methodology, how to to that: Methods of carrying out automatic tests - AVLab.pl

So, Level 2 is when the malware is captured to Quarantine (before running!): The system level, i.e. a virus has been downloaded, but it hasn’t been allowed to run.

The Level 3 is after when the signatures can be used: we can see it on the detection logs of e.g. firewalls (cislogs.sdb) or auto-sandbox (C:\VTRoot\*, HKLM\SYSTEM\VritualRoot\*). These two IoC shows that malware was blocked by modern protection.

Very similar to another security product with different IoC.

Indeed, we can be more transparent (thanks for the idea!) and add an all IoC to the Table Results.

I hope, everything is clearer now :)

Take a look at the March 1 base dump: Comodo stopped Level 2 malware in quarantine.

And another screen - malware is stopped by aut-sandbox (Level 3).
 

Attachments

  • Zrzut ekranu 2022-03-3 o 15.04.20.png
    Zrzut ekranu 2022-03-3 o 15.04.20.png
    410.6 KB · Views: 23
  • Zrzut ekranu 2022-03-3 o 15.08.37.png
    Zrzut ekranu 2022-03-3 o 15.08.37.png
    180.4 KB · Views: 18
Last edited:

cruelsister

Level 40
Verified
Honorary Member
Top poster
Content Creator
Well-known
Apr 13, 2013
2,905
Level 3 represents modern protection without any signatures. In such cases, a virus is run in the operating system. It is the most dangerous situation but needed because it shows true effectiveness of protection against and 0-day files – a threat unknown to a developer of protection software.
Far too many of the popular tests mindlessly chose quantity over quality by testing various AM applications against vast numbers of malware samples (many of which could be essentially redundant) without actually running the samples in a test system (where VM naive systems are preferred, of course). Although a sugnature only based tests are amusing, they are hardly real world.

It is very nice to see a test that understands that the true measure of the effectiveness of anti-malware applications does not reside in signature based (dumb) detection. but instead will also emphasize the ability to prevent infections based on mechanism of action.

Dziękuję bardzo!

 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top poster
Developer
Well-known
Dec 23, 2014
7,245
I cannot agree with you.

Level 3 is responsible for non-signature protection, because at this level, the sample is always running. Before that, malware has a chance to be detected, because it is in the system for about 60 seconds.

I did not intend to contradict your post but only make it slightly clearer. I did not say that non-signature protection is excluded from Level 3. I only said that "it is not generally true that Level 3 of this test represents modern protection without any signatures or shows the true effectiveness of protection against fresh/unknown malware." I had in mind that there can still be many signature-based detections in the cloud backend at Level 3 (so I mentioned that you had probably in mind the offline signatures).
  1. Some AVs can delay the file execution at Level 3 and check the file against the cloud backend (fast signatures, behavior-based detections, detonation in the sandbox, etc). So, in this case, the file is not really executed at Level 3. The analysis and detection are made at the pre-execution stage.
  2. An example can be Microsoft Defender when the BAFS feature was not triggered before execution. That is what exactly happens in this test. Other examples: Avast CyberCapture, Avast Hardened Mode, Norton Insight, Windows SmartScreen for Explorer, some Defender's ASR rules, Emsisoft Behavior Blocker reputation check, Comodo Firewall File Lookup (before sandboxing), etc.
  3. From point 1 it also follows that Level 3 can often include detections of already known malware and the file is not obliged to run at this level.
  4. Some AVs can also analyze and detect the file after it has been executed. Such post-execution features can be triggered by the telemetry sent to the cloud backend (like in the case of Microsoft Defender). Also at this stage, fast signatures can be used.
So I would use other words to describe this level: Level 3 represents the protection against fresh malware.
I intentionally did not use the words "modern protection", because the protection used by Emsisoft and Comodo is rather old (but still very effective for fresh samples).
Of course, Level 3 includes also many behavior-based detections, sandboxing, etc., so it also represents in a great deal the non-signature protection. But to be sure if the detection/block was signature dependent or not, one has to always take a closer look at the AV particular detection.(y)
 
Last edited:

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top poster
Developer
Well-known
Dec 23, 2014
7,245
If the intention of this test was to show which AV has got the best protection against threats blocked at Level 3, then one can make the conclusions as follows:
  1. Comodo, Emsisoft, and Malwarebytes are among the top products. This does not exclude any of the other AVs except Defender. We already know that Defender blocked 78% of threats at Level 3 and no more. Other AVs could in theory block more on Level 3 with disabled web protection.
  2. One cannot say much about the protection against threats blocked at Level 3 provided by other AVs, except that for Defender and Avast this protection is good (only Comodo, Emsisoft, and Malwarebytes blocked more at Level 3, for sure).
  3. One cannot exclude the possibility that the protection of Defender at Level 3 might be better compared to Avira, F-Secure, and SecureAPlus (they did not block more at Level 3).
  4. The fact that Defender missed 22% of threats is not conclusive here, because these threats might be blocked on Level 1 and 2 by other AVs (so irrelevant for Level 3 blocks). Even if Defender would use web protection (like SmartScreen) or BAFS then these threats would be blocked at Level 1 or 2, so the protection at Level 3 would not change for Defender, too.
As we can see the test results for blocking threats at Level 3 are not very informative except for Comodo, Emsisoft, and Malwarebytes.:(

Edit.
The task would be much simpler when testing all products on the same footing, by forcing AVs to use only Level 3. Of course, such tests could not be reliable as a measure of overall protection.
 
Last edited:

MacDefender

Level 16
Verified
Top poster
Oct 13, 2019
792
I did not intend to contradict your post but only make it slightly clearer. I did not say that non-signature protection is excluded from Level 3. I only said that "it is not generally true that Level 3 of this test represents modern protection without any signatures or shows the true effectiveness of protection against fresh/unknown malware." I had in mind that there can still be many signature-based detections in the cloud backend at Level 3 (so I mentioned that you had probably in mind the offline signatures).
  1. Some AVs can delay the file execution at Level 3 and check the file against the cloud backend (fast signatures, behavior-based detections, detonation in the sandbox, etc). So, in this case, the file is not really executed at Level 3. The analysis and detection are made at the pre-execution stage.
  2. An example can be Microsoft Defender when the BAFS feature was not triggered before execution. That is what exactly happens in this test. Other examples: Avast CyberCapture, Avast Hardened Mode, Norton Insight, Windows SmartScreen for Explorer, some Defender's ASR rules, Emsisoft Behavior Blocker reputation check, Comodo Firewall File Lookup (before sandboxing), etc.
  3. From point 1 it also follows that Level 3 can often include detections of already known malware and the file is not obliged to run at this level.
  4. Some AVs can also analyze and detect the file after it has been executed. Such post-execution features can be triggered by the telemetry sent to the cloud backend (like in the case of Microsoft Defender). Also at this stage, fast signatures can be used.
So I would use other words to describe this level: Level 3 represents the protection against fresh malware.
I intentionally did not use the words "modern protection", because the protection used by Emsisoft and Comodo is rather old (but still very effective for fresh samples).
Of course, Level 3 includes also many behavior-based detections, sandboxing, etc., so it also represents in a great deal the non-signature protection. But to be sure if the detection/block was signature dependent or not, one has to always take a closer look at the AV particular detection.(y)
I would also add that both F-Secure DeepGuard and Kaspersky System Watcher have signatures too. They are runtime behavior blockers who can have signature updates pertaining to the runtime behavior, like a HIPS with downloadable rules.
 

wat0114

Level 8
Verified
Well-known
Apr 5, 2021
363
What blocking mechanisms are typically being used in Level 3 protection? Is it sandboxing, outbound firewall control, scripting protection, HIPS, or maybe a combination of these, or something else? I'm guessing with Comodo products it's sandboxing of the payload and/or reputation signature analysis?
 
Last edited:

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top poster
Developer
Well-known
Dec 23, 2014
7,245
What blocking mechanisms are typically being used in Level 3 protection? Is it sandboxing, outbound firewall control, scripting protection, HIPS, or maybe a combination of these, or something else? I'm guessing with Comodo products it's sandboxing of the payload and/or reputation analysis?
That can depend on the AV. Anyway, at Level 3 the AVs usually trigger the strongest features because the threat is going to be executed (even if the execution is delayed). Some AVs use only pre-execution features, others can use also post-execution features based on the telemetry of running processes. But, the strong pre-execution features can be triggered also at Levels 1 or 2, if the AV protection is closely integrated with the web browser. An example of such protection can be Defender's BAFS, which works best with Chromium browsers. It can sometimes trigger at Level 1 or 2, even stronger pre-execution protection than at Level 3.
Unfortunately, this test is prepared in a special way that forces Defender to use only Level 3, which mimics the scenario when the already infected system downloads/drops/execute payloads.
Among the non-signature features triggered at Level 3 are HIPS, Local ASR, Local AI, AMSI, Local behavior-based detections, Local sandboxing, and their cloud counterparts (including Deep learning). Some AVs can also trigger reputation-based features or provide Network protection for the executed processes.

Examples of behavior blocking (related to Defender but probably also common for other AVs):
  • Credential dumping from LSASS
  • Cross-process injection
  • Process hollowing
  • User Account Control bypass
  • Tampering with antivirus (such as disabling it or adding the malware as exclusion)
  • Contacting Command and Control (C&C) to download payloads
  • Coin mining
  • Boot record modification
  • Pass-the-hash attacks
  • Installation of root certificate
  • Exploitation attempt for various vulnerabilities
 
Last edited:

wat0114

Level 8
Verified
Well-known
Apr 5, 2021
363
Among the non-signature features triggered at Level 3 are HIPS, Local ASR, Local AI, AMSI, Local behavior-based detections, Local sandboxing, and their cloud counterparts (including Deep learning). Some AVs can also trigger reputation-based features or provide Network protection for the executed processes.

Examples of behavior blocking (related to Defender but probably also common for other AVs):
  • Credential dumping from LSASS
  • Cross-process injection
  • Process hollowing
  • User Account Control bypass
  • Tampering with antivirus (such as disabling it or adding the malware as exclusion)
  • Contacting Command and Control (C&C) to download payloads
  • Coin mining
  • Boot record modification
  • Pass-the-hash attacks
  • Installation of root certificate
  • Exploitation attempt for various vulnerabilities

So many more than I knew of. Thanks!
 

MacDefender

Level 16
Verified
Top poster
Oct 13, 2019
792
Yeah, traditional examples for consumer AVs include “behavior blocking” (monitoring Win32 API calls made by a process), as well as less exciting approaches like inspecting memory contents or network activity using a signature scanner.

The latter, in particular, was seen quite a bit for ESET in the Malware Hub. A lot of malware is broken into two stages…. The first stage is a highly obfuscated and difficult to detect stub that’s meant to fetch the real payload and help it evade detection. The second stage tends to be recycled from existing malware and contains a lot more complex code that’s difficult to hide from signature scanner. Though ESET detected almost everything upon static scan (level 2 or earlier), some of the detections were at runtime either thanks to the network scanning or looking at temporary files during execution.
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
117
What blocking mechanisms are typically being used in Level 3 protection? Is it sandboxing, outbound firewall control, scripting protection, HIPS, or maybe a combination of these, or something else? I'm guessing with Comodo products it's sandboxing of the payload and/or reputation signature analysis?

As Andy replied to you - it depends on AV. In our tests, Level 3 is the most dangerous, because it is already running malware. That's why proactive protection and dynamic analysis in the cloud is in the game, e.g. for Avira (by the way, their infrastructure is based on a modified Cuckoo Sandbox, if anyone is interested - I think they have it described deep in the documents somewhere, or they had it in the past). Moreover the Cuckoo Sandbox is used in VoodooShield and in some project by CERT Poland: Strengthening our malware analysis capabilities

Though ESET detected almost everything upon static scan (level 2 or earlier), some of the detections were at runtime either thanks to the network scanning or looking at temporary files during execution.

Eset SS is included in March edition advanced in the wild malware test, therefore please follow me on this brilliant forum :) Results will be available in mid April.
 

cruelsister

Level 40
Verified
Honorary Member
Top poster
Content Creator
Well-known
Apr 13, 2013
2,905
Was Cloud-delivered protection enabled or disabled in the tested products?
Cloud detection implies signature based detection which was not the point of the test, and it is refreshing to view a testing scenario that releases us from the shackles of the typical scan and compare AV philosophy.
 

wat0114

Level 8
Verified
Well-known
Apr 5, 2021
363
Cloud detection implies signature based detection which was not the point of the test, and it is refreshing to view a testing scenario that releases us from the shackles of the typical scan and compare AV philosophy.

Okay that makes sense, thanks. It's just that I didn't see anywhere in the AVLabs test page how, exactly, the individual products were configured with regards to which of their individual settings were enabled or disabled. Maybe that's all listed somewhere, and I'm too blind to see them :unsure:

EDIT

I even tried finding more information on individual product setup here: Methods of carrying out automatic tests - AVLab.pl

but so far I haven't been able to find anything that clearly illustrates how each tested product was configured for this particular test.
 
Last edited:
Top