This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
We encourage you to compare these results with others and take informed decisions on what security products to use.
Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.
I did notice that their database has indeed improved against PE executable malware, but for whatever reason it remains a horror versus scriptors and this is not good at all.
Unfournatelly, a Community doesn't have access to their malware sample hash, however we can read the AV-C and AV-T's methodolgy and you have right.The difference between this test and Malware Protection tests made by AV-Test or AV-Comparatives is that the samples are (on average) a few days old in AVLab tests and a few weeks old in AV-Test/AV-Comparatives tests. Also, the Malware Protection tests are used as a kind of reference (subsidiary) tests to the Real-World tests.
Anyway, it is not generally true that Level 3 of this test represents modern protection without any signatures or shows the true effectiveness of protection against fresh/unknown malware. I think that the author had in mind only the local offline signatures. Also, the true effectiveness of protection is narrowed here only to EXE payloads (scripting methods are skipped).
|C:\ProgramData\Comodo\Cis\Quarantine\*||Malware was quarantined|
|C:\VTRoot\*||Virus was run in a sandbox|
|C:\ProgramData\Comodo\Firewall Pro\cislogs.sdb||Information on an event in a firewall|
|HKLM\SYSTEM\VritualRoot\*||Information on running in a sandbox|
Far too many of the popular tests mindlessly chose quantity over quality by testing various AM applications against vast numbers of malware samples (many of which could be essentially redundant) without actually running the samples in a test system (where VM naive systems are preferred, of course). Although a sugnature only based tests are amusing, they are hardly real world.Level 3 represents modern protection without any signatures. In such cases, a virus is run in the operating system. It is the most dangerous situation but needed because it shows true effectiveness of protection against and 0-day files – a threat unknown to a developer of protection software.
I cannot agree with you.
Level 3 is responsible for non-signature protection, because at this level, the sample is always running. Before that, malware has a chance to be detected, because it is in the system for about 60 seconds.
I would also add that both F-Secure DeepGuard and Kaspersky System Watcher have signatures too. They are runtime behavior blockers who can have signature updates pertaining to the runtime behavior, like a HIPS with downloadable rules.I did not intend to contradict your post but only make it slightly clearer. I did not say that non-signature protection is excluded from Level 3. I only said that "it is not generally true that Level 3 of this test represents modern protection without any signatures or shows the true effectiveness of protection against fresh/unknown malware." I had in mind that there can still be many signature-based detections in the cloud backend at Level 3 (so I mentioned that you had probably in mind the offline signatures).
So I would use other words to describe this level: Level 3 represents the protection against fresh malware.
- Some AVs can delay the file execution at Level 3 and check the file against the cloud backend (fast signatures, behavior-based detections, detonation in the sandbox, etc). So, in this case, the file is not really executed at Level 3. The analysis and detection are made at the pre-execution stage.
- An example can be Microsoft Defender when the BAFS feature was not triggered before execution. That is what exactly happens in this test. Other examples: Avast CyberCapture, Avast Hardened Mode, Norton Insight, Windows SmartScreen for Explorer, some Defender's ASR rules, Emsisoft Behavior Blocker reputation check, Comodo Firewall File Lookup (before sandboxing), etc.
- From point 1 it also follows that Level 3 can often include detections of already known malware and the file is not obliged to run at this level.
- Some AVs can also analyze and detect the file after it has been executed. Such post-execution features can be triggered by the telemetry sent to the cloud backend (like in the case of Microsoft Defender). Also at this stage, fast signatures can be used.
I intentionally did not use the words "modern protection", because the protection used by Emsisoft and Comodo is rather old (but still very effective for fresh samples).
Of course, Level 3 includes also many behavior-based detections, sandboxing, etc., so it also represents in a great deal the non-signature protection. But to be sure if the detection/block was signature dependent or not, one has to always take a closer look at the AV particular detection.
That can depend on the AV. Anyway, at Level 3 the AVs usually trigger the strongest features because the threat is going to be executed (even if the execution is delayed). Some AVs use only pre-execution features, others can use also post-execution features based on the telemetry of running processes. But, the strong pre-execution features can be triggered also at Levels 1 or 2, if the AV protection is closely integrated with the web browser. An example of such protection can be Defender's BAFS, which works best with Chromium browsers. It can sometimes trigger at Level 1 or 2, even stronger pre-execution protection than at Level 3.What blocking mechanisms are typically being used in Level 3 protection? Is it sandboxing, outbound firewall control, scripting protection, HIPS, or maybe a combination of these, or something else? I'm guessing with Comodo products it's sandboxing of the payload and/or reputation analysis?
Among the non-signature features triggered at Level 3 are HIPS, Local ASR, Local AI, AMSI, Local behavior-based detections, Local sandboxing, and their cloud counterparts (including Deep learning). Some AVs can also trigger reputation-based features or provide Network protection for the executed processes.
Examples of behavior blocking (related to Defender but probably also common for other AVs):
- Credential dumping from LSASS
- Cross-process injection
- Process hollowing
- User Account Control bypass
- Tampering with antivirus (such as disabling it or adding the malware as exclusion)
- Contacting Command and Control (C&C) to download payloads
- Coin mining
- Boot record modification
- Pass-the-hash attacks
- Installation of root certificate
- Exploitation attempt for various vulnerabilities
What blocking mechanisms are typically being used in Level 3 protection? Is it sandboxing, outbound firewall control, scripting protection, HIPS, or maybe a combination of these, or something else? I'm guessing with Comodo products it's sandboxing of the payload and/or
Though ESET detected almost everything upon static scan (level 2 or earlier), some of the detections were at runtime either thanks to the network scanning or looking at temporary files during execution.
Cloud detection implies signature based detection which was not the point of the test, and it is refreshing to view a testing scenario that releases us from the shackles of the typical scan and compare AV philosophy.Was Cloud-delivered protection enabled or disabled in the tested products?
Cloud detection implies signature based detection which was not the point of the test, and it is refreshing to view a testing scenario that releases us from the shackles of the typical scan and compare AV philosophy.