AVLab.pl Summary of the March edition of tests on virus samples from the Internet

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
101
Dear MalwareTips Users,

In March 2022 took place already 17th edition of the Advanced In The Wild Malware Test in which we regularly verify software to protect devices. This time we have used 2050 unique malware samples to check the effectiveness of detection and blocking malicious software. This malware number is always different as testing brings up what is rejected and what is not. Hovewer started from this month we have made more changes:

1. Improved - primary website / redesigned "AVLab Foundation :: CyberSecurity In Poland" to better compatibility with mobile devices and user experience.
2. Added (attached file) - more detailed percentage results due to the React.js application on the backend. The results for March 2022 are available here Recent Results - AVLab Cybersecurity Foundation
3. Improved - Award website "https://avlab.pl/en/awards/" / redesigned as well,to better reward vendors who participate in testing. We added additional marketing information like "Product of the year".
4. Improved methodology started from May 2022 we add MOTW file mark (thanks @Andy Ful)

The latest results can be found in the URL above and in the coming months we have to work on better Yara rules.
 

Attachments

  • app 1.png
    app 1.png
    322.2 KB · Views: 97
  • app 2.png
    app 2.png
    164.7 KB · Views: 96
Last edited:

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
101
Hi Adrian Ścibor, I see that ESET Smart Security Premium caught everything in Level 1. I want to know if all these were blocked automatically by ESET or the LiveGuard feature which sends files downloaded from the internet to their cloud sandbox for analysis was required?

Unfouratelly, we do not store such of logs if malware is not executed on Windows. Here it is important to clarify at which level (1, 2 or 3) LiveGuard works according to our methodology?

Blocked 100% at Level 1 means that malware could not be executed (L3) or moved from the c:\...\Download path to another location (L2). So, it is about protection for known samples for Eset software / cloud / collective protection.
 

Andy Ful

Level 81
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
7,008
...
Blocked 100% at Level 1 means that malware could not be executed (L3) or moved from the c:\...\Download path to another location (L2). So, it is about protection for known samples for Eset software / cloud / collective protection.

Although this reasoning is probably true for most AVs, it cannot be generalized to all AVs, especially for samples with MOTW.
It is possible that Microsoft Defender is a special case, but it can trigger protection for new malware on Level 1.

  • In many cases, this process can reduce the response time for new malware from hours to seconds.

It would be interesting to know if some other AVs use similar protection (Eset LiveGuard?).
It is probable that we should make the term "known samples" more precise. Can new samples be also known samples?
Can Levels 1 and 2 be related to the pre-execution stage and Level 3 to post-execution?

I am not sure if the below description of Level 3 is generally correct:
LEVEL 3:
The analysis level, i.e. a virus has been run and blocked by a tested product.
We know that in some cases the sample can be analyzed on Level 1 similarly to Level 3. Maybe the "analysis level" should be changed to "post-execution level"?
 

SeriousHoax

Level 43
Verified
Top poster
Well-known
Mar 16, 2019
3,202
Unfouratelly, we do not store such of logs if malware is not executed on Windows. Here it is important to clarify at which level (1, 2 or 3) LiveGuard works according to our methodology?

Blocked 100% at Level 1 means that malware could not be executed (L3) or moved from the c:\...\Download path to another location (L2). So, it is about protection for known samples for Eset software / cloud / collective protection.
My bad. Level 1 means all were blocked at browser level, so it means nothing was able to download on the disk. In that case, LiveGuard won't trigger, as it triggers on Level 2 or Level 3 only.
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
101
It would be interesting to know if some other AVs use similar protection (Eset LiveGuard?).
It is probable that we should make the term "known samples" more precise. Can new samples be also known samples?

This shows that Level 1 is the correct name - specifies well, but does not break it down into its constituent parts - complicating this, breaking it down, in a general sense mean the same thing at the end of the day - the malware was blocked early, before it was executed.

One would have to ask someone familiar with the Eset product if LiveGuard is some sort of blocker of unknown samples during download until analysis from the cloud or reputation information is received.

Can Levels 1 and 2 be related to the pre-execution stage and Level 3 to post-execution?
Yes.
I am not sure if the below description of Level 3 is generally correct:

We know that in some cases the sample can be analyzed on Level 1 similarly to Level 3. Maybe the "analysis level" should be changed to "post-execution level"?

Basically, sample can be analyzed at Level 1 automatically in the cloud initiated by AV product, but it is NEVER open by the user.

"Executing (Level 3)" is what distinguishes Level 1,2 from Level 3. So yes - coulbd be named as "post-execution level". Formally named by us as "analysis level (post-execute)".
 

Andy Ful

Level 81
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
7,008
Is it correct to say that Level 3 detection is a proactive one (behavior blocker/HIPS and so on) ?

No, I am afraid. Proactive is usually related to the detection of new, unknown threats, without the need for a specific signature. This can be done on all levels.

This is a generic term for technologies used by an anti-malware product to detect new, unknown threats, without the need for a specific signature.

They include heuristics, behavioural analysis, application allowlisting and exploit detection.

https://encyclopedia.kaspersky.com/glossary/proactive-detection/

Behavior blockers (not the same as behavior-based detections) and HIPS are related to post-execution protection.
 

Andy Ful

Level 81
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
7,008
My bad. Level 1 means all were blocked at browser level, so it means nothing was able to download on the disk. In that case, LiveGuard won't trigger, as it triggers on Level 2 or Level 3 only.

You were not "bad". Level 1 includes also successfully downloaded samples. This level is related to AV integration with the web browser. But it can also be integrated with general downloads (like Norton Download Insight) or when files are dropped to disk.

LEVEL 1:
The browser level, i.e. a virus has been stopped before or right after it has been downloaded.

I am not sure if Eset's LiveGuard is integrated with the web browser's downloads, but the AVLab test results strongly suggest that LiveGuard works on Level 1.
 
Last edited:

SeriousHoax

Level 43
Verified
Top poster
Well-known
Mar 16, 2019
3,202
You were not "bad". Level 1 includes also successfully downloaded samples. This level is related to AV integration with the web browser.
I see. But it's still unlikely that LiveGuard was triggered. In my testing nowadays, it seems to activate on execution most of the time. Also, verdicts obtained from LiveGuard takes 5 minutes or more. If something like that happened, AVLab probably would have kept a log of that. ESET also has HTTPS scanning, so it has the ability to intercept threats early in the browser.
 

Andy Ful

Level 81
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
7,008
I see. But it's still unlikely that LiveGuard was triggered. In my testing nowadays, it seems to activate on execution most of the time. Also, verdicts obtained from LiveGuard takes 5 minutes or more. If something like that happened, AVLab probably would have kept a log of that. ESET also has HTTPS scanning, so it has the ability to intercept threats early in the browser.

I am not an expert in Eset matters, but this can be interesting to you:

LiveGuard employs technology originally built for businesses to safeguard their diverse networks from both known and never-before-seen types of threats. Specifically, this is accomplished through a cloud sandbox that pulls suspicious files – whether those downloaded by web browsers or by email services like Microsoft Outlook and Mozilla Thunderbird, or extracted from archives, or found on USB drives plugged into devices – to a secure cloud platform for analysis.


The above explanation clearly states that LiveGuard should work at Level 1. The file types can be configured in Eset:

Automatic submission of suspicious samples

These samples will also be sent to ESET if the detection engine does not detect them. For example, samples that nearly missed the detection or one of the ESET Smart Security Premium protection modules consider these samples suspicious or behaving unclear (the default maximum sample size is 64MB).

•Executables – Includes executable files like .exe, .dll, .sys.

•Archives – Includes archive filetypes like .zip, .rar, .7z, .arch, .arj, .bzip, .gzip, .ace, .arc, .cab.

•Scripts – Includes script filetypes like .bat, .cmd, .hta, .js, .vbs, .ps1.

•Other – Includes filetypes like .jar, .reg, .msi, .sfw, .lnk.

•Possible Spam emails – Allows sending possible spam parts or whole possible spam emails with attachments to ESET for further analysis. Enabling this option improves global spam detection, including improvements to future spam detection.

•Delete executables, archives, scripts, other samples and possible spam emails from ESET's servers – Defines when to delete samples submitted for analysis by ESET LiveGuard.

•Documents – Includes Microsoft Office or PDF documents with or without active content.

•Delete documents from ESET's servers – Defines when to delete documents submitted for analysis by ESET LiveGuard.
 
Last edited:

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
101
Is it correct to say that Level 3 detection is a proactive one (behavior blocker/HIPS and so on) ?

Yes - you can see this especially with Comodo or Emsisoft. In the database we have clearly that for example for Comodo all threats are blocked thanks to Comodo Sandbox (based on antivirus indicators). For Emsisoft a large part is only blocked after post-execution. Or SecureAgePlus (CatchPlus formally launched recently) - here the blocking based on vendor rules works actively.

Yes - if the product has the features you mentioned then they are particularly useful in Level 3.

Here you go -> a few Comodo's indicators in AVLab methodology:

C:\ProgramData\Comodo\Cis\Quarantine
C:\VTRoot
C:\ProgramData\Comodo\Firewall Pro\cislogs.sdb
HKLM\SYSTEM\VritualRoot
C:\Program Files\COMODO\COMODO Internet Security\virtkiosk.exe
C:\Program Files\COMODO\COMODO Internet Security\cmdvirth.exe
C:\ProgramData\Comodo\Cis\tempscrpt\

Let see on database as example - screen shoot:
same regex: /HKLM\\SYSTEM\\VritualRoot/gi
It means that malware has been sandboxed by Comodo. Definitely proactive behavior.
 

Attachments

  • comodo database dump.png
    comodo database dump.png
    40.6 KB · Views: 42

Andy Ful

Level 81
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
7,008
Is it correct to say that Level 3 detection is a proactive one (behavior blocker/HIPS and so on) ?

It seems that you have got two different answers (Yes and No). The problem was with the meaning of "and so on".

@Adrian Ścibor answered that Level 3 detections/blocks (like behavior blocker/HIPS) are included in proactive detections. His answer is correct if your "so on" can mean "post-execution and non-signature detections". But, it is not generally true. In some cases the file is executed (Level 3) and next detected by the signature in the cloud, so it is not a proactive detection.

I answered that the category of Level 3 detections can be much smaller than proactive detections, so Level 3 detections are not the same as many proactive detections. I did not mention that many Level 3 detections (like behavior blocker/HIPS) are proactive ones.:)(y)
 
Last edited:

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
101
I answered that the category of Level 3 detections can be much smaller than proactive detections, so Level 3 detections are not the same as many proactive detections. I did not mention that many Level 3 detections (like behavior blocker/HIPS) are proactive ones.:)(y)

I'd have to think about it, but as a quick aside, are you sure waiting for a response from the cloud isn't proactive protection? How would you like to distinguish which file security response from the cloud is proactive and which is not when it comes to post-execution scenario?
 

Andy Ful

Level 81
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
7,008
I'd have to think about it, but as a quick aside, are you sure waiting for a response from the cloud isn't proactive protection? How would you like to distinguish which file security response from the cloud is proactive and which is not when it comes to post-execution scenario?

I think that waiting for a response from the cloud (longer than 0.01 sec) can strongly suggest proactive detection at any level (L1, L2, or L3). The problem can be how to measure this in the test. There can be random delays related to the Internet connection, cloud workload, etc. Also splitting the pre-execution detections into L1 and L2 does not help. It is possible that most of the L2 events start as L1 but end with a delay when the sample is copied. If so, then most L2 detections could be proactive, but still many proactive detections would be included in L1.
Furthermore, in some cases, the behavior-based detections can be done in a shorter time than 0.01 sec. It would be hard to distinguish signature detection in the cloud from behavior-based detection. It seems that the proper way would be getting the information directly from the detection name used by the concrete AV for the concrete sample. In some cases, additional information from the logs can be sufficient to recognize the proactive detection (like in the case of Comodo sandboxing). In the case of Defender, some proactive detections/blocks can be recognized by Event IDs (ASR rules have got ID = 1121).
 
Last edited:
  • Like
Reactions: Correlate

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
101
I think that waiting for a response from the cloud (longer than 0.01 sec) can strongly suggest proactive detection at any level (L1, L2, or L3). The problem can be how to measure this in the test. There can be random delays related to the Internet connection, cloud workload, etc. Also splitting the pre-execution detections into L1 and L2 does not help. It is possible that most of the L2 events start as L1 but end with a delay when the sample is copied. If so, then most L2 detections could be proactive, but still many proactive detections would be included in L1.
Furthermore, in some cases, the behavior-based detections can be done in a shorter time than 0.01 sec. It would be hard to distinguish signature detection in the cloud from behavior-based detection. It seems that the proper way would be getting the information directly from the detection name used by the concrete AV for the concrete sample. In some cases, additional information from the logs can be sufficient to recognize the proactive detection (like in the case of Comodo sandboxing). In the case of Defender, some proactive detections/blocks can be recognized by Event IDs (ASR rules have got ID = 1121).
Very, very complicated and almost impossible to implement.

First - some AVs have encrypted logs, so it is impossible to read those logs. In such cases we rely on our own observations, indicators. It will not be possible to extract malware classification.

2. Malware classification - crazy complicated to automate: what and when should be considered as proactive detection. Each vendor may use different classification.

3. As you mentioned - random delays in internet connection. Possible network issues in occasional test cases.

4. And probably other problems that would be during implements.

Changing L1, L2, L3 naming to the proposed one will be very difficult and expensive to implement.

We will lose ourselves in the details. I wanted the methodology to be uncomplicated and easy to understand. Meanwhile, we talk about additional complicating factors.