AVLab.pl Summary of the March edition of tests on virus samples from the Internet

[upnorth]

I wanted the methodology to be uncomplicated and easy to understand.

Please continue with that as otherwise I wouldn't be surprised you will start loose viewers instead of gain. There will always be some that crave and demand more and extra much how they want things to be created and shared, but I would say the genuine majority is the exact opposite. Otherwise these type of tests will always only be possible to be interpreted by very few.

 

[Andy Ful]

Very, very complicated and almost impossible to implement.
...
Changing L1, L2, L3 naming to the proposed one will be very difficult and expensive to implement.

We will lose ourselves in the details. I wanted the methodology to be uncomplicated and easy to understand. Meanwhile, we talk about additional complicating factors.

I am not sure what you mean by my proposition. My proposition is to use only two levels: "pre-execution stage" (L1+L2) and "on(post)-execution stage" (L3). This will be more understandable for readers. From a few threads about AVLab testing, it follows that readers can have serious problems with understanding the meaning of Levels 1,2, and 3. These levels are strictly connected with the testing methodology without a clear connection with the AV proactive features. One could recover some connections by analyzing the logs, detection names, event IDs, etc. but this would be time-consuming and would probably require some help from the AV vendors. The only AV testing lab that tries to do it (in a limited way) is MRG Effitas:

My proposition is only a loose suggestion. There is nothing wrong with Levels 1,2, and 3, although the current description of these levels can be misguiding for some readers.

 

[Andy Ful]

To be clear, this is what I propose:

LEVEL A: pre-execution level
The malware has been detected before it could be executed.

I would not use the term "browser level". Technically, this term is correct for the AVLab test because all samples are downloaded via a web browser. Anyway, many AVs can detect threats on Level 1 even if the malware was not downloaded via the web browser.

LEVEL 2:
The system level, i.e. a virus has been downloaded, but it hasn’t been allowed to run.

This level probably includes the events that start as Level 1 and end when the sample is moved to another location. The sample is not allowed to run because the analysis has been finished with a delay, and this is not related to the process of moving the sample to another location.
This level could have some value when the samples were downloaded to the network shares and next moved to the local disk. The Level 2 events are very rare in AVLab tests. I propose to join this level with level 1 to get LEVEL A.

LEVEL B: on(post)-execution level
The malware could be executed and has been detected/blocked "on-execution" or "post-execution".

I would not use the term "analysis level", because a similar analysis can be done in many cases on Level A, especially for the files downloaded from the Internet.

As I said, this is only a loose proposition.

 

[SeriousHoax]

I am not an expert in Eset matters, but this can be interesting to you:


The above explanation clearly states that LiveGuard should work at Level 1. The file types can be configured in Eset:

You're right. But my answer is based on my experience with ESET and that is, threats known to ESET's web protection/sig+heur will get blocked at browser level (Level 1) if the file is not too big and LiveGuard gets trigger at Level 2 or Level 3.

About Emsisoft, Emsisoft don't scan file on access or creation. It only detects threats either by signature or behavior blocker after you try to execute it.

• Scan level– The slider control allows you to balance the File Guard’s scan level between best performance and best protection as follows:
  • Default – Scans programs when they are started. This option has the least effect on the performance of your system while still ensuring that Malware is prevented from executing. Inactive malware may remain undetected until you run a manual scan. This is the recommended setting.
  • Thorough – Scans all files when they are created or modified, for example when a file is downloaded or copied onto your computer from a USB stick. Since files are not an immediate threat unless they are executed, this option may find inactive malware sooner, but your machine is still protected with the default setting. If you do not experience performance issues, this is still a reasonable setting to use.
  • Paranoid – Scans all files when they are read by any program so that simply selecting a file is sufficient to cause it to be scanned. On a typical computer, there are usually thousands of files being read in the background every minute, so this option naturally slows down the overall performance of your computer quite dramatically. We don’t recommend making use of it, but keep it available for those who want to be absolutely sure everything gets immediately detected without delays. This can be useful for temporary use in situations where Emsisoft Anti-Malware is installed on an already-infected computer, to aid in cleanup.

 

[Adrian Ścibor]

@Andy Ful
In fact, the results from March 2022 are already ready under your idea. Just combine pre-execution (L1+L2) and post-execution (L3).

Instead of L1, L2, L3 we have in the results e.g. for Avast (no changes):
PRE-EXECUTION: 16.49%
POST-EXECUTION: 83.51%

Malwarebytes Premium (no changes)
PRE-EXECUTION: 3.37%
POST-EXECUTION: 96.63%

Webroot Antivirus (some changes)
PRE-EXECUTION: (L1 62.55% + L2 2.24%)
POST-EXECUTION: 96.63%
FAIL: 0,05%

---

Referring to MRG Effitas - Auto Level and Behaviour Blocked.

Auto Block: note that we configure the software to automatically delete, move to quarantine, without asking the user. Does Auto Block apply to L1+L2? I suppose so...

Behavior Block: does this apply to L3 as a post-execution scenario?

---

Dear Readers - do you think the name change is relevant here to take advantage of POST and PRE Execution?

 

[Andy Ful]

Referring to MRG Effitas - Auto Level and Behaviour Blocked.

Auto Block: note that we configure the software to automatically delete, move to quarantine, without asking the user. Does Auto Block apply to L1+L2? I suppose so...

Behavior Block: does this apply to L3 as a post-execution scenario?
...

It is very probable, because Auto block + Behavior block = 100% blocked events (at the first part of the test; the second part is after 24 hours). If so, then the name "Behavior Block" used by MRG Effitas includes also detection by signatures in the cloud (the term behavior is slightly abused). It would also mean that MRG Effitas uses very similar levels as AV Lab.
The naming in the testing area is not well established and can be sometimes misguiding. For example, the event when the user tries to execute something and the execution is suspended for file analysis, is not truly a post-execution event (the code is not yet executed) and also not truly a pre-execution event. Probably a better name would be the "on-execution" for such events.

 

[Adrian Ścibor]

---

---

It is very probable, because Auto block + Behavior block = 100% blocked events (at the first part of the test; the second part is after 24 hours). If so, then the name "Behavior Block" used by MRG Effitas includes also detection by signatures in the cloud (the term behavior is slightly abused). It would also mean that MRG Effitas uses very similar levels as AV Lab.
The naming in the testing area is not well established and can be sometimes misguiding. For example, the event when the user tries to execute something and the execution is suspended for file analysis, is not truly a post-execution event (the code is not yet executed) and also not truly a pre-execution event. Probably a better name would be the "on-execution" for such events.

I understand that in this case it would be useful to get the opinion of other users, what they think about the "L1, L2, L3" name change to:
L1+L2 = new name something like -> Blocked Before Execute
L3 = new name somethink like -> On Execute.

Currently name: Recent Results - AVLab Cybersecurity Foundation

LEVEL 1:
The browser level, i.e. a virus has been stopped before or right after it has been downloaded.

LEVEL 2:
The system level, i.e. a virus has been downloaded, but it hasn’t been allowed to run.

LEVEL 3:
The analysis level, i.e. a virus has been run and blocked by a tested product.

FAIL:
The failure, i.e. a virus hasn’t been blocked and it has infected a system.

 

[Andy Ful]

From the MRG Effitas report:

Live URL test is conducted by the following procedure.
...
• The test case is marked as “Blocked” if either the security application blocks the URL where the malicious binary was located, or the security application blocks the malicious binary whilst it was being downloaded to the machine.
• The test case is marked as “Behaviour Blocked” if the security application blocks the malicious binary when it is executed and either automatically blocks it or postpones its execution and warns the user that the file is malicious and awaiting user input.
...

These descriptions exactly describe Levels A and B proposed by me (if we skip phishing/malicious URL check like in AVLab tests).

It is funny, that I did not realize (and did not notice) that the "Behaviour Blocked" events in MRG Effitas tests are equal to on(post)-execution events, so they also include malware detected by signatures in the cloud. I was misguided by the word "Behaviour" and I thought that MRG Effitas put some effort to recognize the events blocked/detected behaviorally. This proves that correct naming can be important.

Before realizing it, I have treated the change L1,L2,L3 ---> LA: pre-execution , LB: on(post)- execution, as a minor improvement to make the test results simpler and clearer. Now, I think that this renaming can prevent misunderstanding of the results.

 

[Andy Ful]

The interpretation of the results is slightly simpler when we focus on the file execution:

Emsisoft: Files downloaded by the web browser are not checked automatically by the cloud backend until they are executed. Emsisoft uses file reputation lookup for EXE files - such protection can hardly miss any EXE sample.

Eset: Files downloaded by the web browser are checked automatically by the cloud backend before they are executed. Eset can detonate in cloud sandbox the EXE files downloaded via the web browser. Such protection can hardly miss any EXE sample downloaded directly via the web browser.



Post edited.

 

[Andy Ful]

Avast: Files downloaded by the web browser are not checked automatically by the cloud backend until they are executed. The 16,49% detection is related to the local pre-execution features (mostly offline signatures). When the sample is executed, Avast triggers the cloud backend - the unrecognized & suspicious EXE files downloaded via the web browser can be detonated in the cloud sandbox (Hardened Mode enabled). This can hardly miss any EXE sample downloaded directly via the web browser, except very rare samples with strong anti-sandbox features.

Avira: The files downloaded via web browser are scanned by Avira Browser Safety (web browser extension, part of cloud backend) that could detect 76,49% of samples. The full cloud backend is triggered on file execution which improves the protection by 23,22%.
The cloud backend does not include detonation in the cloud sandbox (available in Avira Prime).

Comodo: Files downloaded by the web browser are not checked automatically by the cloud backend until they are executed. The 0.34% detection is probably related to offline signatures. The cloud backend is triggered on file execution. The execution is postponed until the analysis is finished. The files recognized as malicious are blocked, benign files are allowed, and unknown files are auto-sandboxed in the local Comodo Sandbox. This can hardly miss any EXE sample, except for rare samples that can escape the default sandbox (the default sandbox is not set to the strongest option).

 

[itman71]

Since AVLab testing enhancements are being discussed here, I'll add my comments.

Eset offers two consumer suite versions, Internet Security and Smart Security Premium. The later which was tested by AVLab is the product offering cloud scanning capability that Eset advertises will protect you against 0-day malware. Also. the cost of ESSP is significantly higher than EIS. The problem is how does a consumer validate that ESSP cloud scanning is effective as claimed? I also assume this same situation exists with other AV vendors product offerings.

I therefore propose another test level be added. That is a test performed with the test server Internet connection disabled. This test would parallel the periodic Malware Protection test performed by A-V Comparatives. Obviously, the test with the Internet connection disabled would be performed first to prevent any cloud scanning detection's influencing its detection rates. Finally, it is imperative that malware test samples are representative of what the vendor claims to detect in its cloud scanner. In Eset's case that would be anything downloaded via browser or e-mail. Also, the samples should include executable's, scripts, etc..

 

[Adrian Ścibor]

@itman71 Interesting opinion, thank you. We will consider your comment. It will be easy to implement. Technically nothing will change except turning off the internet connection / freezing virus database for some time.

 

[itman71]

@itman71 Interesting opinion, thank you. We will consider your comment. It will be easy to implement. Technically nothing will change except turning off the internet connection / freezing virus database for some time.

Thanks for your consideration of my suggestion.

To Eset users even if the above was implemented, the comparison between EIS and ESSP in regards to ESSP cloud scanning capability is not as straight forward as it seems. This is because EIS also has an Eset cloud component to it.

Referencing the latest A-V Comparatives Malware Protection test: Malware Protection Test March 2022 , Eset had identical off-line and on-line detection scores of 96.1%. However, its on-line protection score was 99.7%. This results in a deviation of 3.6%. Assumed is your seeing the result of Eset accessing it's cloud to get the latest blacklist detection's and signatures.

Therefore, any off-line testing of ESSP in regards to LiveGuard cloud scanning effectiveness must also factor in loss of cloud blacklist detections and latest signatures. A rough approximation would be if ESSP off-line/on-line testing resulted in a protection deviation greater than 3.6%, then one could conclude that difference was due to the ESSP LiveGuard cloud scanning factor.

 

[Adrian Ścibor]

Since AVLab testing enhancements are being discussed here, I'll add my comments.

Eset offers two consumer suite versions, Internet Security and Smart Security Premium. The later which was tested by AVLab is the product offering cloud scanning capability that Eset advertises will protect you against 0-day malware. Also. the cost of ESSP is significantly higher than EIS. The problem is how does a consumer validate that ESSP cloud scanning is effective as claimed? I also assume this same situation exists with other AV vendors product offerings.

I therefore propose another test level be added. That is a test performed with the test server Internet connection disabled. This test would parallel the periodic Malware Protection test performed by A-V Comparatives. Obviously, the test with the Internet connection disabled would be performed first to prevent any cloud scanning detection's influencing its detection rates. Finally, it is imperative that malware test samples are representative of what the vendor claims to detect in its cloud scanner. In Eset's case that would be anything downloaded via browser or e-mail. Also, the samples should include executable's, scripts, etc..

Hi, I'm back to your idea. I wonder what if we turn off the Internet, and some Vendors use their cloud scanning to realize protection against unknown samples? They may explain that we have excluded an important aspect of protection...

 

[upnorth]

if we turn off the Internet, and some Vendors use their cloud scanning to realize protection against unknown samples? They may explain that we have excluded an important aspect of protection...

True, and best would be to actually reach out directly to some of those major AV vendors researchers and developers, so you can also get the correct full version of why.

 

[itman71]

Hi, I'm back to your idea. I wonder what if we turn off the Internet, and some Vendors use their cloud scanning to realize protection against unknown samples? They may explain that we have excluded an important aspect of protection...

No need to disable the Internet connection.

Best way to test Eset's LiveGuard effectives would be to test both ESSP and EIS. EIS doesn't contain the LiveGuard component. Note: ESSP will only use LiveGuard after local heuristic scanning is completed. As such, your malware samples need to contain .exe's, scripts, etc.. not detected by existing signature. Also assumed is these samples would show as not detected by EIS.

-EDIT- Also and again, EIS needs to be tested first prior to ESSP. ESSP will create a blacklist detection for anything flagged by LiveGuard processing that in turn, would be detected by EIS scanning.