AVLab.pl May 2022: Advanced In The Wild Malware Test (changes based on user suggestions)

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
114
Dear Readers!

Last edition of Advanced In The Wild Malware Test we had a lot discuss about methodology. The primary changes we added are MOTW (mark of the web) feature and change wording (Level 1, Level 2, Level 3 to Post-Launch and Pre-Launch).

In May 2022, we completed the 18th edition of the Advanced In the Wild Malware Test. We carry out this analysis for our readers systematically in order to show the effectiveness of security from various developers in the long run. This time we used 1925 samples of malware.

Tested softwares (always the latest version – our testing system updates antiviruses and signatures once a day):
  • Avast Free Antivirus
  • Avira Antivirus Pro
  • Bitdefender Total Security
  • Comodo Advanced Endpoint Protection (for businesses)
  • Comodo Internet Security
  • Emsisoft Business Security (for businesses)
  • Malwarebytes Premium
  • Mks_vir Endpoint Security (for businesses)
  • Microsoft Defender (Windows 10 with SmartScreen disabled – explanation in article)
  • CatchPulse (formerly known as SecureAPlus Pro)
  • Webroot Antivirus

Please note!

In May we experimented with Microsoft Defender. We disabled SmartScreen in order to verify the true effectiveness of protection against malware without the use of technology that is not part of the antivirus, but the operating system. Microsoft’s results were not satisfactory, because 98,6%.

In July 2022, in the next edition the SmartScreen technology for Microsoft Defender will be enabled in order to compare results. Experience on the example will tell whether the level of protection is significantly higher. We will present the conclusions in the next summary.

New! Level 1 + Level 2 = Pre-launch

Start from May 2022 we change Level 1 and Level 2 are combined into single level: Pre-launch (previously Level 1 or Level 2)

Level 3 remains the same level with new name: Post-launch (previously Level 3)

Reclassification of levels is for marketing to simplify the methodology and make it easier to understand the tests that users have suggested. Thank you!

So the new classification concerns detecting malware samples:
  • Before they are launched in the system (Pre-launch)
  • and samples which are blocked after launch (Post-launch). This is the most dangerous situation but experience shows that such cases require tests in the field.
Please read full comment: Advanced In The Wild Malware Test: We Check How Malware Is Blocked Before And After Being Launched - AVLab Cybersecurity Foundation

Recent Results: Recent Results - AVLab Cybersecurity Foundation
 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top poster
Developer
Well-known
Dec 23, 2014
7,179
Somehow, the Microsoft Defender "Block At First sight feature" does not work at all in this test. It can be seen from the result web page:
1656508835176.png

Normally, the files downloaded from the Internet via Chrome are tested via Defender's Block At First Sight (BAFS) immediately after download (LEVEL 1). I am not sure how the test methodology used in this test might invalidate BAFS. :unsure:
With working BAFS the Defender's results should be similar to the test made in March 2021:

1656510487867.png

It would be good to contact Microsoft to clarify the results.
 
Last edited:

Razza

Level 3
Aug 12, 2014
103
Normally, the files downloaded from the Internet via Chrome are tested via Defender's Block At First Sight (BAFS) immediately after download (LEVEL 1). I am not sure how the test methodology used in this test might invalidate BAFS.

On this test they disabled SmartScreen not sure if Defender need that for BAFS to work, I don't use Defender so don't know how it works under the hood.
 
  • Like
Reactions: Nevi and geminis3

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top poster
Developer
Well-known
Dec 23, 2014
7,179
On this test they disabled SmartScreen not sure if Defender need that for BAFS to work, I don't use Defender so don't know how it works under the hood.
No, SmartScreen is unrelated to BAFS. Furthermore, SmartScreen for Explorer works at the Post-launch level.
If the BAFS worked and Defender has got the result like in the current test, then this would mean that all samples used in the test were totally unknown to Defender. The known samples downloaded from the Internet via Edge or Chrome are detected by BAFS (= cloud signatures + fast path detections) on Level 1. Blocking 98.6% of totally unknown samples would be a great result for any AV that does not use file reputation lookup.
 
Last edited:

ErzCrz

Level 12
Verified
Top poster
Well-known
Aug 19, 2019
587
@Adrian Ścibor Which version of Comodo Internet Security was tested?

EDIT: I only ask as forum Premium version ends in .8012 but website Pro Version where your eventually given the option to stay with free version ends in .8032 but no release notes for the latter. When I run CIS, I use the .8012 offline installer.
 
Last edited:

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top poster
Developer
Well-known
Dec 23, 2014
7,179
It looks like the automated method of downloading samples does not support the Microsoft Defender Block At First Sight feature.

1656540906048.png

When the user downloads manually the sample from the Internet via Edge or Chrome then the file gets MOTW and BAFS is automatically triggered. But, when Firefox is used, the BAFS does not work, even if the file has got MOTW.
This can be seen when downloading the Eicar samples via Edge (Chrome) and Firefox. So, MOTW alone is not sufficient to trigger BAFS.

My guess is that the automated method of downloading the samples in the last few AVLab tests can have a similar issue with BAFS, as the method used by Firefox.
 
Last edited:

ErzCrz

Level 12
Verified
Top poster
Well-known
Aug 19, 2019
587
This has always been a thing with firefox and MD in the past. Not tried it with the MD addon but I can definitely download all I want and it won't get detected until accessing the file.
 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top poster
Developer
Well-known
Dec 23, 2014
7,179
BAFS works perfectly for me with Firefox, according to the BAFS test site: BAFS - Microsoft Defender Testground

Is the BAFS test file detected automatically just after download without opening it? It should be if detected via BAFS.
When I use this testground webpage with Firefox, the test samples are always downloaded without any action from the Defender. When I try to open the sample from Firefox, it is blocked by SmartScreen for Explorer. If I ignore SmartScreen then the sample is sometimes detected by Defender and sometimes it is allowed to run. If I set Highest Cloud Protection Level via ConfigureDefender (restart is required) then BAFS test samples are blocked very often. With default settings, the samples are blocked only sometimes.
The new test sample is created each time after refreshing the website.
 
Last edited:

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
114
Hello :) Let me answer one by one:

Microsoft got the worst results. It seems that Microsoft always performs poorly in your tests. Avira didn't perform well either.

Btw, where is ESET?

ESET was tested in the previous edition. From time to time we change vendors that do not permanently participate in the tests. In July, we are adding two new solutions to the tests. Starting tomorrow, July 1, the next edition starts + Microsoft Defender with SmartScreen enabled (still Chrome browser, but SS works for files at launch regardless of browser). The BAFS is enabled by the way (whole defaults settings except SS).

@Adrian Ścibor Which version of Comodo Internet Security was tested?

EDIT: I only ask as forum Premium version ends in .8012 but website Pro Version where your eventually given the option to stay with free version ends in .8032 but no release notes for the latter. When I run CIS, I use the .8012 offline installer.
CIS Pro version was tested (X.8032). An update is checked every day, so we always test on the latest versions. The latest one is probably .8032.

Where is Sophos?

As above. We don't have a push for Sophos. Maybe that will change if there is demand.
 

mellowtones242

Level 2
Verified
Aug 11, 2018
94
Hello :) Let me answer one by one:



ESET was tested in the previous edition. From time to time we change vendors that do not permanently participate in the tests. In July, we are adding two new solutions to the tests. Starting tomorrow, July 1, the next edition starts + Microsoft Defender with SmartScreen enabled (still Chrome browser, but SS works for files at launch regardless of browser). The BAFS is enabled by the way (whole defaults settings except SS).


CIS Pro version was tested (X.8032). An update is checked every day, so we always test on the latest versions. The latest one is probably .8032.



As above. We don't have a push for Sophos. Maybe that will change if there is demand.

I know MT is more consumer driven but, if possible, I would like to see the following below added to see how they stack up.

Symantec Endpoint Protection
McAfee Endpoint Security
Trend Micro Apex One
Sophos Intercept X
Sentinel One - Singularity Platform
Crowd Strike - Falcon
Eset Protect
CylancePROTECT
VMware Carbon Black Cloud
Cybereason Defense Platform
 
  • Like
Reactions: Adrian Ścibor

South Park

Level 9
Verified
Jun 23, 2018
402
Is the BAFS test file detected automatically just after download without opening it? It should be if detected via BAFS.
When I use this testground webpage with Firefox, the test samples are always downloaded without any action from the Defender. When I try to open the sample from Firefox, it is blocked by SmartScreen for Explorer. If I ignore SmartScreen then the sample is sometimes detected by Defender and sometimes it is allowed to run. If I set Highest Cloud Protection Level via ConfigureDefender (restart is required) then BAFS test samples are blocked very often. With default settings, the samples are blocked only sometimes.
The new test sample is created each time after refreshing the website.
The last two times I tested, it was detected automatically before Firefox completed the download. (I have the cloud protection level set to Highest with C_D.)
 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top poster
Developer
Well-known
Dec 23, 2014
7,179
The last two times I tested, it was detected automatically before Firefox completed the download. (I have the cloud protection level set to Highest with C_D.)
Interesting. Could you retest it on the newest Firefox with Defender default settings (restart required)?
I am curious what setting could make Firefox work differently with BAFS.
 
Last edited:
  • Like
Reactions: Gandalf_The_Grey

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top poster
Developer
Well-known
Dec 23, 2014
7,179
@Adrian Ścibor

I think that the testing methodology for Avast can be improved.
For now, the samples are detected by the Avast signatures, or CyberCapture is triggered. If CyberCapture is unable to confirm that the EXE file with MOTW is safe, then it is always detonated in the Avast cloud sandbox.

1656693297552.png


The problem is that the sandbox analysis can last a few hours and the file is locked until it will be recognized as safe. So many samples are considered by the AVLab testing environment as blocked before the sandbox analysis will be finished. Among 2000 samples there can be several samples that will run in the sandbox but refuse to do malicious things.
I do not know how close to perfection is the Avast cloud sandbox. But, if it would not perfect, the test results will be still perfect.

Post edited/shortened for more clarity.
 
Last edited: