AVLab.pl Modern protection without signatures – comparison test on real threats (Advanced In The Wild Malware Test)

[Adrian Ścibor]

Greetings People! I know there are more important things going on right now, but all of us - as people from Poland and you, as a worldwide community, need to take care of our digital security. You know this, but your colleagues or family are not technical people. Therefore, as always, every other month we started with protection test of modern antivirus solutions on so-called samples in the wild that can be found in various areas of the Internet, also captured by our honeypots. This test is completely automated, so it can run continuously 24 hours a day.

In January 2022, we checked the following applications to protect PCs:

• Avast Free Antivirus
• Avira Antivirus Pro
• Comodo Advanced Endpoint Protection
• Comodo Internet Security
• Emsisoft Business Security
• F-Secure Total (new)
• Malwarebytes Premium (new)
• Microsoft Defender
• SecureAPlus Pro
• Webroot Antivirus

Unlike other testing laboratories, our tests are fully transparent, so we provide a complete list of virus samples. Key information and test results are available at Recent Results - Advanced In The Wild Malware Test.

Summary of January edition of the test​

Notice the Level 3 of analysis because it shows real protection against 0-day samples. But beware! Certain antiviruses of the next generation intentionally do not have protection in a browser (Level 1). Sometimes they do not have traditional protection based on signatures (Level 1 and Level 2), so without proper interpretation, such tests could favor other protection solutions. Not ours!

In our tests, we do not award negative or positive points for early blocking of threats. Simply put, security products must stop a threat in any way – designed by a developer. The final result is whether or not this has been achieved

The so-called Level 1 shows early blocking of threats in a browser or on a hard drive.

If this fails, the next is Level 2: a virus is scanned by the antivirus based on signatures when moving from X to Y folder. Obviously, only if such protection exists. In this test, there are many test cases when samples have not been tested by developers yet, so the next level of analysis is crucial.

Level 3 represents modern protection without any signatures. In such cases, a virus is run in the operating system. It is the most dangerous situation but needed because it shows true effectiveness of protection against and 0-day files – a threat unknown to a developer of protection software.

Post Scriptum. We are gratefull you for voting for AVLab in the AV-Comparatives survey, as we were ranked 4th in 2020. In 2021 we are already 3rd (behind AV-Test and AV-C).

1. Our summaries: AVLab among the leaders of trusted testing laboratories in Europe and worldwide (summary of the whole 2021) - AVLab.pl
2. AV-C post: AV-Comparatives Security Survey 2022 published

 

[ForgottenSeer 92963]

Although it is fun to see how an AV protects without signatures, it is against their own testing philosophy

"Simply put, security products must stop a threat in any way – designed by a developer"

Taking one element out of the mix (without signatures) does not comply with the "designed by a (the AV's) developer" criterium.

A pity Kaspersky is not included in the test, Eugene's System watcher has a strong reputation of stopping threats

 

[Andy Ful]

...
Level 3 represents modern protection without any signatures. In such cases, a virus is run in the operating system. It is the most dangerous situation but needed because it shows true effectiveness of protection against and 0-day files – a threat unknown to a developer of protection software.

Congrats on the 3rd place.

It seems that the scenario of this particular test is as below:

1. The system is already infected.
2. The malware is trying to download/drop/execute the payload.

So, only the actions against point 2 are tested. The testing methodology intentionally ignores the initial web-based threats and is focused on the protection against the payloads. It is an interesting approach similar to Malware Protection testing. The difference between this test and Malware Protection tests made by AV-Test or AV-Comparatives is that the samples are (on average) a few days old in AVLab tests and a few weeks old in AV-Test/AV-Comparatives tests. Also, the Malware Protection tests are used as a kind of reference (subsidiary) tests to the Real-World tests.

Anyway, it is not generally true that Level 3 of this test represents modern protection without any signatures or shows the true effectiveness of protection against fresh/unknown malware. I think that the author had in mind only the local offline signatures. Also, the true effectiveness of protection is narrowed here only to EXE payloads (scripting methods are skipped).

Such a scenario is related to the business networks where the payloads are often applied to the clean machines via lateral movement. The efficient protection requires at least some kind of Network Protection, or a behavior blocker independent of MOTW, or some Advanced Threat Protection features. The solutions that do not have such additional protection (like Defender free on defaults) should not be used in the business environment.
In the case of Defender, one should use in SMBs the Microsoft Defender for Endpoint or another paid subscription, or activate the Defender's advanced features.

 

[marcopaone]

What a Disaster Windows defender. :/

 

[Andy Ful]

What a Disaster Windows defender. :/

True if you have meant Defender free on default settings in the business networks.
It seems that Microsoft and Avast use very different methods to convince small businesses to buy paid AV versions. Avast uses Ads and Microsoft uses PowerShell.

The AVLab test simply proves that Microsoft is right when insisting that in businesses one should use Defender paid versions (like Microsoft Defender for Endpoint) or activate the advanced Defender features.

Also, the results of Emsisoft and Malwarebytes are impressive (no Network protection).

Post shortened and edited.

 

[Andy Ful]

Adrian Ścibor,​

Although the testing methodology is interesting, it also has got some questionable points.

1. The samples are a few days old (on average).
2. The samples are mostly the *.exe files (some files had a spoofed DLL extension on VT).

As we can see from the test results most AVs have no problem with detecting such payloads by cloud backend (fast signatures and behavior-based modules).
But, in the real attacks on businesses, the payloads will be much fresher. In many cases, the payloads are changed after a few hours, so they may be often unknown also in the cloud. Furthermore, in the compromised business networks the scripts are often used before applying the final EXE payload. Detecting suspicious/malicious scripts will prevent many such attacks. That is why the real protection against the threats in the already compromised business network can be different from the results of this test.

Edit.
The Malware Hub is a good place to see the protection against the very fresh samples (including scripts).

 

[Adrian Ścibor]

Although it is fun to see how an AV protects without signatures, it is against their own testing philosophy

Taking one element out of the mix (without signatures) does not comply with the "designed by a (the AV's) developer" criterium.

A pity Kaspersky is not included in the test, Eugene's System watcher has a strong reputation of stopping threats

Please,do not interpret wrong the methodology. Basically we use default settings for protection. An exception is to enable silent mode or automatic blocking, if any, so that the warning message from AV is automatically marked as quarantined.

@Andy Ful We can put malware samples from MalwareTips HUB if you wish cooperate. As I know is needed a permission to access this section. That's why I can't download anything myself.

 

[Nightwalker]

Malwarebytes is truly impressing these days, I always thought that since version 4.0 it was much better in real world scenarios than people gave credit for, but now it is showing good to stellar results in AV testings too.

 

[Andy Ful]

...
@Andy Ful We can put malware samples from MalwareTips HUB if you wish cooperate. As I know is needed a permission to access this section. That's why I can't download anything myself.

Sharing the samples could be probably interesting for both AVLab and MH.
Unfortunately, some AVs have got postinfection detection features. So, maybe this could be possible if AVLab and MH would agree to test monthly different AVs.

Edit.
You can PM @harlan4096, @upnorth, or @silversurfer for more information.

 

[ForgottenSeer 92963]

Please,do not interpret wrong the methodology. Basically we use default settings for protection. An exception is to enable silent mode or automatic blocking, if any, so that the warning message from AV is automatically marked as quarantined.

There is no interpretation. Your website/pdf states literally: "Simply put, security products must stop a threat in any way – designed by a developer". A developer of an AV probably did not design his product to be used without signatures.

Maybe something gets lost in the polish english - english -dutch translation, because your answer confuses me even more (Basically we use default settings for protection). Which AntiVirus has its signatures disabled by default?

 

[Moonhorse]

Expected results for comodo , its rock solid... well defender got destroyed by this test, but does not mean its bad at overall

Thanks for posting to mt

 

[Andy Ful]

There is no interpretation. Your website/pdf states literally: "Simply put, security products must stop a threat in any way – designed by a developer". A developer of an AV probably did not design his product to be used without signatures.

In this test, the malware can be stopped in any way which means that it can be stopped on Levels 1, 2, or 3. If the malware is not stopped on Levels 1, 2 but only on Level 3, it means that it was stopped without using offline signatures.
As the author mentioned in the OP:

But beware! Certain antiviruses of the next generation intentionally do not have protection in a browser (Level 1). Sometimes they do not have traditional protection based on signatures (Level 1 and Level 2), so without proper interpretation, such tests could favor other protection solutions. Not ours!

 

[upnorth]

Please,do not interpret wrong the methodology. Basically we use default settings for protection. An exception is to enable silent mode or automatic blocking, if any, so that the warning message from AV is automatically marked as quarantined.

@Andy Ful We can put malware samples from MalwareTips HUB if you wish cooperate. As I know is needed a permission to access this section. That's why I can't download anything myself.

@Andy Ful is Not a Hub member, so those kind of Hub questions has to be done with me and @harlan4096 . Please read here for more information:

https://malwaretips.com/threads/get-involved-in-the-malware-hub.75479/#post-956766

 

[ForgottenSeer 92963]

In this test, the malware can be stopped in any way which means that it can be stopped on Levels 1, 2, or 3. If the malware is not stopped on Levels 1, 2 but only on Level 3, it means that it was stopped without using offline signatures.

The title of this thread said "Modern protection without signatures", that is why I thought the AV's were tested without signatures. Are you telling me that the AV-products were tested with their signatures enabled?

 

[oldschool]

Are you telling me that the AV-products were tested with their signatures enabled?

Yes.

 

[Andy Ful]

The title of this thread said "Modern protection without signatures", that is why I thought the AV's were tested without signatures. Are you telling me that the AV-products were tested with their signatures enabled?

I think so - there is no information in the testing methodology about skipping signatures. The phrase "Modern protection without signatures" is probably related to Level 3. Generally, the AVLab tests show how the protection against the EXE payloads is divided between Levels 1,2, and 3. See also my posts:
https://malwaretips.com/threads/mod...d-in-the-wild-malware-test.112630/post-977487
https://malwaretips.com/threads/mod...d-in-the-wild-malware-test.112630/post-977472

 

[MacDefender]

Yeah I agree, I think the title might be causing alarm because of the way it's phrased or translated. What I'm reading is that it more pertains to "level 3", when both layers of static scanning have missed the sample and it's treated more like what the Malware Hub calls "dynamic" on-execution testing, as opposed to static scanning. I don't see a mention of a signature/realtime disabled (e.g. "bonus behavior blocker test").

I'm a little surprised about the Defender results unless it was done offline or something. Most tests of Defender score a lot better in static detection of malware samples. Defender (the customer version) doesn't seem to do much in terms of dynamic behavior blocking currently, its form of dynamic protection appears to be sandbox detonation and holding off on execution for sandbox evaluation, and I am pretty sure the enterprise version offers better protection in this regard compared to the customer version.

 

[Andy Ful]

...
I'm a little surprised about the Defender results unless it was done offline or something. Most tests of Defender score a lot better in static detection of malware samples. ..

You should not be disappointed. Most of the tested samples are not the initial malware, but payloads. So, this test does not show how are the chances to be infected in the home environment. It is more appropriate for showing how successful would be the lateral movement in the already compromised business network. In the case of Defender, it is clear that on default settings it is not the best solution. This is not a surprise because the default settings are intended by Microsoft for the home environment (no lateral movement).
Similar results can be seen in the older tests made by MRG Effitas (years 2018 and 2019) where Defender was tested on default settings. In the more recent MRG Effitas tests, Defender uses advanced settings (including ASR rules), so the results are very good:

MRG Effitas tests 360° Assessment & Certification.

Missed samples in the tests Q1-Q4 of the year 2021 (In the wild 360, PUA, Financial, Ransomware + Exploit & Fileless)
Symantec..............= 2 ....... + 1e
Avast......................= 4 .......+ 1e
Malwarebytes........= 2 ......+ 5e
Bitdefender............= 6.5 ....+ 0e
Microsoft...............= 6.5 ....+ 4e
Sophos..................=*11 .....+ *0e (* included an averaged result for Q4 2021)
ESET ......................= 13 ... + 0e
F-Secure.................= 17.5 ..+ 0e (many missed PUA)
Avira.......................= 64.5 ..+ 4e
Trend Micro...........= 114 .. + 9e


Exploit & Fileless - missed samples in the last 2 years (Q4 2019 - Q4 2021)
Bitdefender..........................0e
Eset .....................................0e
F-Secure .............................4e
Microsoft............................6e
Symantec ...........................6e
Avast .................................7.5e
Trend Micro........................18e
Avira ...................................16e

Due to a small number of Exploit & Fileless samples, I used a period of about 2 years (9 tests).

 

[ErzCrz]

I enjoy these tests. Not sure why I even bother using signatures with Comodo these days but glad containment was reliable as always

 

[Andy Ful]

I enjoy these tests. Not sure why I even bother using signatures with Comodo these days but glad containment was reliable as always

Yes, in such tests (PE executables) Comodo will always be among the top solutions.