AVLab.pl 19th edition of protection test against malicious software

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
124
Hi All! A few changes for the July 2022 edition.In general we have included two new configurations in the analysis: Malwarebytes Nebula for companies with EDR (Endpoint Detection and Response) and Microsoft Defender with the SmartScreen technology enabled. This is the answer to your previous comments...
We have used 2185 malware samples to check the protection against threats in the wild.

We always test the latest software, so we intentionally do not provide version number because the testing system updates them once a day:
  • Avast Free Antivirus
  • Avira Antivirus Pro
  • Emsisoft Business Security (for business)
  • G Data Total Security
  • Malwarebytes Premium
  • Malwarebytes Nebula (for business, new in our test)
  • Mks_vir Endpoint Security (for business)
  • Microsoft Defender (Windows 10 with SmartScreen enabled)
  • CatchPulse (formerly SecureAPlus Pro, check why)
  • Webroot Antivirus
  • Xcitium – ZeroThreat Advanced (former name after rebranding: Comodo Advanced Endpoint Protection)
  • Xcitium Internet Security (formerly Comodo Internet Security)
Additional configuration...

Please read the summary to get more information: 19th Edition Of Protection Test Against Malicious Software - AVLab Cybersecurity Foundation
Recent results: Recent Results - AVLab Cybersecurity Foundation

Since the May edition, as you already know, we have introduced new designations. L1 and L2 have changed to - PRE-LAUNCH. So on, L3 has been renamed to POST-LAUNCH.

In the next months, we will change minor technical things to make everything work better with each other on the backend. It is not out of the question that we will make a process tree of what happened one by one from the launch of malware in the POST-LAUNCH level of analysis. We'll see what we get out of it.

By the way - we will be doing an EDR test in 2022..
 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
7,368
Hi,

Is the reversed order of "Pre-Launch" and "Post-Launch" intended?

1661289387833.png
 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
7,368
It looks that SmartScreen (in Microsoft Defender test) for explorer works properly when files are run via API Vmware (almost all files are blocked Post-Launch). But still, the Microsoft Defender BAFS is somehow bypassed by this API.
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
124
Looks like a mistake since it's the only product in reverse order.
Fixed! Thanks
It looks that SmartScreen (in Microsoft Defender test) for explorer works properly when files are run via API Vmware (almost all files are blocked Post-Launch). But still, the Microsoft Defender BAFS is somehow bypassed by this API.
Does this apply to local www/IP addresses?
 
F

ForgottenSeer 95367

I do not look at these tests any longer because the results are always the same: 100% or very, very close to it. The only differentiation between the products is pre- and post-launch. Other than that (which does not matter much), they all pass at a very high rate.

@Adrian Ścibor

Tests of fileless, scripts, banking, key\screen-logging, advanced ransomware, etc - like the ones you've done in the past - are much more indicative and informative as to the true protection capabilities of a security product. I understand that vendors do not want to pay for such tests and these tests require much set-up on your part.
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
124
I do not look at these tests any longer because the results are always the same: 100% or very, very close to it. The only differentiation between the products is pre- and post-launch. Other than that (which does not matter much), they all pass at a very high rate.

@Adrian Ścibor

Tests of fileless, scripts, banking, key\screen-logging, advanced ransomware, etc - like the ones you've done in the past - are much more indicative and informative as to the true protection capabilities of a security product. I understand that vendors do not want to pay for such tests and these tests require much set-up on your part.
We published the online banking test earlier this year 2022, and there will still be an EDR test this year. In addition to the Advanced In The Wild Malware Test series, we may do 1 or 2 manual tests throughout the year. This is all that our time allows, as AVLab also carries out other educational tasks, not just tests. Sorry to hear you think that or we also need to think about funding the tests from the community. It is not excluded that in 2023 we will do a new series of tests for home users banking protection or others.
 
F

ForgottenSeer 95367

We published the online banking test earlier this year 2022, and there will still be an EDR test this year. In addition to the Advanced In The Wild Malware Test series, we may do 1 or 2 manual tests throughout the year. This is all that our time allows, as AVLab also carries out other educational tasks, not just tests. Sorry to hear you think that or we also need to think about funding the tests from the community. It is not excluded that in 2023 we will do a new series of tests for home users banking protection or others.
@Adrian Ścibor
  • I do not think funding should come from the community.
  • I think vendors are cheap, game the test lab system, and really do not want any lab to tear their products apart in advanced tests (they would not pay the 50,000+ Euros for such deep, advanced testing anyway).
  • I hold the work that AVLab.pl does in high regard; I think many others think and feel the same.
  • AVLab does some rather unique tests that no other testing organization performs.
  • Tests of signatures are not interesting because in this day almost all of the vendors are at 98+% success.
 
F

ForgottenSeer 95367

@Adrian Ścibor

Why do you not test SpyShelter Firewall more often?

You once posted (in comments on your website) that SpyShelter Firewall offers the highest\ultimate level of protection for any security software.
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
124
@Adrian Ścibor

Why do you not test SpyShelter Firewall more often?

You once posted (in comments on your website) that SpyShelter Firewall offers the highest\ultimate level of protection for any security software.
This test is all about anti-malware solutions. The vendor does not want to go in the direction of these tests as I remember. However as for other tests - they are not interested or something. But if someone refuses once or twice at your ask, it's not nice to you and ask a third time and a fourth.
 
F

ForgottenSeer 95367

This test is all about anti-malware solutions. The vendor does not want to go in the direction of these tests as I remember. However as for other tests - they are not interested or something. But if someone refuses once or twice at your ask, it's not nice to you and ask a third time and a fourth.
@Adrian Ścibor

I understand DatPol does not want to spend the money for tests.

You still think SpyShelter Firewall is very strong protection. Among the best, correct?
 
  • Like
Reactions: Jan Willy

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
7,368
Nowadays, the main problem with testing comes from malware morphism (polymorphism, metamorphism, etc.).
Due to malware morphism, around 90% of all malware samples are detected only once on a single computer and never seen again (unique to one PC). This is possible because most malware samples are short living and are quickly replaced by morphed variants. From this, it follows that most malware signatures are for "dead" malware samples. Such signatures cannot protect anyone, except for some cases related to malware hunting (when the signature is created before the malware could attack someone).

When performing a test with few-day-old samples, most of them (probably more than 75%) can be already detected by signatures and 90% of them were "dead". So, we have a strange situation. For many AVs, more than one-half of the detections can be for "dead" samples. These detections can be very important for the AV scorings in the test, but unimportant for protection in the wild. In fact, the test scorings can significantly depend on how quickly the AV vendors added the signatures for "dead" samples and cannot reflect real-life protection. From several tests, we know that Trend Micro (and some other vendors) do not care much about signature completeness. When one of MT members asked the TM staff about poor results in Malware Protection tests performed by AV-Comparatives, the answer was that the missed samples were not important for users' protection. As we know, Trend Micro is a top AV in Real-World tests.

The above issue can be overcome by testing the malware samples in real-time.

 
Last edited:

mellowtones242

Level 2
Verified
Aug 11, 2018
95
We published the online banking test earlier this year 2022, and there will still be an EDR test this year. In addition to the Advanced In The Wild Malware Test series, we may do 1 or 2 manual tests throughout the year. This is all that our time allows, as AVLab also carries out other educational tasks, not just tests. Sorry to hear you think that or we also need to think about funding the tests from the community. It is not excluded that in 2023 we will do a new series of tests for home users banking protection or others.
Do you have vender list for EDR test?
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
124
Do you have vender list for EDR test?
Not quite yet. We will definitely take vendors who are in contact with us, because it's important to have access to the console quickly. It can be a problem with some companies to get access to the product in time. So there is nothing to reveal yet. This will be the first edition of EDR - what is most important - it will not be a traditional test of security. Rather Red and Blue Teaming simulator of attack scenarios and the summary, conclusions.

I don't rule out that, we relied on two features for comparison: attack visibility (so-called attack context with alert) in the admin console and telemetry. This is because a lack of visibility or telemetry could mean that the product failed to meet protection expectations, or did so too late. This can manifest itself by encrypting parts of the vendor before the protection software can stop the escalation of a cyber attack. In addition, EDR's automation should makes it an effective tool for large and small organizations of any skill level.
 

mellowtones242

Level 2
Verified
Aug 11, 2018
95
Not quite yet. We will definitely take vendors who are in contact with us, because it's important to have access to the console quickly. It can be a problem with some companies to get access to the product in time. So there is nothing to reveal yet. This will be the first edition of EDR - what is most important - it will not be a traditional test of security. Rather Red and Blue Teaming simulator of attack scenarios and the summary, conclusions.

I don't rule out that, we relied on two features for comparison: attack visibility (so-called attack context with alert) in the admin console and telemetry. This is because a lack of visibility or telemetry could mean that the product failed to meet protection expectations, or did so too late. This can manifest itself by encrypting parts of the vendor before the protection software can stop the escalation of a cyber attack. In addition, EDR's automation should makes it an effective tool for large and small organizations of any skill level.
Sounds interesting, looking forward to your findings!
 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
7,368
When testing the real-life protection, the correct method would be a real-time test, for example:
  1. Download the potential threat, detonate it in the test sandbox, and confirm that it is malicious.
  2. Next, check if the same sample is still present on the web. If not then skip it. If there is a different potential threat that replaced the older one, then download it and go to point 3 without checking the sample in the sandbox.
  3. Run the test with this sample against AVs.
  4. If the sample from point 2 was not checked in the sandbox, then do it. Skip the test for this sample if it is not malicious.
Point 2 is important to avoid the detection of "dead" samples. I think that such a test can be automated similarly to AVLab tests.
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top