AVLab.pl 19th edition of protection test against malicious software

[Adrian Ścibor]

Hi All! A few changes for the July 2022 edition.In general we have included two new configurations in the analysis: Malwarebytes Nebula for companies with EDR (Endpoint Detection and Response) and Microsoft Defender with the SmartScreen technology enabled. This is the answer to your previous comments...
We have used 2185 malware samples to check the protection against threats in the wild.

We always test the latest software, so we intentionally do not provide version number because the testing system updates them once a day:

• Avast Free Antivirus
• Avira Antivirus Pro
• Emsisoft Business Security (for business)
• G Data Total Security
• Malwarebytes Premium
• Malwarebytes Nebula (for business, new in our test)
• Mks_vir Endpoint Security (for business)
• Microsoft Defender (Windows 10 with SmartScreen enabled)
• CatchPulse (formerly SecureAPlus Pro, check why)
• Webroot Antivirus
• Xcitium – ZeroThreat Advanced (former name after rebranding: Comodo Advanced Endpoint Protection)
• Xcitium Internet Security (formerly Comodo Internet Security)

Additional configuration...

Please read the summary to get more information: 19th Edition Of Protection Test Against Malicious Software - AVLab Cybersecurity Foundation
Recent results: Recent Results - AVLab Cybersecurity Foundation

Since the May edition, as you already know, we have introduced new designations. L1 and L2 have changed to - PRE-LAUNCH. So on, L3 has been renamed to POST-LAUNCH.

In the next months, we will change minor technical things to make everything work better with each other on the backend. It is not out of the question that we will make a process tree of what happened one by one from the launch of malware in the POST-LAUNCH level of analysis. We'll see what we get out of it.

By the way - we will be doing an EDR test in 2022..

 

[Andy Ful]

Hi,

Is the reversed order of "Pre-Launch" and "Post-Launch" intended?

 

[SeriousHoax]

Hi,

Is the reversed order of "Pre-Launch" and "Post-Launch" intended?

View attachment 268731

Looks like a mistake since it's the only product in reverse order.

 

[Andy Ful]

It looks that SmartScreen (in Microsoft Defender test) for explorer works properly when files are run via API Vmware (almost all files are blocked Post-Launch). But still, the Microsoft Defender BAFS is somehow bypassed by this API.

 

[Adrian Ścibor]

Looks like a mistake since it's the only product in reverse order.

Fixed! Thanks

It looks that SmartScreen (in Microsoft Defender test) for explorer works properly when files are run via API Vmware (almost all files are blocked Post-Launch). But still, the Microsoft Defender BAFS is somehow bypassed by this API.

Does this apply to local www/IP addresses?

 

[ForgottenSeer 95367]

I do not look at these tests any longer because the results are always the same: 100% or very, very close to it. The only differentiation between the products is pre- and post-launch. Other than that (which does not matter much), they all pass at a very high rate.

@Adrian Ścibor

Tests of fileless, scripts, banking, key\screen-logging, advanced ransomware, etc - like the ones you've done in the past - are much more indicative and informative as to the true protection capabilities of a security product. I understand that vendors do not want to pay for such tests and these tests require much set-up on your part.

 

[Andy Ful]

Fixed! Thanks

Does this apply to local www/IP addresses?

It applies to the files downloaded from the Internet (so not local adresses):

 

[Adrian Ścibor]

I do not look at these tests any longer because the results are always the same: 100% or very, very close to it. The only differentiation between the products is pre- and post-launch. Other than that (which does not matter much), they all pass at a very high rate.

@Adrian Ścibor

Tests of fileless, scripts, banking, key\screen-logging, advanced ransomware, etc - like the ones you've done in the past - are much more indicative and informative as to the true protection capabilities of a security product. I understand that vendors do not want to pay for such tests and these tests require much set-up on your part.

We published the online banking test earlier this year 2022, and there will still be an EDR test this year. In addition to the Advanced In The Wild Malware Test series, we may do 1 or 2 manual tests throughout the year. This is all that our time allows, as AVLab also carries out other educational tasks, not just tests. Sorry to hear you think that or we also need to think about funding the tests from the community. It is not excluded that in 2023 we will do a new series of tests for home users banking protection or others.

 

[ForgottenSeer 95367]

We published the online banking test earlier this year 2022, and there will still be an EDR test this year. In addition to the Advanced In The Wild Malware Test series, we may do 1 or 2 manual tests throughout the year. This is all that our time allows, as AVLab also carries out other educational tasks, not just tests. Sorry to hear you think that or we also need to think about funding the tests from the community. It is not excluded that in 2023 we will do a new series of tests for home users banking protection or others.

@Adrian Ścibor

• I do not think funding should come from the community.
• I think vendors are cheap, game the test lab system, and really do not want any lab to tear their products apart in advanced tests (they would not pay the 50,000+ Euros for such deep, advanced testing anyway).
• I hold the work that AVLab.pl does in high regard; I think many others think and feel the same.
• AVLab does some rather unique tests that no other testing organization performs.
• Tests of signatures are not interesting because in this day almost all of the vendors are at 98+% success.

 

[ForgottenSeer 95367]

We published the online banking test earlier this year 2022

@Adrian Ścibor

2022 online banking test:

Test of security solutions in blocking attacks on Internet banking

 

[ForgottenSeer 95367]

@Adrian Ścibor

Why do you not test SpyShelter Firewall more often?

You once posted (in comments on your website) that SpyShelter Firewall offers the highest\ultimate level of protection for any security software.

 

[Jan Willy]

@Adrian Ścibor

Why do you not test SpyShelter Firewall more often?

You once posted (in comments on your website) that SpyShelter Firewall offers the highest\ultimate level of protection for any security software.

Is it still in development? No updates since 12-10-2021.

 

[Adrian Ścibor]

@Adrian Ścibor

Why do you not test SpyShelter Firewall more often?

You once posted (in comments on your website) that SpyShelter Firewall offers the highest\ultimate level of protection for any security software.

This test is all about anti-malware solutions. The vendor does not want to go in the direction of these tests as I remember. However as for other tests - they are not interested or something. But if someone refuses once or twice at your ask, it's not nice to you and ask a third time and a fourth.

 

[ForgottenSeer 95367]

This test is all about anti-malware solutions. The vendor does not want to go in the direction of these tests as I remember. However as for other tests - they are not interested or something. But if someone refuses once or twice at your ask, it's not nice to you and ask a third time and a fourth.

@Adrian Ścibor

I understand DatPol does not want to spend the money for tests.

You still think SpyShelter Firewall is very strong protection. Among the best, correct?

 

[Adrian Ścibor]

@Adrian Ścibor

I understand DatPol does not want to spend the money for tests.

You still think SpyShelter Firewall is very strong protection. Among the best, correct?

Really, it's been a long time since I've certified it. At least a two or 2-3-4 years ago.

 

[Andy Ful]

Nowadays, the main problem with testing comes from malware morphism (polymorphism, metamorphism, etc.).
Due to malware morphism, around 90% of all malware samples are detected only once on a single computer and never seen again (unique to one PC). This is possible because most malware samples are short living and are quickly replaced by morphed variants. From this, it follows that most malware signatures are for "dead" malware samples. Such signatures cannot protect anyone, except for some cases related to malware hunting (when the signature is created before the malware could attack someone).

When performing a test with few-day-old samples, most of them (probably more than 75%) can be already detected by signatures and 90% of them were "dead". So, we have a strange situation. For many AVs, more than one-half of the detections can be for "dead" samples. These detections can be very important for the AV scorings in the test, but unimportant for protection in the wild. In fact, the test scorings can significantly depend on how quickly the AV vendors added the signatures for "dead" samples and cannot reflect real-life protection. From several tests, we know that Trend Micro (and some other vendors) do not care much about signature completeness. When one of MT members asked the TM staff about poor results in Malware Protection tests performed by AV-Comparatives, the answer was that the missed samples were not important for users' protection. As we know, Trend Micro is a top AV in Real-World tests.

The above issue can be overcome by testing the malware samples in real-time.

Windows Defender Antivirus cloud protection service: Advanced real-time defense against never-before-seen malware - Microsoft Security Blog

For cybercriminals, speed is the name of the game. It takes newly released malware an average of just four hours to achieve its goal—steal financial information, extort money, or cause widespread damage. In a recent report, the Federal Trade Commission (FTC) said that cybercriminals will use...

www.microsoft.com

2023 Threat Report | OpenText Cybersecurity

Cyber threats are changing the way we do everything. Learn more in the 2023 OpenText Cybersecurity Threat Report

www.webroot.com

 

[mellowtones242]

We published the online banking test earlier this year 2022, and there will still be an EDR test this year. In addition to the Advanced In The Wild Malware Test series, we may do 1 or 2 manual tests throughout the year. This is all that our time allows, as AVLab also carries out other educational tasks, not just tests. Sorry to hear you think that or we also need to think about funding the tests from the community. It is not excluded that in 2023 we will do a new series of tests for home users banking protection or others.

Do you have vender list for EDR test?

 

[Adrian Ścibor]

Do you have vender list for EDR test?

Not quite yet. We will definitely take vendors who are in contact with us, because it's important to have access to the console quickly. It can be a problem with some companies to get access to the product in time. So there is nothing to reveal yet. This will be the first edition of EDR - what is most important - it will not be a traditional test of security. Rather Red and Blue Teaming simulator of attack scenarios and the summary, conclusions.

I don't rule out that, we relied on two features for comparison: attack visibility (so-called attack context with alert) in the admin console and telemetry. This is because a lack of visibility or telemetry could mean that the product failed to meet protection expectations, or did so too late. This can manifest itself by encrypting parts of the vendor before the protection software can stop the escalation of a cyber attack. In addition, EDR's automation should makes it an effective tool for large and small organizations of any skill level.

 

[mellowtones242]

Not quite yet. We will definitely take vendors who are in contact with us, because it's important to have access to the console quickly. It can be a problem with some companies to get access to the product in time. So there is nothing to reveal yet. This will be the first edition of EDR - what is most important - it will not be a traditional test of security. Rather Red and Blue Teaming simulator of attack scenarios and the summary, conclusions.

I don't rule out that, we relied on two features for comparison: attack visibility (so-called attack context with alert) in the admin console and telemetry. This is because a lack of visibility or telemetry could mean that the product failed to meet protection expectations, or did so too late. This can manifest itself by encrypting parts of the vendor before the protection software can stop the escalation of a cyber attack. In addition, EDR's automation should makes it an effective tool for large and small organizations of any skill level.

Sounds interesting, looking forward to your findings!

 

[Andy Ful]

When testing the real-life protection, the correct method would be a real-time test, for example:

1. Download the potential threat, detonate it in the test sandbox, and confirm that it is malicious.
2. Next, check if the same sample is still present on the web. If not then skip it. If there is a different potential threat that replaced the older one, then download it and go to point 3 without checking the sample in the sandbox.
3. Run the test with this sample against AVs.
4. If the sample from point 2 was not checked in the sandbox, then do it. Skip the test for this sample if it is not malicious.

Point 2 is important to avoid the detection of "dead" samples. I think that such a test can be automated similarly to AVLab tests.