AVLab.pl September 2023 - The Advanced In-The-Wild Malware Test (multi-layer protection is the key to effective protection against malware)

[Adrian Ścibor]

Dear All,

We have been already published a new publication and the results of the Advanced In-The-Wild Malware Test.

Publication to reading and conclusions: Multi-layer Protection – The Key To Effective Protection Against Malware – Conclusions From The Test In September 2023 » AVLab Cybersecurity Foundation

Recent Results in details: https://avlab.pl/en/recent-results/

Awards & Product Cards in details: Awards » AVLab Cybersecurity Foundation

And some changes have been added in this edition. Let's check our transparency webpage: Changelog » AVLab Cybersecurity Foundation

 

[harlan4096]

Getting this in results link:

 

[Adrian Ścibor]

Getting this in results link:

View attachment 279412

Indeed. It was. Thanks! It has been fixed.

 

[Moonhorse]

So sophos first, not expected that
F secure second - expected
G-data took forever? Its eggcellent but its very sloooow

 

[simmerskool]

so far just skimmed results, all tested blocked everything, but did not see MS Defender?

 

[Anthony Qian]

On the website (Recent Results » AVLab Cybersecurity Foundation):

Sophos
Blocked: 343/343
Total: 99.36%

Trendmicro
Blocked: 343/343
Total: 99.36%

Did I miss something? Why aren't the total protection rates of the above two products 100%?

 

[Xeno1234]

On the website (Recent Results » AVLab Cybersecurity Foundation):

Sophos
Blocked: 343/343
Total: 99.36%

Trendmicro
Blocked: 343/343
Total: 99.36%

Did I miss something? Why aren't the total protection rates of the above two products 100%?

Same with Bitdefender. Blocked 343/343 but it wasnt 100%.

 

[Adrian Ścibor]

In the signature-based and cloud query tests, most of the time all score 100.


MS Defender protection is dismal when it does not detect by signature or cloud (includes SmartScreen and Intelligent Security Graph). Check its ability to protect against banking trojans in 2022 and 2019. @Adrian Ścibor 's team test methodology in these is more real-world scenario than the MRG Effitas banking simulator tests, but both provide something for those interested to consider.

Indeed

@Adrian Ścibor you should explain that isolated browser sessions (e.g. Safe Browsing and Safe Pay) are solid protection only when the user does not copy-paste identity and authentication credentials from outside the isolated browser. Lots of readers do not understand how banking trojans work. They do not know that in banking protection testing a banking trojan or simulator is running on the real system during the entire test procedure. So anything performed outside of the isolated browser session, such as typing or copy-paste, can be captured by the banking trojan and transmitted to the attacker.

We are preparing a new publication of the banking modules test, and we plan to publish in November 2023.

On the website (Recent Results » AVLab Cybersecurity Foundation):

Sophos
Blocked: 343/343
Total: 99.36%

Trendmicro
Blocked: 343/343
Total: 99.36%

Did I miss something? Why aren't the total protection rates of the above two products 100%?

Our apologies to all. It was editorial mistake and it is fixed. In this edition, we have 100% detection-rate (combined protection) for all Vendors.

 

[Bill W]

@Adrian Ścibor you should explain that isolated browser sessions (e.g. Safe Browsing and Safe Pay) are solid protection only when the user does not copy-paste identity and authentication credentials from outside the isolated browser. Lots of readers do not understand how banking trojans work. They do not know that in banking protection testing a banking trojan or simulator is running on the real system during the entire test procedure. So anything performed outside of the isolated browser session, such as typing or copy-paste, can be captured by the banking trojan and transmitted to the attacker.

I think this is a hugely important comment. At the end the day, if malware can capture our keystrokes and clipboard info within an isolated browser, then we need to ask more from our AV solution.

 

[ddave]

So sophos first, not expected that
F secure second - expected
G-data took forever? Its eggcellent but its very sloooow

If I am not making a huge mistake, I think you are misinterpreting the value of "RT"

Remediation Time Average (RT): The time expressed in seconds from the introduction of malware into the system by a browser, through the launch to detecting and resolving security incident. Occurs only at the POST-Launch level.

So it's prefectly fine that G Data has "not available" as RT result and this value doesn't lead back to some sort of ranking.

 

[Moonhorse]

View attachment 279445

If I am not making a huge mistake, I think you are misinterpreting the value of "RT"

Remediation Time Average (RT): The time expressed in seconds from the introduction of malware into the system by a browser, through the launch to detecting and resolving security incident. Occurs only at the POST-Launch level.

So it's prefectly fine that G Data has "not available" as RT result and this value doesn't lead back to some sort of ranking.

what i meant is that g-data takes forever to remove malware it finds , its on next level in slowness, but guess youre right here

 

[Jonny Quest]

Not capture keystrokes or clipboard contents from within the isolated browser. It can be captured outside of the isolated browser. Isolated browsers do not permit browser extensions. That means no password manager extension and therefore no way to access user names and passwords within the isolated browser. So users copy their credentials from their non-isolated browser or local password manager, and then paste it into the isolated browser. At the moment of copy, a malware can capture the information.

F-Secure's Vault is permitted in the Secure Browsing & Banking window, as well as 1Password. Since it's contained in that safe environment, those are both safe to use, correct?

 

[Adrian Ścibor]

View attachment 279445

If I am not making a huge mistake, I think you are misinterpreting the value of "RT"

Remediation Time Average (RT): The time expressed in seconds from the introduction of malware into the system by a browser, through the launch to detecting and resolving security incident. Occurs only at the POST-Launch level.

So it's prefectly fine that G Data has "not available" as RT result and this value doesn't lead back to some sort of ranking.

No, because all threats were blocked in pre_lanuch, so according to the legend:

Remediation Time Average (RT): The time expressed in seconds from the introduction of malware into the system by a browser, through the launch to detecting and resolving security incident. Occurs only at the POST-Launch level.

Thus, the RT score at the pre_launch level is always 0 seconds. If you think we should count the average from this: RT pre + RT post - let us know. According to your thinking, the result of Remediation Time for G Data should be 0s.

 

[Jonny Quest]

F-Secure is not an isolated browser in the way that Bitdefender Safepay or Kaspersky Safemoney are. The F-Secure banking mode only blocks network connections to unapproved destinations. It also disconnects all untrusted applications from the internet.

It is always best practice to capture any identity and authentication credentials from within the banking mode or isolated browser session. There is a reason why Bitdefender and Kaspersky provide a virtual keyboard inside Safepay/Safemoney, however very, very few users actually utilize these features.

If I recall correctly Avast permitted a password manager. I did not know about F-Secure's. Bitdefender did have Wallet for use inside Safepay, but that has been done away with.

It is not that the vendors cannot implement protections outside of the isolated browser environment. They do. But some did not get it right and this was demonstrated during @Adrian Ścibor 's banking tests in 2019. Kaspersky, Bitdefender both failed to protect clipboard data outside Safepay/Safemoney. Many others such as F-Secure failed also.

Because nobody really explains how banking trojans work, users think that they are fully protected when typing or copy-pasta with a security program installed. With a banking trojan, it is running on the system. With Man-in-the-Browser, there is malicious code inside the webpage itself. Safepay and Safemoney are not going to stop Man-in-the-Browser. They might protect credentials as long as the user fastidiously does not do careless things such as switching back and forth to the desktop and typing/copying objects to the clipboard.

You can read the AVLab banking test reports for more details. @Adrian Ścibor will be the first to tell you that you cannot interpret their test results as a sweeping generalization to extend to every and all banking trojans or other malicious code. What that means is that the user has to be careful and fastidious about the typing and copying.

Thank you for the thoughtful reply, Greyton, it was helpful

Overview Of Techniques And Attacks In Windows 11 » AVLab Cybersecurity Foundation

Question - What does the name of the "Advanced In-The-Wild Malware Test" mean?

avlab.pl

 

[simmerskool]

If one of those approved applications that is still permitted to connect outbound is hijacked or absued by banking malware, oh well you're bacon is cooked

no offense, but "absued" is not in the English dictionary. I thought I'd learn a new (to me) word, but alas, I guess it's a typo. I do those tooo.

 

[Sandbox Breaker]

100% again for all. Why is then when I get zero days ... Most of these solutions fail. Vendor Agnosticism seems to serve well. Totally unrealistic. That's like saying body armor stopped all rounds... Even ones not rated for Yeah your level 3 vest stopped a .50.

 

[Adrian Ścibor]

100% again for all. Why is then when I get zero days ... Most of these solutions fail. Vendor Agnosticism seems to serve well. Totally unrealistic. That's like saying body armor stopped all rounds... Even ones not rated for Yeah your level 3 vest stopped a .50.

As you wrote - when it comes to 0-day, there may not be a perfect solution. However, 0-day is not a targeted APT attack. You can write a zero-day in a few lines in ChatGTP and there is no finesse here. This is why multi-layered protection is important. Look out for solutions that isolate unknown samples if you have doubts about 100% protection.

If you look at the malware comparison CSV file, maybe less than 10% of the samples used in the test are 0-day for the MKS_VIR engine, which we use for initial sample analysis. This is the signature: INFECTION: HEUR.RoundKick.W.

edit: So it's not like we're only using known threats. We use in-the-wild threats, and they are what they are. A 0-day for vendor X is not necessarily a 0-day for Y.

With 0-day file (not url) will always be detected at post_lanuch level, there is no other way. I am not writing here about specific protection, such as the threat emulation while downloading in the browser (Zone Alarm / Check Point?)

 

[Sandbox Breaker]

As you wrote - when it comes to 0-day, there may not be a perfect solution. However, 0-day is not a targeted APT attack. You can write a zero-day in a few lines in ChatGTP and there is no finesse here. This is why multi-layered protection is important. Look out for solutions that isolate unknown samples if you have doubts about 100% protection.

If you look at the malware comparison CSV file, maybe less than 10% of the samples used in the test are 0-day for the MKS_VIR engine, which we use for initial sample analysis. This is the signature: INFECTION: HEUR.RoundKick.W.

edit: So it's not like we're only using known threats. We use in-the-wild threats, and they are what they are. A 0-day for vendor X is not necessarily a 0-day for Y.

With 0-day file (not url) will always be detected at post_lanuch level, there is no other way. I am not writing here about specific protection, such as the threat emulation while downloading in the browser (Zone Alarm / Check Point?)

I always speak about layers and human expertise. Whether USB or Web vector or hand on key attacks. I and many other see this 100% as 100% snakeoil. I respect your work and mean no disrespect.