AVLab.pl September 2023 - The Advanced In-The-Wild Malware Test (multi-layer protection is the key to effective protection against malware)

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
171
Dear All,

We have been already published a new publication and the results of the Advanced In-The-Wild Malware Test.

Publication to reading and conclusions: Multi-layer Protection – The Key To Effective Protection Against Malware – Conclusions From The Test In September 2023 » AVLab Cybersecurity Foundation

Recent Results in details: https://avlab.pl/en/recent-results/

Awards & Product Cards in details: Awards » AVLab Cybersecurity Foundation

And some changes have been added in this edition. Let's check our transparency webpage: Changelog » AVLab Cybersecurity Foundation
 

Attachments

  • september 2023 summary and remediation time.png
    september 2023 summary and remediation time.png
    325.9 KB · Views: 176
  • RECENT RESULTS 2023 SEPTEMBER obrazek  wyrozniajacy.png
    RECENT RESULTS 2023 SEPTEMBER obrazek wyrozniajacy.png
    508.5 KB · Views: 174
  • september 2023 in numbers.png
    september 2023 in numbers.png
    720.4 KB · Views: 132

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
171
In the signature-based and cloud query tests, most of the time all score 100.


MS Defender protection is dismal when it does not detect by signature or cloud (includes SmartScreen and Intelligent Security Graph). Check its ability to protect against banking trojans in 2022 and 2019. @Adrian Ścibor 's team test methodology in these is more real-world scenario than the MRG Effitas banking simulator tests, but both provide something for those interested to consider.
Indeed :)

@Adrian Ścibor you should explain that isolated browser sessions (e.g. Safe Browsing and Safe Pay) are solid protection only when the user does not copy-paste identity and authentication credentials from outside the isolated browser. Lots of readers do not understand how banking trojans work. They do not know that in banking protection testing a banking trojan or simulator is running on the real system during the entire test procedure. So anything performed outside of the isolated browser session, such as typing or copy-paste, can be captured by the banking trojan and transmitted to the attacker.
We are preparing a new publication of the banking modules test, and we plan to publish in November 2023.

On the website (Recent Results » AVLab Cybersecurity Foundation):

Sophos
Blocked: 343/343
Total: 99.36%

Trendmicro
Blocked: 343/343
Total: 99.36%

Did I miss something? Why aren't the total protection rates of the above two products 100%?
Our apologies to all. It was editorial mistake and it is fixed. In this edition, we have 100% detection-rate (combined protection) for all Vendors.
 

Bill W

Level 1
Verified
Mar 30, 2013
29
@Adrian Ścibor you should explain that isolated browser sessions (e.g. Safe Browsing and Safe Pay) are solid protection only when the user does not copy-paste identity and authentication credentials from outside the isolated browser. Lots of readers do not understand how banking trojans work. They do not know that in banking protection testing a banking trojan or simulator is running on the real system during the entire test procedure. So anything performed outside of the isolated browser session, such as typing or copy-paste, can be captured by the banking trojan and transmitted to the attacker.
I think this is a hugely important comment. At the end the day, if malware can capture our keystrokes and clipboard info within an isolated browser, then we need to ask more from our AV solution.
 
  • Like
Reactions: [correlate]

ddave

Level 2
Verified
Nov 17, 2014
86
So sophos first, not expected that
F secure second - expected
G-data took forever? Its eggcellent but its very sloooow
Screenshot 2023-10-29 at 10-09-54 Recent Results » AVLab Cybersecurity Foundation.png


If I am not making a huge mistake, I think you are misinterpreting the value of "RT"

Remediation Time Average (RT): The time expressed in seconds from the introduction of malware into the system by a browser, through the launch to detecting and resolving security incident. Occurs only at the POST-Launch level.

So it's prefectly fine that G Data has "not available" as RT result and this value doesn't lead back to some sort of ranking.
 

Moonhorse

Level 36
Verified
Top Poster
Content Creator
Well-known
May 29, 2018
2,599
View attachment 279445

If I am not making a huge mistake, I think you are misinterpreting the value of "RT"

Remediation Time Average (RT): The time expressed in seconds from the introduction of malware into the system by a browser, through the launch to detecting and resolving security incident. Occurs only at the POST-Launch level.

So it's prefectly fine that G Data has "not available" as RT result and this value doesn't lead back to some sort of ranking.
what i meant is that g-data takes forever to remove malware it finds , its on next level in slowness, but guess youre right here
 

Jonny Quest

Level 13
Mar 2, 2023
628
Not capture keystrokes or clipboard contents from within the isolated browser. It can be captured outside of the isolated browser. Isolated browsers do not permit browser extensions. That means no password manager extension and therefore no way to access user names and passwords within the isolated browser. So users copy their credentials from their non-isolated browser or local password manager, and then paste it into the isolated browser. At the moment of copy, a malware can capture the information.
F-Secure's Vault is permitted in the Secure Browsing & Banking window, as well as 1Password. Since it's contained in that safe environment, those are both safe to use, correct?
 
Last edited:

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
171
View attachment 279445

If I am not making a huge mistake, I think you are misinterpreting the value of "RT"

Remediation Time Average (RT): The time expressed in seconds from the introduction of malware into the system by a browser, through the launch to detecting and resolving security incident. Occurs only at the POST-Launch level.

So it's prefectly fine that G Data has "not available" as RT result and this value doesn't lead back to some sort of ranking.
No, because all threats were blocked in pre_lanuch, so according to the legend:

Remediation Time Average (RT): The time expressed in seconds from the introduction of malware into the system by a browser, through the launch to detecting and resolving security incident. Occurs only at the POST-Launch level.

Thus, the RT score at the pre_launch level is always 0 seconds. If you think we should count the average from this: RT pre + RT post - let us know. According to your thinking, the result of Remediation Time for G Data should be 0s.
 

Jonny Quest

Level 13
Mar 2, 2023
628
F-Secure is not an isolated browser in the way that Bitdefender Safepay or Kaspersky Safemoney are. The F-Secure banking mode only blocks network connections to unapproved destinations. It also disconnects all untrusted applications from the internet.

It is always best practice to capture any identity and authentication credentials from within the banking mode or isolated browser session. There is a reason why Bitdefender and Kaspersky provide a virtual keyboard inside Safepay/Safemoney, however very, very few users actually utilize these features.

If I recall correctly Avast permitted a password manager. I did not know about F-Secure's. Bitdefender did have Wallet for use inside Safepay, but that has been done away with.

It is not that the vendors cannot implement protections outside of the isolated browser environment. They do. But some did not get it right and this was demonstrated during @Adrian Ścibor 's banking tests in 2019. Kaspersky, Bitdefender both failed to protect clipboard data outside Safepay/Safemoney. Many others such as F-Secure failed also.

Because nobody really explains how banking trojans work, users think that they are fully protected when typing or copy-pasta with a security program installed. With a banking trojan, it is running on the system. With Man-in-the-Browser, there is malicious code inside the webpage itself. Safepay and Safemoney are not going to stop Man-in-the-Browser. They might protect credentials as long as the user fastidiously does not do careless things such as switching back and forth to the desktop and typing/copying objects to the clipboard.

You can read the AVLab banking test reports for more details. @Adrian Ścibor will be the first to tell you that you cannot interpret their test results as a sweeping generalization to extend to every and all banking trojans or other malicious code. What that means is that the user has to be careful and fastidious about the typing and copying.

Thank you for the thoughtful reply, Greyton, it was helpful :)

 
  • Like
Reactions: Nevi and Trident

simmerskool

Level 30
Verified
Top Poster
Well-known
Apr 16, 2017
1,986
If one of those approved applications that is still permitted to connect outbound is hijacked or absued by banking malware, oh well you're bacon is cooked
no offense, but "absued" is not in the English dictionary. I thought I'd learn a new (to me) word, but alas, I guess it's a typo. I do those tooo.
 
  • Like
Reactions: Nevi and Trident

Sandbox Breaker

Level 9
Well-known
Jan 6, 2022
389
100% again for all. Why is then when I get zero days ... Most of these solutions fail. Vendor Agnosticism seems to serve well. Totally unrealistic. That's like saying body armor stopped all rounds... Even ones not rated for :ROFLMAO: (n) Yeah your level 3 vest stopped a .50. 💯
 
  • Like
Reactions: Nevi

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
171
100% again for all. Why is then when I get zero days ... Most of these solutions fail. Vendor Agnosticism seems to serve well. Totally unrealistic. That's like saying body armor stopped all rounds... Even ones not rated for :ROFLMAO: (n) Yeah your level 3 vest stopped a .50. 💯

As you wrote - when it comes to 0-day, there may not be a perfect solution. However, 0-day is not a targeted APT attack. You can write a zero-day in a few lines in ChatGTP and there is no finesse here. This is why multi-layered protection is important. Look out for solutions that isolate unknown samples if you have doubts about 100% protection.

If you look at the malware comparison CSV file, maybe less than 10% of the samples used in the test are 0-day for the MKS_VIR engine, which we use for initial sample analysis. This is the signature: INFECTION: HEUR.RoundKick.W.

edit: So it's not like we're only using known threats. We use in-the-wild threats, and they are what they are. A 0-day for vendor X is not necessarily a 0-day for Y.

With 0-day file (not url) will always be detected at post_lanuch level, there is no other way. I am not writing here about specific protection, such as the threat emulation while downloading in the browser (Zone Alarm / Check Point?)
 
Last edited:

Sandbox Breaker

Level 9
Well-known
Jan 6, 2022
389
As you wrote - when it comes to 0-day, there may not be a perfect solution. However, 0-day is not a targeted APT attack. You can write a zero-day in a few lines in ChatGTP and there is no finesse here. This is why multi-layered protection is important. Look out for solutions that isolate unknown samples if you have doubts about 100% protection.

If you look at the malware comparison CSV file, maybe less than 10% of the samples used in the test are 0-day for the MKS_VIR engine, which we use for initial sample analysis. This is the signature: INFECTION: HEUR.RoundKick.W.

edit: So it's not like we're only using known threats. We use in-the-wild threats, and they are what they are. A 0-day for vendor X is not necessarily a 0-day for Y.

With 0-day file (not url) will always be detected at post_lanuch level, there is no other way. I am not writing here about specific protection, such as the threat emulation while downloading in the browser (Zone Alarm / Check Point?)
I always speak about layers and human expertise. Whether USB or Web vector or hand on key attacks. I and many other see this 100% as 100% snakeoil. I respect your work and mean no disrespect.
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top