AVLab.pl Summary of the Advanced In-The-Wild Malware Test – September 2024

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
214
Dear Readers!

With this summary we conclude this year’s penultimate series of Advanced In-The-Wild Malware Tests. As of September 2024, we have made a few changes under the hood in the Windows 10/11 security software testing application.

The first and biggest is that we have developed a new method of capturing more evidence of malware sample detection during the test in the form of screenshots taken several times a minute.

We use an OCR tool to read text from the images. Based on this text, we compare the keywords with matching alerts for the anti-virus software in question. If there is a positive match, we store all the records in a database, from which summaries are generated for the vendors. This image recognition capability also allows us to more thoroughly analyse potential malware samples before qualifying them for testing – we reject installers, unwanted (non-malicious) applications, corrupted files and other files that cannot be run in a Windows 11 environment:

As an example - a "system error" is reported for a potentially malicious keylog.exe application:

system-error-keylog.png


During the initial selection, the corrupted malware sample is rejected due to the detection of a Windows error with the keywords "system error":

system-error-in-backend-output.png


Evidence of sample removal from the assay at the preliminary analysis stage:

backend-resposne-768x16.png


In summary, the OCR tool we have incorporated into the testing process is used to capture anti-virus alerts and to more accurately identify potential malware samples before they are qualified for testing. The screenshots provide the vendors with further and irrefutable evidence of whether or not malware has been detected.

The OCR tool is an additional opinion from the test, alongside the Sysmon logs and the logs generated by the security software under test. See our methodology page for more details.

Security alerts example screenshots:


kaspersky_proof_september_2024.png


webroot_proof_september_2024.png

We have added optional changed onto the backend as well. You can read them all on the transparency website: Changelog » AVLab Cybersecurity Foundation

September 2024

Results: Recent Results » AVLab Cybersecurity Foundation

And the publication: Summary Of The Advanced In-The-Wild Malware Test - September 2024 » AVLab Cybersecurity Foundation

We are also working with other vendors to add them to the tests. This is not always technically easy, so we ask for your understanding.

I hope that these changes will contribute to even better confidence in us and in our tests.
 

Bot

AI-powered Bot
Apr 21, 2016
4,476
Thanks for the update! The new method of capturing evidence of malware detection via screenshots and OCR tool sounds promising. It's good to know that non-malicious applications and corrupted files are being rejected for testing. Looking forward to seeing more vendors added to the tests.
 
  • Like
Reactions: Gandalf_The_Grey

Gandalf_The_Grey

Level 83
Verified
Honorary Member
Top Poster
Content Creator
Well-known
Apr 24, 2016
7,389
Threat neutralisation and removal
The fastest to neutralise and remove threats correctly were the vendors:
– Kaspersky Plus (took an average of 1.56 seconds on the set of samples used)
– F-Secure Total (1.7 seconds on the set of samples used)
– K7 Total Security (6.1 seconds)
Thanks for the test (y)
Great job done here by Kaspersky and F-Secure.
I hope you can add Microsoft Defender soon.
 

simmerskool

Level 38
Verified
Top Poster
Well-known
Apr 16, 2017
2,729
Threat neutralisation and removal

Thanks for the test (y)
Great job done here by Kaspersky and F-Secure.
I hope you can add Microsoft Defender soon.
@Adrian Ścibor yes very good testing, but wondering why MS Defender is not included as it is surely readily available. Please include it next, or give us a heads up why not... thanks!
 

Vitali Ortzi

Level 26
Verified
Top Poster
Well-known
Dec 12, 2016
1,552
Dear Readers!

With this summary we conclude this year’s penultimate series of Advanced In-The-Wild Malware Tests. As of September 2024, we have made a few changes under the hood in the Windows 10/11 security software testing application.

The first and biggest is that we have developed a new method of capturing more evidence of malware sample detection during the test in the form of screenshots taken several times a minute.

We use an OCR tool to read text from the images. Based on this text, we compare the keywords with matching alerts for the anti-virus software in question. If there is a positive match, we store all the records in a database, from which summaries are generated for the vendors. This image recognition capability also allows us to more thoroughly analyse potential malware samples before qualifying them for testing – we reject installers, unwanted (non-malicious) applications, corrupted files and other files that cannot be run in a Windows 11 environment:

As an example - a "system error" is reported for a potentially malicious keylog.exe application:

View attachment 285805

During the initial selection, the corrupted malware sample is rejected due to the detection of a Windows error with the keywords "system error":

View attachment 285806

Evidence of sample removal from the assay at the preliminary analysis stage:

View attachment 285809

In summary, the OCR tool we have incorporated into the testing process is used to capture anti-virus alerts and to more accurately identify potential malware samples before they are qualified for testing. The screenshots provide the vendors with further and irrefutable evidence of whether or not malware has been detected.

The OCR tool is an additional opinion from the test, alongside the Sysmon logs and the logs generated by the security software under test. See our methodology page for more details.

Security alerts example screenshots:


View attachment 285807

View attachment 285808

We have added optional changed onto the backend as well. You can read them all on the transparency website: Changelog » AVLab Cybersecurity Foundation

September 2024

Results: Recent Results » AVLab Cybersecurity Foundation

And the publication: Summary Of The Advanced In-The-Wild Malware Test - September 2024 » AVLab Cybersecurity Foundation

We are also working with other vendors to add them to the tests. This is not always technically easy, so we ask for your understanding.

I hope that these changes will contribute to even better confidence in us and in our tests.
Pishing , exploitation are the most common vectors to get into organizations nowadays
I would love intercept X , defender ATP etc tested for exploit mitigations And popular products that offer pishing protection tested too


As especially extensive exploit mitigation testing isn't done anymore by labs
 

Vitali Ortzi

Level 26
Verified
Top Poster
Well-known
Dec 12, 2016
1,552
Pretty impressed by K7 blocking 9/10 pre-execution!
Thank you for sharing, looking forward to further tests, and happy K7 joined your testing :)
Av industry is really competitive and it's not bad when you consider what it has to compete with meanwhile it keeps excellent performance usage
Still would recommend something like eset instead just because it does better in testing
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top