Evaluation of protection solutions based on telemetry data of malware in-the-wild (May 2023)

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
175
Hello Everyone!

I would like to announce that we have published the results for May 2023. What has changed since the March 2023 edition?

You'll find details in the Changelog, but the highlights are:

1. NEW: We have started to create individual website for producers - you will find links on the Awards website for now.
2. NEW: During the selection of URL samples, we added a technology partner scanner to get a better opinion about the file. You will find details about each file in a new column in "3rd party detection". To see that, please download CSV table from Recent Results.
3. NEW: We implemented a new feature in the testing application that allows to collect telemetry data from selected logs of the tested solution. Developers can receive more feedback on blocked threats upon special request.
4. Improved: This month’s changes also include generating improved CSV reports from the test with additional telemetry data that we share with all developers.

I hope that the changes we developed are important and useful for you :)

Cumulative Results: Recent Results » AVLab Cybersecurity Foundation

Publication: Evaluation Of Protection Solutions Based On Telemetry Data Of Malware In-the-wild (May 2023) » AVLab Cybersecurity Foundation

Tested solutions for home and small office installation in May 2023:

  1. Avast Free Antivirus
  2. Bitdefender Antivirus Free
  3. G Data Total Security
  4. Kaspersky Plus
  5. Malwarebytes Premium
  6. Microsoft Defender
  7. Quick Heal Total Security
  8. Webroot Antivirus
  9. Xcitium Internet Security
Solutions for business and governmental institutions:
  1. Emsisoft Business Security
  2. Malwarebytes Endpoint Protection
  3. Xcitium ZeroThreat Advanced
Developers who wish to cooperate with us please contact us via AMTSO or directly at our contact webpage.

If you have any suggestions, let me know! :)

 

Attachments

  • test in the numbers may 2023.jpg
    test in the numbers may 2023.jpg
    442.5 KB · Views: 119
  • recent results may 2023.jpg
    recent results may 2023.jpg
    52.5 KB · Views: 128
  • malware selecting.jpg
    malware selecting.jpg
    112.1 KB · Views: 128

Trident

Level 27
Verified
Top Poster
Well-known
Feb 7, 2023
1,687
Bitdefender weirdly has very high failure rate on unsigned and packed executables with high number of detections on VT. Not the type that have 4/70. Do they have some issue with the cloud or something? Because I checked their failures and on VT they are covered (of course telemetry may have been derived from the test and maybe they added detection after). One was scanned on 15th of May and already was detected.

I love the CSV table with the hashes!!!
 

RansomwareRemediation

Level 4
Verified
Well-known
Jun 22, 2020
163
Hello Everyone!

I would like to announce that we have published the results for May 2023. What has changed since the March 2023 edition?

You'll find details in the Changelog, but the highlights are:

1. NEW: We have started to create individual website for producers - you will find links on the Awards website for now.
2. NEW: During the selection of URL samples, we added a technology partner scanner to get a better opinion about the file. You will find details about each file in a new column in "3rd party detection". To see that, please download CSV table from Recent Results.
3. NEW: We implemented a new feature in the testing application that allows to collect telemetry data from selected logs of the tested solution. Developers can receive more feedback on blocked threats upon special request.
4. Improved: This month’s changes also include generating improved CSV reports from the test with additional telemetry data that we share with all developers.

I hope that the changes we developed are important and useful for you :)

Cumulative Results: Recent Results » AVLab Cybersecurity Foundation

Publication: Evaluation Of Protection Solutions Based On Telemetry Data Of Malware In-the-wild (May 2023) » AVLab Cybersecurity Foundation

Tested solutions for home and small office installation in May 2023:

  1. Avast Free Antivirus
  2. Bitdefender Antivirus Free
  3. G Data Total Security
  4. Kaspersky Plus
  5. Malwarebytes Premium
  6. Microsoft Defender
  7. Quick Heal Total Security
  8. Webroot Antivirus
  9. Xcitium Internet Security
Solutions for business and governmental institutions:
  1. Emsisoft Business Security
  2. Malwarebytes Endpoint Protection
  3. Xcitium ZeroThreat Advanced
Developers who wish to cooperate with us please contact us via AMTSO or directly at our contact webpage.

If you have any suggestions, let me know! :)
why no test Bitdefender paid version ? ...
Grretings.
 

SeriousHoax

Level 47
Verified
Top Poster
Well-known
Mar 16, 2019
3,630
Bitdefender weirdly has very high failure rate on unsigned and packed executables with high number of detections on VT. Not the type that have 4/70. Do they have some issue with the cloud or something? Because I checked their failures and on VT they are covered (of course telemetry may have been derived from the test and maybe they added detection after). One was scanned on 15th of May and already was detected.

I love the CSV table with the hashes!!!
Could be related to Bitdefender Free's lack of some protection features that are available in the paid like AMSI, memory scanning, command line scanner and some more.
For example, the PowerShell based ransomware that I talked to you about a few days ago which was instantly detected by BD paid version after executing thanks to AMSI while BD Free didn't react in my first attempt and on my second test it was detected by the behavior blocker but not before desktop files were encrypted.
Kaspersky Free and Avast Free are more complete free products.
I also checked those 5 missed by BD and two pair of them were the same malware variant basically. A more diverse set of samples would've been better, but probably hard to find.
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
175
Bitdefender weirdly has very high failure rate on unsigned and packed executables with high number of detections on VT. Not the type that have 4/70. Do they have some issue with the cloud or something? Because I checked their failures and on VT they are covered (of course telemetry may have been derived from the test and maybe they added detection after). One was scanned on 15th of May and already was detected.

I love the CSV table with the hashes!!!
Antivirus engines implemented on VirusTotal operate from the command line. In this connection, they may not be able to access the functionality which form part of real security suites. It proves a practical approach to testing. For example, malware which will be blocked by a firewall module, it will not be blocked by an antivirus engine on VirusTotal in a realistic scenario.

As we read in the official document, antivirus engines on VirusTotal are binary versions, operating from the command line. They will not behave exactly the same as versions which we install on computers. In other words, engines implemented on VirusTotal usually do not have a firewall, scanning in the cloud, sandbox, HIPS, DLP, blocking script viruses, and other modules.
We are tired of repeating that VirusTotal was not designed as a tool to perform antivirus comparative analyses, but as a tool that checks suspicious samples with several antivirus solutions and helps antivirus labs by sending them the malware they have failed to detect. Those who use VirusTotal to perform antivirus comparative analyses should know that they are making many implicit errors in their methodology.
– source: https://support.virustotal.com/hc/e...de-statistics-comparing-antivirus-performance
This is the first real reason which shows why not to follow the opinion from the VirusTotal scanning vs real machine. The second reason probably is more relevant, but it is not documented:

Malware which is uploaded using online panel into the VirusTotal service IS NOT LAUNCHED in certain cases. It performs a static analysis of a file, i.e. checksums are calculated, DLLs are extracted, Windows API functions are disclosed, and links with other malicious campaigns are revealed. Every file is scanned by antivirus engine, however, a dynamic analysis is performed only for binary files in the vendor cloud. Consequently, analyzed EXE files will show a virus activity, but for instance VBS scripts, malicious invoices, PDF file, or macro viruses in DOCX files not always.
 

Trident

Level 27
Verified
Top Poster
Well-known
Feb 7, 2023
1,687
Antivirus engines implemented on VirusTotal operate from the command line. In this connection, they may not be able to access the functionality which form part of real security suites. It proves a practical approach to testing. For example, malware which will be blocked by a firewall module, it will not be blocked by an antivirus engine on VirusTotal in a realistic scenario.

As we read in the official document, antivirus engines on VirusTotal are binary versions, operating from the command line. They will not behave exactly the same as versions which we install on computers. In other words, engines implemented on VirusTotal usually do not have a firewall, scanning in the cloud, sandbox, HIPS, DLP, blocking script viruses, and other modules.

This is the first real reason which shows why not to follow the opinion from the VirusTotal scanning vs real machine. The second reason probably is more relevant, but it is not documented:

Malware which is uploaded using online panel into the VirusTotal service IS NOT LAUNCHED in certain cases. It performs a static analysis of a file, i.e. checksums are calculated, DLLs are extracted, Windows API functions are disclosed, and links with other malicious campaigns are revealed. Every file is scanned by antivirus engine, however, a dynamic analysis is performed only for binary files in the vendor cloud. Consequently, analyzed EXE files will show a virus activity, but for instance VBS scripts, malicious invoices, PDF file, or macro viruses in DOCX files not always.
Yeah, I am aware of that :D

But still, very weird performance from Bitdefender. Makes you wonder if they are neglecting the free version somehow.

As @SeriousHoax pointed, there are some features disabled in BItdefender Free (memory scanning, command line monitoring and others), but that's a PEEXE.
And on VT memory scanning and command line monitoring cant'be responsible for the detection since the file is not executed.
There were no "memory" (unpacked payload) and command line parameters on VT.
In contrast, the locally installed Bitdefender free has the benefit of performing all pre-execution and post-execution analysis included within the anti-malware SDK.

Bitdefender Free has the highest fail rate again.
 
Last edited:

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top