Evaluation of protection solutions based on telemetry data of malware in-the-wild (May 2023)

[Adrian Ścibor]

Hello Everyone!

I would like to announce that we have published the results for May 2023. What has changed since the March 2023 edition?

You'll find details in the Changelog, but the highlights are:

1. NEW: We have started to create individual website for producers - you will find links on the Awards website for now.
2. NEW: During the selection of URL samples, we added a technology partner scanner to get a better opinion about the file. You will find details about each file in a new column in "3rd party detection". To see that, please download CSV table from Recent Results.
3. NEW: We implemented a new feature in the testing application that allows to collect telemetry data from selected logs of the tested solution. Developers can receive more feedback on blocked threats upon special request.
4. Improved: This month’s changes also include generating improved CSV reports from the test with additional telemetry data that we share with all developers.

I hope that the changes we developed are important and useful for you

Cumulative Results: Recent Results » AVLab Cybersecurity Foundation

Publication: Evaluation Of Protection Solutions Based On Telemetry Data Of Malware In-the-wild (May 2023) » AVLab Cybersecurity Foundation

Tested solutions for home and small office installation in May 2023:​

1. Avast Free Antivirus
2. Bitdefender Antivirus Free
3. G Data Total Security
4. Kaspersky Plus
5. Malwarebytes Premium
6. Microsoft Defender
7. Quick Heal Total Security
8. Webroot Antivirus
9. Xcitium Internet Security

Solutions for business and governmental institutions:

1. Emsisoft Business Security
2. Malwarebytes Endpoint Protection
3. Xcitium ZeroThreat Advanced

Developers who wish to cooperate with us please contact us via AMTSO or directly at our contact webpage.

If you have any suggestions, let me know!

 

[Scirious]

In the following page under Bitdefender it says Avast Free Antivirus: Recent Results » AVLab Cybersecurity Foundation

 

[Trident]

Bitdefender weirdly has very high failure rate on unsigned and packed executables with high number of detections on VT. Not the type that have 4/70. Do they have some issue with the cloud or something? Because I checked their failures and on VT they are covered (of course telemetry may have been derived from the test and maybe they added detection after). One was scanned on 15th of May and already was detected.

I love the CSV table with the hashes!!!

 

[RansomwareRemediation]

Hello Everyone!

I would like to announce that we have published the results for May 2023. What has changed since the March 2023 edition?

You'll find details in the Changelog, but the highlights are:

1. NEW: We have started to create individual website for producers - you will find links on the Awards website for now.
2. NEW: During the selection of URL samples, we added a technology partner scanner to get a better opinion about the file. You will find details about each file in a new column in "3rd party detection". To see that, please download CSV table from Recent Results.
3. NEW: We implemented a new feature in the testing application that allows to collect telemetry data from selected logs of the tested solution. Developers can receive more feedback on blocked threats upon special request.
4. Improved: This month’s changes also include generating improved CSV reports from the test with additional telemetry data that we share with all developers.

I hope that the changes we developed are important and useful for you

Cumulative Results: Recent Results » AVLab Cybersecurity Foundation

Publication: Evaluation Of Protection Solutions Based On Telemetry Data Of Malware In-the-wild (May 2023) » AVLab Cybersecurity Foundation

Tested solutions for home and small office installation in May 2023:​

1. Avast Free Antivirus
2. Bitdefender Antivirus Free
3. G Data Total Security
4. Kaspersky Plus
5. Malwarebytes Premium
6. Microsoft Defender
7. Quick Heal Total Security
8. Webroot Antivirus
9. Xcitium Internet Security

Solutions for business and governmental institutions:

1. Emsisoft Business Security
2. Malwarebytes Endpoint Protection
3. Xcitium ZeroThreat Advanced

Developers who wish to cooperate with us please contact us via AMTSO or directly at our contact webpage.

If you have any suggestions, let me know!

why no test Bitdefender paid version ? ...
Grretings.

 

[SeriousHoax]

Bitdefender weirdly has very high failure rate on unsigned and packed executables with high number of detections on VT. Not the type that have 4/70. Do they have some issue with the cloud or something? Because I checked their failures and on VT they are covered (of course telemetry may have been derived from the test and maybe they added detection after). One was scanned on 15th of May and already was detected.

I love the CSV table with the hashes!!!

Could be related to Bitdefender Free's lack of some protection features that are available in the paid like AMSI, memory scanning, command line scanner and some more.
For example, the PowerShell based ransomware that I talked to you about a few days ago which was instantly detected by BD paid version after executing thanks to AMSI while BD Free didn't react in my first attempt and on my second test it was detected by the behavior blocker but not before desktop files were encrypted.
Kaspersky Free and Avast Free are more complete free products.
I also checked those 5 missed by BD and two pair of them were the same malware variant basically. A more diverse set of samples would've been better, but probably hard to find.

 

[Adrian Ścibor]

Bitdefender weirdly has very high failure rate on unsigned and packed executables with high number of detections on VT. Not the type that have 4/70. Do they have some issue with the cloud or something? Because I checked their failures and on VT they are covered (of course telemetry may have been derived from the test and maybe they added detection after). One was scanned on 15th of May and already was detected.

I love the CSV table with the hashes!!!

Antivirus engines implemented on VirusTotal operate from the command line. In this connection, they may not be able to access the functionality which form part of real security suites. It proves a practical approach to testing. For example, malware which will be blocked by a firewall module, it will not be blocked by an antivirus engine on VirusTotal in a realistic scenario.

As we read in the official document, antivirus engines on VirusTotal are binary versions, operating from the command line. They will not behave exactly the same as versions which we install on computers. In other words, engines implemented on VirusTotal usually do not have a firewall, scanning in the cloud, sandbox, HIPS, DLP, blocking script viruses, and other modules.

We are tired of repeating that VirusTotal was not designed as a tool to perform antivirus comparative analyses, but as a tool that checks suspicious samples with several antivirus solutions and helps antivirus labs by sending them the malware they have failed to detect. Those who use VirusTotal to perform antivirus comparative analyses should know that they are making many implicit errors in their methodology.
– source: https://support.virustotal.com/hc/e...de-statistics-comparing-antivirus-performance

This is the first real reason which shows why not to follow the opinion from the VirusTotal scanning vs real machine. The second reason probably is more relevant, but it is not documented:

Malware which is uploaded using online panel into the VirusTotal service IS NOT LAUNCHED in certain cases. It performs a static analysis of a file, i.e. checksums are calculated, DLLs are extracted, Windows API functions are disclosed, and links with other malicious campaigns are revealed. Every file is scanned by antivirus engine, however, a dynamic analysis is performed only for binary files in the vendor cloud. Consequently, analyzed EXE files will show a virus activity, but for instance VBS scripts, malicious invoices, PDF file, or macro viruses in DOCX files not always.

 

[Trident]

Antivirus engines implemented on VirusTotal operate from the command line. In this connection, they may not be able to access the functionality which form part of real security suites. It proves a practical approach to testing. For example, malware which will be blocked by a firewall module, it will not be blocked by an antivirus engine on VirusTotal in a realistic scenario.

As we read in the official document, antivirus engines on VirusTotal are binary versions, operating from the command line. They will not behave exactly the same as versions which we install on computers. In other words, engines implemented on VirusTotal usually do not have a firewall, scanning in the cloud, sandbox, HIPS, DLP, blocking script viruses, and other modules.

This is the first real reason which shows why not to follow the opinion from the VirusTotal scanning vs real machine. The second reason probably is more relevant, but it is not documented:

Malware which is uploaded using online panel into the VirusTotal service IS NOT LAUNCHED in certain cases. It performs a static analysis of a file, i.e. checksums are calculated, DLLs are extracted, Windows API functions are disclosed, and links with other malicious campaigns are revealed. Every file is scanned by antivirus engine, however, a dynamic analysis is performed only for binary files in the vendor cloud. Consequently, analyzed EXE files will show a virus activity, but for instance VBS scripts, malicious invoices, PDF file, or macro viruses in DOCX files not always.

Yeah, I am aware of that

But still, very weird performance from Bitdefender. Makes you wonder if they are neglecting the free version somehow.

As @SeriousHoax pointed, there are some features disabled in BItdefender Free (memory scanning, command line monitoring and others), but that's a PEEXE.
And on VT memory scanning and command line monitoring cant'be responsible for the detection since the file is not executed.
There were no "memory" (unpacked payload) and command line parameters on VT.
In contrast, the locally installed Bitdefender free has the benefit of performing all pre-execution and post-execution analysis included within the anti-malware SDK.

Bitdefender Free has the highest fail rate again.