Community!
If you've been following us, you've probably noticed that we've published the new results for the July edition of the Advanced In-The-Wild Malware Test.
Let me quote a few changes from our article:
Firstly, developers of the tested security solutions will receive additional extended logs in the form of an event tree with correlated processes, newly created and deleted files, commands executed in the terminal, Internet connections, and other instructions executed by the malware. This will allow for faster and easier analysis of what happened at the point of contact between the malware and the tested security package. These logs can also serve as additional evidence of the lack of response of the antivirus/EDR engine to malware:
As an example, one of the malware samples dropped files into the TEMP folder to initiate malicious activity on the system.
Here, malware queries known DNS providers and lesser-known IP servers in order to use C&C (an attempt to obtain additional instructions from a server controlled by the attacker).
- In addition, at the end of June 2025, we updated our Windows 11 operating systems to the latest versions in order to use the most up-to-date Windows 11 builds during testing.
- Finally, starting with this edition, we are capturing information from even more LOLBINs:
The changelog is updated: Changelog » AVLab Cybersecurity Foundation
The article: Analysis Of System Protection Against Active Online Malware – July 2025 » AVLab Cybersecurity Foundation
The Recent Results web-page: Recent Results » AVLab Cybersecurity Foundation
From the other hand
Additionally, I would like to inform that from 2026 onward, it will be more difficult to obtain the Excellent certificate. Based on historical results, the 99% threshold is too low, so we are considering two options:
1) “Excellent” for 100% only and “Certified” for 99.0-99.9% - the Excellent certificate must remain unique.
2) Raise the threshold to 99.6% minimum for the Excellent certificate. We must allow for a certain margin of error for the requirements of the Microsoft Virus Initiative program. At this phase two clusters will be created:
2.1 solutions in the range 99.6-100 will receive Excellent
2.2 solutions in the range 99-99.5 will receive Certified
If you only award “Excellent” for 100% effectiveness, you are creating an illusion of perfection. Do you agree?
In practice, 100% in a test sometimes does not mean 100% in reality – because a test always has a limited sample set. This may be a coincidence (luck in sample selection) rather than the actual superiority of the product. This is why some people may criticize option 1) because no one can test the entire population of malware or guarantee that there will be no errors in the tests.
We can talk about clustering in another thread so as not to cause confusion here.
I will keep you informed of any changes. The year 2025 will be summarized according to the old rules.[/B]
If you've been following us, you've probably noticed that we've published the new results for the July edition of the Advanced In-The-Wild Malware Test.
Let me quote a few changes from our article:
Firstly, developers of the tested security solutions will receive additional extended logs in the form of an event tree with correlated processes, newly created and deleted files, commands executed in the terminal, Internet connections, and other instructions executed by the malware. This will allow for faster and easier analysis of what happened at the point of contact between the malware and the tested security package. These logs can also serve as additional evidence of the lack of response of the antivirus/EDR engine to malware:
As an example, one of the malware samples dropped files into the TEMP folder to initiate malicious activity on the system.
Here, malware queries known DNS providers and lesser-known IP servers in order to use C&C (an attempt to obtain additional instructions from a server controlled by the attacker).
- In addition, at the end of June 2025, we updated our Windows 11 operating systems to the latest versions in order to use the most up-to-date Windows 11 builds during testing.
- Finally, starting with this edition, we are capturing information from even more LOLBINs:
The changelog is updated: Changelog » AVLab Cybersecurity Foundation
The article: Analysis Of System Protection Against Active Online Malware – July 2025 » AVLab Cybersecurity Foundation
The Recent Results web-page: Recent Results » AVLab Cybersecurity Foundation
From the other hand
Additionally, I would like to inform that from 2026 onward, it will be more difficult to obtain the Excellent certificate. Based on historical results, the 99% threshold is too low, so we are considering two options:
1) “Excellent” for 100% only and “Certified” for 99.0-99.9% - the Excellent certificate must remain unique.
2) Raise the threshold to 99.6% minimum for the Excellent certificate. We must allow for a certain margin of error for the requirements of the Microsoft Virus Initiative program. At this phase two clusters will be created:
2.1 solutions in the range 99.6-100 will receive Excellent
2.2 solutions in the range 99-99.5 will receive Certified
If you only award “Excellent” for 100% effectiveness, you are creating an illusion of perfection. Do you agree?
In practice, 100% in a test sometimes does not mean 100% in reality – because a test always has a limited sample set. This may be a coincidence (luck in sample selection) rather than the actual superiority of the product. This is why some people may criticize option 1) because no one can test the entire population of malware or guarantee that there will be no errors in the tests.
We can talk about clustering in another thread so as not to cause confusion here.
I will keep you informed of any changes. The year 2025 will be summarized according to the old rules.[/B]
