Testing real-time protection of antiviruses with 10.124 Sample

Dexter_Morgan31 said:
How do you think companies test samples? Open one-to-one in explorer?
Click to expand...

Here is an example that shows differences with your test:

Test Procedure​

The Malware Protection Test assesses a security program’s ability to protect a system against infection by malicious files before, during or after execution. The methodology used for each product tested is as follows. Prior to execution, all the test samples are subjected to on-access and on-demand scans by the security program, with each of these being done both offline and online. Any samples that have not been detected by any of these scans are then executed on the test system, with Internet/cloud access available, to allow e.g. behavioural detection features to come into play. If a product does not prevent or reverse all the changes made by a particular malware sample within a given time period, that test case is considered to be a miss. If the user is asked to decide whether a malware sample should be allowed to run, and in the case of the worst user decision system changes are observed, the test case is rated as “user-dependent”.
Click to expand...
www.av-comparatives.org

Malware Protection Test March 2025

AV-Comparatives' Consumer Malware Protection Test for March 2025 is now released!
www.av-comparatives.org www.av-comparatives.org

Why do they insist on manually executing undetected samples from Explorer? Probably it matters.
Why do they run each sample in the same environment (system is reverted to its initial state each time)? Probably it matters.
 
Last edited:
  • Like
  • +Reputation
  • Applause
Reactions: Jack, roger_m, Khushal and 3 others
Andy Ful said:
Here is an example that shows differences with your test:

www.av-comparatives.org

Malware Protection Test March 2025

AV-Comparatives' Consumer Malware Protection Test for March 2025 is now released!
www.av-comparatives.org www.av-comparatives.org
Click to expand...
This test is not a scanning test, Sorry but we have to run directly for testing. (We tried to scan and after run 1 year ago, 500-1000 viruses left by antivirus, We still cant open one to one.)
 
  • Like
Reactions: Khushal
Dexter_Morgan31 said:
This test is not a scanning test
Click to expand...
So it would be more appropriate to change the thread title from "Testing real-time protection of antiviruses with 10.124 Sample" to "Testing post-execution protection of antiviruses with 10.124 Sample" as "real-time" protection encompasses both pre-execution (scanning) and post-execution (behavioral).
 
  • Like
Reactions: Behold Eck, roger_m, Trident and 2 others
Dexter_Morgan31 said:
Then i ask to you. How we supposed to test 10124 sample? I really ask to do right.
Click to expand...

You cannot in a sensible time and resources. AV testing labs use many resources (computers, people, etc.), and one test can last a few months. Even with all their effort, those tests are still questionable in some aspects. They do their best, but the testing subject is too fast and too complex.
If you really want to test 10000 samples, then it would be hard to make the testing methodology more realistic.
It can be done with 100-200 fresh samples. You can ask @Shadowra about details.
 
  • Like
  • +Reputation
  • Hundred Points
Reactions: Behold Eck, Dimitriss, roger_m and 7 others
winball501 said:
Why TPSC using test script? what's difference with my tool?
Click to expand...

You do not use Python, which can be an advantage for your tool. However, the idea of running many samples one by one in a short time by a single tool is rather an unrealistic scenario. It is hard to reliably interpret the test results in the context of in-the-wild protection. Of course, such tests may be interpreted as a "load (stress) test", as @Trident already mentioned. Most AVs are not trained to protect against "load (stress) tests".
 
Last edited:
  • Like
  • +Reputation
Reactions: EASTER, Game Of Thrones, Khushal and 2 others
Andy Ful said:
You do not use Python, which can be an advantage for your tool. However, the idea of running many samples one by one in a short time by a single tool is rather an unrealistic scenario. It is hard to reliably interpret the test results in the context of in-the-wild protection. Of course, such tests may be interpreted as a "load (stress) test", as @Trident already mentioned.
Click to expand...
Rather unrealistic? It's completely unrealistic and will never happen in the real world. If my system was hit with 10,000 malware at once id pull the shotgun out of the closet and redecorate my wall with fine particles.
 
  • HaHa
  • Like
Reactions: Behold Eck, roger_m, harlan4096 and 2 others
Shadowra said:
We've already said it, but bombarding an antivirus in real-time is useless, especially with tools or scripts....

If you want to execute, it's best to do what I do: execute, wait for the actions (Malware or AV) and then move on to the next one.
Click to expand...
Totally agree. Running a malware test via a script would yield questionable results as it won't account for things that use Sleep functions or just plain old multi-stage malware. A malware tester MUST be very familiar with the malware used and not just be a collector.

Further, running 100 samples via a script is a Gangbang. Running 10,000+ files via a script is ClickBait.
 
  • Like
  • Hundred Points
  • +Reputation
Reactions: kylprq, EASTER, roger_m and 8 others
I'm just a noob, far be it from me to play the arbiter of truth at MWT.com. But I am a huge fan of logic, even though I lack the skills on display at this forum.

OK I'll concede to your knowledge :"This isn't the proper way to test"... BUT

If I were testing using the horde method, and I tested 5 well known, respected AV's, and threw 50k bad actors at them all , and 2 survived. That would still seem to me, to say something about the 2 that made it through the test.

Then if those same 2, could manage a proper test, with flying colors, that would add even more evidence to the results.

OR am I missing this by a long shot?
 
  • Like
  • Hundred Points
Reactions: Behold Eck, Trident, Dexter_Morgan31 and 1 other person
OK maybe I didn't explain that right.... Let's say you and I go to a vendor to buy some boots.

The store owner says these boots are really good, and they are water proof.

And I happen to go camping, and while camping a fire breaks out. I walk thru some very high temps and fire, and the boots end up not only protecting my feet, but end up looking good too. Then we could put a check mark in 2 categories for those boots. Even though I tested them not only in water, but fire also, it provides us more data
 
  • Like
Reactions: Dexter_Morgan31 and Trident
annaegorov said:
I'm just a noob, far be it from me to play the arbiter of truth at MWT.com. But I am a huge fan of logic, even though I lack the skills on display at this forum.

OK I'll concede to your knowledge :"This isn't the proper way to test"... BUT

If I were testing using the horde method, and I tested 5 well known, respected AV's, and threw 50k bad actors at them all , and 2 survived. That would still seem to me, to say something about the 2 that made it through the test.

Then if those same 2, could manage a proper test, with flying colors, that would add even more evidence to the results.

OR am I missing this by a long shot?
Click to expand...
Testing an AV with a "horde" of malware is like testing computers by dropping them off a 10-story building. In both scenarios, you might see which one survives the initial, brutal impact, but this crude test tells you nothing about the real-world performance.
 
  • Like
  • +Reputation
Reactions: roger_m, simmerskool and lostpass
cruelsister said:
Totally agree. Running a malware test via a script would yield questionable results as it won't account for things that use Sleep functions or just plain old multi-stage malware. A malware tester MUST be very familiar with the malware used and not just be a collector.
Click to expand...
Yes, you're exactly right. The same principle applies to zip folder on desktop execution, perhaps even more so.

Running a malware sample directly from the desktop is a fundamentally flawed testing method for the same reason a simple script is, it ignores the context of the attack.

A true infection route is a sequence of events, and modern security products are designed to detect suspicious patterns within that sequence. By executing a file directly, you skip most of the steps where detection would occur.
 
  • Like
Reactions: roger_m and simmerskool

You may also like...