Testing real-time protection of antiviruses with 10.124 Sample

[Trident]

The files were scanned. Web shield and other technologies scan the file with the same anti-malware engine that’s used for the on-demand scan (which eliminated >90% of the samples). The same scan is also performed just before you execute the file.

Stating that “when executing from disk you bypass the on-disk scanner” is wrong.

Malware, even when coming as downloads, doesn’t always come from sources like 181.019.23.65/malware/stealers/explorer.exe.

It can be distributed via trusted platforms (like discord, WhatsApp Web, GitHub and so on) and can also be executed from portable drives. Worms like Xworm are in the wild, happily dropping malicious shortcuts on USB. They also come with convenient built-in modules like UAC bypasses that allow easy execution of the set mp-preference command.

Whilst real-world testing and attack replication are important, I am sad to see them used as excuse (by knowledgeable members) for vendors trying to profiteer with minimum reinvestments, detecting and blocking only the most common, prevalent, widespread malware, under the most frequent and favourable conditions.
Web filters and context awareness are not a replacement for efficient anti-malware, they should be ran alongside of it and each layer must be formidable on its own, not good this, flimsy that.
That’s the meaning of defence in depth, not “let me fix the broken door with chewing gum and WD40”.

No anti-malware, no matter the test and where the malware came from, no matter where third-party tools were used or not, ever has an excuse for leaving over a hundred pieces of malware ON DISK (not even fileless) and active in memory. This is not anti-malware, it’s placebo.

 

[Parkinsond]

A guy recently was complaining about Kaspersky not being lineant towards tools/PUPs etc anymore. This test would have cleared any doubts he must have.

But it detected tools on my side.

 

[monkeylove]

Would core isolation, etc., in Windows Security help?

 

[Nunzio_77]

In my opinion, Avast didn't detect PUPs because, if I remember correctly, this protection isn't enabled by default. Therefore, it's recommended to enable it (a recommendation AVLab also provides). AVLab also suggests enabling enhanced protection, so I think Avast's PUP results would have been different.

 

[Andy Ful]

Dexter_Morgan31,​

My posts are neutral and complementary to your opening post. I do not contradict anything in the test results. I assume that you made a lot of effort to make it as good as possible. However, some additional information about the test is welcome. Many readers can have a problem with realizing what kind of test it is and how it is related to in-the-wild protection. Complementary information is included in my posts (comparison with AV-Comparatives Malware Protection tests, etc.).

Each kind of test has some limitations and can only partially show the in-the-wild reality. So, it is natural to discuss the limitations of testing 1-4 week old samples and limitations related to executing many samples in a short time.

1. Running malware samples via a testing tool can matter. Some AVs can check files differently when executed directly from the Explorer, as compared to files executed by a trusted application. I do not insist that this is a case in your test, but no one has probably confirmed this in practice with this tool.
2. The test results (and also "corrupted status") can depend on the sample order. By changing the order in which samples are run, one can get different results.
3. You use the "corrupted" and "partially corrupted" criteria. It would be good to define them more clearly.

 

[Andy Ful]

No anti-malware, no matter the test and where the malware came from, no matter where third-party tools were used or not, ever has an excuse for leaving over a hundred pieces of malware ON DISK (not even fileless) and active in memory. This is not anti-malware, it’s placebo.

It is an unnecessary and risky generalisation (overinterpretation) of this test.

 

[Shadowra]

We've already said it, but bombarding an antivirus in real-time is useless, especially with tools or scripts....

If you want to execute, it's best to do what I do: execute, wait for the actions (Malware or AV) and then move on to the next one.

 

[Andy Ful]

We've already said it, but bombarding an antivirus in real-time is useless, especially with tools or scripts....

If you want to execute, it's best to do what I do: execute, wait for the actions (Malware or AV) and then move on to the next one.

Your tests are much closer to reality.
But even you would not try to manually execute 10 thousand samples. This is a special kind of test that can be interesting only for a narrow group of users, but misunderstood by most readers.

 

[annaegorov]

We've already said it, but bombarding an antivirus in real-time is useless, especially with tools or scripts....

If you want to execute, it's best to do what I do: execute, wait for the actions (Malware or AV) and then move on to the next one.

10,000 samples!!! In some circles this test method could be called "Horde Mode"

 

[ForgottenSeer 123960]

I'd like to clarify my perspective on the testing discussions, as my goal is constructive feedback, not conflict. I recognize the value of community-driven tests for engagement and exploration.

My primary concern is for the guests and new users who come to this forum for guidance. Without proper context, it's easy for them to misinterpret the results of informal tests. They might either become overconfident in their security or be discouraged from using a product that would have served them well.

I feel I've made my point on the importance of distinguishing these tests from real-world conditions. It seems best for me to step back from this specific topic now, but I hope we can all keep our less-experienced readers in mind.

 

[Dexter_Morgan31]

I understand your comments about skipping protection layers. Yes you're right that's why you can count this test MPT (Malware Protection Test) but i dont understand how you expect executing viruses from us. We cant open 10.124 Samples from Explorer, Im sorry but are you kidding about this? Whatever. This test is still valid test, Absolutely this is not enough, we also need to Zero-Day Viruses to get ideas more properly. However, you cant say this test is unvalid, It can still give you idea.

 

[Dexter_Morgan31]

Additionally, i need to say i dont trust companies like AV-TEST or AV-Comparatives. This is my thought.

 

[Dexter_Morgan31]

You use the "corrupted" and "partially corrupted" criteria. It would be good to define them more clearly.

I will care to that in next test but i remember i wrote reports how machine corrupted

 

[Dexter_Morgan31]

In my opinion, Avast didn't detect PUPs because, if I remember correctly, this protection isn't enabled by default. Therefore, it's recommended to enable it (a recommendation AVLab also provides). AVLab also suggests enabling enhanced protection, so I think Avast's PUP results would have been different.

We wanted to make test in default settings.

 

[Dexter_Morgan31]

Running malware samples via a testing tool can matter. Some AVs can check files differently when executed directly from the Explorer, as compared to files executed by a trusted application. I do not insist that this is a case in your test, but no one has probably confirmed this in practice with this tool.

I dont think this really matter. Because it has no difference from opening file from explorer. It just automates processes. What is point that make you think this?

 

[Dexter_Morgan31]

The samples prior to executing were not 10000, an on-demand scan eliminated high number of them.

No we didn't scan samples, We directly open samples after open real-time protection..


If i misunderstood im sorry english isn''t my main language.

 

[Trident]

No we didn't scan samples, We directly open samples after open real-time protection..

Well next time try scan them first.

 

[Parkinsond]

Well next time try scan them first.

That is what the average user do usually, scan the file before execution.

 

[ForgottenSeer 123960]

I dont think this really matter. Because it has no difference from opening file from explorer. It just automates processes. What is point that make you think this?

Why Execution Method Matters

Modern antivirus (AV) and Endpoint Detection and Response (EDR) solutions use a multi-layered approach. Instead of simply asking, "Is this file malicious?" they are designed to ask, "Is this file behaving suspiciously, and what is the context of its execution?"

Behavioral Analysis and Heuristics

Security software monitors the entire chain of events. For instance, a file executed from explorer.exe (i.e., when a user double-clicks it) has a different process parent than a file launched by a custom testing script. A real-world malware attack might be launched by a macro in a Word document (winword.exe), which then launches PowerShell (powershell.exe), which in turn downloads and runs the final malicious file. Each step in this chain is a potential red flag for a good security product.

Therefore, a testing tool that executes the final payload directly can miss the entire chain of suspicious behavior that a robust security product is designed to detect.

 

[Dexter_Morgan31]

Why Execution Method Matters

Modern antivirus (AV) and Endpoint Detection and Response (EDR) solutions use a multi-layered approach. Instead of simply asking, "Is this file malicious?" they are designed to ask, "Is this file behaving suspiciously, and what is the context of its execution?"

Behavioral Analysis and Heuristics

Security software monitors the entire chain of events. For instance, a file executed from explorer.exe (i.e., when a user double-clicks it) has a different process parent than a file launched by a custom testing script. A real-world malware attack might be launched by a macro in a Word document (winword.exe), which then launches PowerShell (powershell.exe), which in turn downloads and runs the final malicious file. Each step in this chain is a potential red flag for a good security product.

Therefore, a testing tool that executes the final payload directly can miss the entire chain of suspicious behavior that a robust security product is designed to detect.

How do you think companies test samples? Open one-to-one in explorer? You really speaking absurdly. How we should open viruses?