Battle Planned: Real-world Test of Trend Micro, ZoneAlarm, Eset and Webroot

[Divine_Barakah]

That’s not the problem, my laptop needs a motherboard apparently. I gotta buy a new one. I am happy to update all threat hunting, if anyone wants to fire up a VM.

Parallels for Mac doesn’t work, it doesn’t allow kernel-level drivers (seems like).

No rush. Get well soon and I am sorry your laptop failed you.

 

[Andy Ful]

@Trident,

From the OP it follows that the test will not be Real-World. Of course, it does not mean that your test will not be interesting.
Making a Real-World test is hardly possible for a single tester.

1. You have to keep the pule of samples realistic and representative, preserving the ratio of file types and malware types used in the wild in the testing period.
   This can be a serious challenge even for professional AV testing labs. For example, one shortcut as an initial vector can deliver different malware payloads hour after hour. Some AVs can detect that shortcut and protect against those payloads without knowing/detecting them. If you skip the shortcut in the test, the AV protection scorings will be biased. The same can be true for documents and scripts (including JavaScript code embedded/attached in emails).
2. You must test the malware immediately after it is seen in the wild to avoid dead samples. Dead samples are malware that infect people in the wild for a short time and disappear before the test is done (cannot infect anyone except the tested computer). With a delay of one day, most samples will be dead.
3. Dead samples can produce contradictory results because most of such samples can infect only one or two computers in the wild. For example, if the Bitdefender user was infected in the wild, Bitdefender can create the signature and detect the malware in the test. So for Bitdefender, that malware was initially FUD (Fully UnDetecteble), but not FUD at the test time. On the contrary, the same malware is still FUD at the test time for other AVs. To make this issue statistically insignificant one must use many random samples and fulfill the requirement from point 1. above.

All the above points (and some more) can make the test results different from the in-the-wild protection and produce unrealistic comparative results.

 

[Andy Ful]

While lab-based testing is valuable for providing a controlled environment and standardized measurements, real-world testing is indispensable for truly assessing a security product's full potential.

Lab-based testing (AV-Test, AV-Comparatives, SE Lab) in the Consumer category is mainly real-world. The real-world and non-real-word results are collected separately and can be seen on the AV lab's websites.

 

[silversurfer]

@Trident,

From the OP it follows that the test will not be Real-World. Of course, it does not mean that your test will not be interesting.
Making a Real-World test is hardly possible for a single tester.

1. You have to keep the pule of samples realistic and representative, preserving the ratio of file types and malware types used in the wild in the testing period.
   This can be a serious challenge even for professional AV testing labs. For example, one shortcut as an initial vector can deliver different malware payloads hour after hour. Some AVs can detect that shortcut and protect against those payloads without knowing/detecting them. If you skip the shortcut in the test, the AV protection scorings will be biased. The same can be true for documents and scripts (including JavaScript code embedded/attached in emails).
2. You must test the malware immediately after it is seen in the wild to avoid dead samples. Dead samples are malware that infect people in the wild for a short time and disappear before the test is done (cannot infect anyone except the tested computer). With a delay of one day, most samples will be dead.
3. Dead samples can produce contradictory results because most of such samples can infect only one or two computers in the wild. For example, if the Bitdefender user was infected in the wild, Bitdefender can create the signature and detect the malware in the test. So for Bitdefender, that malware was initially FUD (Fully UnDetecteble), but not FUD at the test time. On the contrary, the same malware is still FUD at the test time for other AVs. To make this issue statistically insignificant one must use many random samples and fulfill the requirement from point 1. above.

All the above points (and some more) can make the test results different from the in-the-wild protection and produce unrealistic comparative results.

@Andy Ful Don't get me wrong, no offense
Seriously, why someone use the name "Dead samples" that sounds at first for the average forum users like a sample doesn't work anymore unlike the purpose who created the sample to be malicious. Evasive samples are different as that fact is not a dead sample. IMO "Dead samples" is rather confusing or partially misleading...

 

[Andy Ful]

@Andy Ful Don't get me wrong, no offense
Seriously, why someone use the name "Dead samples" that sounds at first for the average forum users like a sample doesn't work anymore unlike the purpose who created the sample to be malicious.

I do not know if "dead" is the best choice, I used it as the opposite of the term "alive" = present on the attacker servers or malicious domains used in the attack. Many samples can be "alive" in the wild for a short time (several minutes up to a few hours, and disappear). The short "alive" time causes a low prevalence of the sample (some AVs are impacted by the sample in the wild, and some are not). The behavior of the sample is often different at the time when it is "alive" in the wild and at the test time when it is "dead". During the test time, some AVs detect the sample by a signature created on the basis of the "alive" behavior. Other AVs will try to detect the sample on the basis of the "dead" behavior. To avoid this, one must test the malware as soon as possible. Any delay increases the error, so more samples must be tested to keep that error sufficiently low.

 

[Andy Ful]

One can probably use the term "retired" instead of "dead".

 

[Trident]

One can probably use the term "retired" instead of "dead".

The samples were very much fresh and alive, as well as all of them were tested it 3 different sandboxes. I can update them any time, any day. Obviously now, in this moment, these samples may be dead. Like my motherboard. Which is what suspended the test.

 

[silversurfer]

One can probably use the term "retired" instead of "dead".

That would even more confusion for most people. Retired can't be a valid term for malware samples, if professionals would say the opposite then good luck for the job

 

[Shadowra]

I'd like to say a few words.
I have read most of the messages, which are very pertinent.

But when I see AV-Comparative and others, I have to intervene.
Please know that I have no animosity towards these test companies, nor towards my fellow testers. But when I don't like it, I tend to say things.

Personally, I give no credit to Av-Comparative, Av-Tests etc...
Not because they're bad, but I very rarely see their test protocols, which for me should be mentioned...
These test editors very rarely execute, contenting themselves with contextual analysis which, frankly, doesn't make a test.

=> Using ONLY EXE files is not enough to make a test.
There are different systems for infecting a machine.
The classic e-mail trap with a PDF/ONE/JS or VBS file that downloads and launches a payload.
There are also exploit sites called Drive-by Download, which will target all vulnerabilities at the same time until the malware is installed.
And don't forget bloated or re-packed files, which can contain malicious code.

=> Relying ONLY on an analysis is TOTALLY insufficient for a test.
Antivirus software doesn't just have an antimalware engine!
There are several shields to counter a threat.
Whether it's basic protection, HIPS/IDS or anti-attack shields, these protections are not often highlighted in these tests.

If one day you'd like to see my test protocol, I'd be delighted to show it to you!

The only one I'll give credit to is @Adrian Ścibor , because it totally covers what I've mentioned.

Enjoy

 

[Andy Ful]

That would even more confusion for most people. Retired can't be a valid term for malware samples, if professionals would say the opposite then good luck for the job

Anyway, that kind of sample must be named in some way. What do you suggest?

 

[Trident]

Isolated module testing

I’m not sure where in this entire discussion, isolated module testing was mentioned. How the modules interact together, you can’t test without debugging, which for Check Point I know how to enable. The rest of the vendors are neither gonna allow us to debug the product, nor will they reveal to you and me how exactly the modules integrate and all this low level info.

It is not necessary to overcomplicate and reinvent the wheel.
A solution either protects or doesn’t protect.

If it doesn’t protect, there is no point putting it in a cradle and singing lullaby’s. The vendor needs to do better.

 

[Trident]

The issue stems from the fact that phishing links and desktop executable's are not fully tested, as evaluations are limited to certain modules. This results in inaccurate negative verdicts for products that fail these partial assessments. This results in a distorted view of the products' capabilities.

I don’t understand what you are talking about. All malware was tested and sandbox reports were linked. Now I will take the site down because it seems to be causing a lot of excitement. Nothing was limited to any modules and no modules were planned to be switched off.

The phishing links were fresh, some of them from my inboxes, but they don’t last forever.

I don’t wanna be dragged in this discussion anymore.

 

[Trident]

I would appreciate a comprehensive response to my inquiry about effective strategies for assisting users in making informed choices.

You need to realise that neither me, nor anybody else on a public forum, owes you, or anybody else anything.

Everything that we do here is done from the goodness of our hearts and based on our good will.

The methodology of the test was clearly published and so was the malware. This makes the test transparent.

I’ve taken into account users feedback, including yours to an extent. This is it.

 

[Trident]

While no one has explicitly claimed anything is owed, it's crucial for the forum to address the misleading labeling of products as 'failed' when they haven't been fully tested. This misinformation leads to user disappointment and unnecessary anxiety about their security.

Nobody is making anyone feel anxious. AV-comparatives for a few years has been putting Trend Micro at the rock bottom. Users keep buying Trend Micro every day and Trend Micro continues to participate in the tests, although they can drop at any time.

It is fair to say that:
Trend Micro doesn’t care about the test
Users don’t care about the test.

If a solution fails to detect malware, then it is a fail. That’s how testing works.

 

[Trident]

The inaccurate testing methods impact a wide range of products and customers. While you're concerned about specific items, it's important to address the broader issue.

I don’t see the problem. I’m an exec. If you want me to think that there is a problem and come up with solution, prove to me that:

• Problem exists. Audience is anxious and scared, and changes security software more frequently than it normally would, because of test results.
• Explain what’s wrong with current testing methodologies, why they should be disapproved. Why we need to reinvent the wheel.

If I see that something is a problem just for one person or for a very small minority, for me this is mostly matter of personal subjective opinions.
Not a real problem that needs solving.

 

[Harputlu]

We are chatting with the test that was not done. It's been 11 days . I wish our friend would do the test and publish it and talk like that.

 

[Jan Willy]

We are chatting with the test that was not done.

I think I already know the results.
Nevertheless, it's nice entertainment. Otherwise this forum would be as boring as that other one.

 

[Andy Ful]

We should be kind to the author of this thread. Testing malware nowadays is a hard and thankless job.

 

[TuxTalk]

We should be kind to the author of this thread. Testing malware nowadays is a hard and thankless job.

We are except some "know it better types" here on the forum

 

[TuxTalk]

We are chatting with the test that was not done. It's been 11 days . I wish our friend would do the test and publish it and talk like that.

Ok you buy him a new laptop, so he can do it, you happy and he is happy !