AV-Comparatives Do you understand Malware Protection tests?

[Andy Ful]

Please, look at this chart:

The above chart comes from the AV-Comparatives Malware Protection test (September 2020, over 10000 samples tested). We can see the extremely poor result for Trend Micro.
Similar results can be seen in the AV_Comparatives test from March 2020 and in the test made by AVLab in May 2020 (over 1000 samples tested):

The test from AVLab is even more striking because all other AVs scored 100% detection (Trend Micro scored only 89%). I think that most people would quickly reach one of the below conclusions:

1. Trend Micro is crap.
2. The tests are unreliable.

But, the purpose of my post is to show the opposite:

1. Trend Micro can provide better protection than many tested AVs and still get such poor detection results.
2. The tests are correct (within the limits of testing procedures).

Let's begin.
The first observation is that these results look like all tested AVs already knew the malware samples except Trend Micro. If so, then Trend Micro will get a poor result because it sees many malware samples as never-seen-before. But, how it can be possible? In the AV-Comparatives tests, the samples are mostly a few weeks old, and in the AVLab tests a few days old.

Here is an example. Many in-the-wild attacks start the infection chain from non-EXE files. Trend Micro can prevent these files from delivering and executing the EXE payloads by using a kind of file reputation - the signature of the EXE payload is not created.
Most AVs (on default settings) often allow the initial non-EXE malware to deliver and execute the EXE payloads. Next, these AVs are forced to fight EXE payloads. Some payloads are blocked. But some others can compromise the system and then AVs will create signatures. So, if the EXE payloads are tested after a few days (weeks) then they will be easily detected by most AVs except Trend Micro.
Furthermore, Trend Micro can protect better because the payloads will not enter the system. In such a scenario Trend Micro can provide better protection and still will get poor detection.

So, the strange results of Trend Micro can be related to the below factors:

1. Most malware attacks do not start from EXE files.
2. In Malware Protection tests the samples are mostly the EXE files.
3. Some AVs have features that can prevent the initial non-EXE malware from delivering & executing the EXE payload (the signature is not created). These AVs can provide very good protection and still get poor detection in Malware Protection tests.

Anyway, if the user intentionally downloads tons of cracks and pirated software (especially via torrents), then for him most attacks will start from EXE files. In such a case the protection of Trend Micro will be worse as compared to most good AVs.

Another AV that can have a similar detection problem is Microsoft Defender. For example, It often uses extended protection against MS Office macros (via AMSI and ASR rules in the business environment). The difference is that Defender has found a way to add the signatures for EXE payloads (that never compromised the real machines) much faster than Trend Micro. So, it can get a good detection while testing a few weeks EXE samples (like in the AV-Comparatives tests). But still, the clear difference can be seen in some AVLabs tests for a few days EXE samples, for example:

In this AVLab test from March 2020, the Defender scored 97% and all other AVs scored 100%.
The reason is similar to the case of Trend Micro. In many cases, the malicious attacks are blocked by AMSI and ASR rules + some other features, and Defender does not have to fight so many EXE payloads. The users are well protected in-the-wild, but the detection in some tests can be still worse as compared to other AVs. Of course, such detections have nothing to do with the protection in-the-wild.

 

[blackice]

A great perspective and food for thought as always, @Andy Ful . Thank you!

 

[RoboMan]

I expect no less than you Andy. As usual, great input and shared knowledge. Thank you!

 

[Venustus]

Thak you @Andy Ful for the analysis

 

[monkeylove]

Also, there's a test methods page here:

Malware Protection Test September 2020

The Malware Protection Test September 2020 assesses program’s ability to protect a system against malicious files before, during or after execution.

www.av-comparatives.org

 

[tipo]

Wow! Very informative post. Thanks!

 

[ErzCrz]

Very well explained!

The file reputation thing with TM is much like your voodoo shield or Comodo's auto-containment though many products use some sort of sandbox these days.

These tests are informative but these days I find less swayed from results to switch or move products.

 

[Cortex]

The top chart at starts at 95% which skews the representation unless you take it into account - I've noticed over the years the charts on AV tests have often started at a higher & higher percentage this makes small increments look much larger - Thanks

 

[Andy Ful]

The top chart at starts at 95% which skews the representation unless you take it into account - I've noticed over the years the charts on AV tests have often started at a higher & higher percentage this makes small increments look much larger - Thanks

Yes. The 95% starting point in the first chart is useful for visualization. One can see without counting that the sum of the missed samples of all other AVs is still much smaller than for Trend Micro alone.

 

[Lenny_Fox]

In the AV-Comparatives tests, the samples are mostly a few weeks old..

This is not true, I have had the chance to visit AV-Comparatives during my study and the validation process of AV-comparatives samples only takes a few minutes to an hour maximum (to validate a new found example is indeed real malware). They have a honeypot network themselves and AV-comparatives works with the University of Innsbruck and an international network of professional malware hunters.

Take away.
The samples found by the AV-companies and malware hunter networks are shared (sometimes with a small delay in time), that is why it is hard to find zero-day smples which manage to pass an antivirus on a fully patched Windows10 PC when performing real world tests (using the full potential of AntiVirus solutions). This is the reason AV-companies score high, not the age of the samples. Maybe in the previous century samples needed to be old to be detected, but with current technology the (herd) detection response is also only a few minutes to an hour. Think about it: when the whole world goes digital and news spreads in seconds around the world, why would AV-companies detect new samples in the snail-mail tempo. AV-test in days and AV-comparatives in weeks (because Austria has higher mountains samples travel slower in Austria than in Germany?)

Disclaimer
When you think I was charmed and fooled by their marketing man I have to disappoint you. AV-Comparatives does not have a marketing guy (and I am a marketeer myself, so I should recognize marketing magic). The tech guy of AV-C was explaining in detail how they separate testing environments and run tests in paralel to prevent samples being shared and giving one (earlier tested) AV and advantage over another (later tested) AV. The facts about age samples and testing and validation process were told by a customer of AV-comparatives (some one from a British consumer protection organization, that organization used AV-C for their own test reports). I checked that guy on LinkedIn so he really was from a consumer defense/interests/protection organization (I don't know the English word for it, in Dutch "consumentenbond" type of organization). I had lunch with the guy after the demo, because we both had to go back to the airport (***)

*** when you are wondering how a poor student can afford to fly home and why a marketing student visited a security company ***
The reason is that I had saved money to surprise my girlfriend for a concert of her favorite band in London. This surprise sort of clashed with my mandatory international study visits. To arrive in London in time I had to fly and did not travel back home with my study mates in a touring bus. I only selected AV-comparatives, because their office is only 15 minutes away from the Airport (and Innsbruck is a regional airport, with a relatively fast and easy check-in procedure), so I could meet in time with my girlfriend. My marketing study coach accepted my story about the security industry being a multi billion market with tech nerds doing the marketing, so a promising new market for marketing professionals.

 

[Nagisa]

Anyway, if the user intentionally downloads tons of cracks and pirated software (especially via torrents), then for him most attacks will start from EXE files. In such a case the protection of Trend Micro will be worse as compared to most good AVs.

Though, I myself have never encountered a non-EXE malware while using my computer for daily reasons. Many people download tools, games, cracks, etc. in a compressed/encrypted archives and this is one of the most frequent way how someone gets infected.

 

[Zeitas]

Personally, I think you should take such tests with a pinch of salt.

 

[Andy Ful]

Though, I myself have never encountered a non-EXE malware while using my computer for daily reasons. Many people download tools, games, cracks, etc. in a compressed/encrypted archives and this is one of the most frequent way how someone gets infected.

Yes, that is true. Sometimes they even suspect that the file is probably malicious, but they think that the AV can save them. After several tries, they are finally infected (with any AV installed).

 

[Andy Ful]

Lenny_Fox,​

You probably think about AV-Comparatives Real-World tests. In the Malware Protection tests, the samples are a few weeks old (as it is written on the top of the chart):

 

[Adrian Ścibor]

@Andy Ful Everyone knows you're a fan of Microsoft Defender and you're always trying to defend the software. Here is another comparison from AVLab based on a few years of testing by different labs (2018-2020). Sorry, only in Polish:

The prob and the cons of Microsoft Defender vs Microsoft ATP, and the native protection of Windows 10 Home vs Pro.

Please use the translator: Korzystasz z Microsoft Defender? Czy wiesz co robisz? - AVLab

PDF: https://avlab.pl/wp-content/uploads/2020/11/AVLab-Microsoft-Defender-w-testach.pdf

 

[Andy Ful]

@Andy Ful Everyone knows you're a fan of Microsoft Defender and you're always trying to defend the software.

... or maybe a Kaspersky fan?
https://malwaretips.com/threads/malware-and-antivirus-needed.106771/post-930270

The anomalous test results are mostly related to Trend Micro. They are much more striking as compared to Defender. I noticed these anomalies when preparing the statistics of popular AVs. You can look at these statistics here:
https://malwaretips.com/threads/the-best-home-av-protection-2019-2020.106485/post-927440

This issue was discussed on two or three MT threads, but the explanation presented by Trend Micro staff is not convincing. The details can be read in the @McMcbrad post (he contacted the staff):
https://malwaretips.com/threads/the-best-home-av-protection-2019-2020.106485/post-927932

By the way, what is your possible explanation of such anomalous results for Trend Micro in AV-Comparatives and AVLab tests? (we can skip the Defender theme because it is not important in this thread)
Does AVLab test only *.exe files and MS Office documents in “The Advanced In The Wild Malware Test” ?
I do not insist on my explanation, but for now, I do not know another one.

 

[Lenny_Fox]

Lenny_Fox,​

You probably think about AV-Comparatives Real-World tests. In the Malware Protection tests, the samples are a few weeks old (as it is written on the top of the chart):

View attachment 254725

In real world test the samples are used as soon as they are verified (to be true malware). So most samples are 10 minutes old.

It does not tell the samples are few weeks old, the samples are collected in the last few weeks. Did you really think they sit on those samples for a few weeks? Something got lost in translation

Samples are used as soon as they are verified malware. Write an mail to AV-C they will confirm it.

 

[Andy Ful]

Samples are used as soon as they are verified malware. Write an mail to AV-C they will confirm it.

Did you notice the note in the report: "Products were tested at the beginning of September with default settings and using their latest updates"? So, the malware samples were gathered from the last few weeks in August and tested in September.

 

[Wasar]

Very informative! Thanks for sharing

 

[Andy Ful]

It is good that @Adrian Ścibor has entered this thread. It would be also interesting to invite someone from the staff of Trend Micro or AV-Comparatives. @McMcbrad and @Lenny_Fox can you do it, or help me to do it?

I would like to clarify that such tests as in the OP are probably the most interesting. It is improbable to get such results by chance. So, this proves that the AV-Comparatives and AVLab have somehow found the potential weak points in the AV protection. But, we do not know how these weak points may impact the overall protection in the wild (in the home environment). For example, in the AV-Test tests, there are no such anomalous results for Trend Micro both for fresh and older samples. In fact, during the two-year period (2019-2020), Trend Micro was compromised 5 times by the web-originated malware and 0 times by the older widespread and prevalent malware (only Norton got better results in the first category).