Security News MRG Effitas 360 Assessment & Certification Programme Q1 2018

motox781

Level 10
Verified
Well-known
Apr 1, 2015
483
I agree with some posters above. Whenever a lab test comes out, I see a lot of:

"What?! This is wrong! My AV should be #1!" or "I don't like 'insert AV brand', they shouldn't be higher than my current AV".

Look, they are just tests. I tend to believe at least one of them is bound to be honest. I'm sure they have more skill and tools than me...so I tend to place some weight into them.

The testing methods between Lab 1 and Lab 2 may be widely different. Could be the reasons why scores are a little off between them. Hell, it could have been an lucky selection of tough malware that month....idk.

I like how they separate the detection into catagories. It shows you a sig, then BB, then how fast they respond in 24hrs, then a miss. It gives a good overview of the product being tested.
 
Last edited:

L0ckJaw

Level 19
Verified
Content Creator
Well-known
Feb 17, 2018
870
for example NSS Labs gave TM a detection rate of 99,8% in Quarter 1 2018 and 100% in Quarter 4 2017.
I will look the link up. MRG test i just dont believe when testing.
 

Ink

Administrator
Verified
Staff Member
Well-known
Jan 8, 2011
22,361
i got a virus from online with webroot.

bitdefender TS froze my PC. i uninstalled it. money wasted.
Use the Trial before you buy.

Advice falls on deaf ears, in the end it's your fault.
 

Evjl's Rain

Level 47
Thread author
Verified
Honorary Member
Top Poster
Content Creator
Malware Hunter
Apr 18, 2016
3,684
I truly don't understand why they have a "blocked in 24 hours" category. It's absurd. If malware executed and performed malicious activity then it should be counted as a miss. Detecting the sample in X number of hours after the infection took place is irrelevant if, like you said, your bank account was drained or all you valuable files were encrypted.
I simply think that "blocked in 24hrs" means they test the products again with a clean snapshot
1st test: miss
24 hours later on a new machine: block by signature or BB

I think they create this category because not many people get the malwares when they are fresh, <24hrs. People usually get them a day or a few days after they have been released
if someone is unlucky enough to get a malware very early and his machine is dead, the malware is probably submitted automatically to the vendor and they can analyze and create signature for it to protect the next victims
 
Last edited:
F

ForgottenSeer 58943

for example NSS Labs gave TM a detection rate of 99,8% in Quarter 1 2018 and 100% in Quarter 4 2017.
I will look the link up. MRG test i just dont believe when testing.

Trend Micro is quite strong IMO. I know an organization with 5,000 computers that hasn't had an infection in 5 years. Running Trend Micro Office Scan Advanced, and Fortigate on their edge. That's quite something, isn't it? For those that will come in and state that's the commercial product vs the home, they should know that as of 2018, the home versions have the same machine learning as the commercial offering.

I don't run Trend, but I believe it's a solid, competent, and protective product with a smashingly good interface. I recommend Trend ALL OF THE TIME to friends/family and it serves them well. Some family used to run Webroot and always got infected, Trend has been better for them.
 

Evjl's Rain

Level 47
Thread author
Verified
Honorary Member
Top Poster
Content Creator
Malware Hunter
Apr 18, 2016
3,684
for example NSS Labs gave TM a detection rate of 99,8% in Quarter 1 2018 and 100% in Quarter 4 2017.
I will look the link up. MRG test i just dont believe when testing.
MRGeffitas's results do reflect some findings in MT hub test
- trend micro has extremely poor signatures for new malwares, even when they are not truely zero-day. It was an everyday story when TM was tested in the hub
- TM has great BB, no doubt so this can compensate for its slowly reactive signatures
- still, TM was frequently infected because its BB couldn't block everything. 20 malwares tested, 0-4/20 by signatures, ~13-14/20 by BB

I do find other MRG's results match the hub test, except for a few vendors

for me, according to all tests I have seen, TM is meh. a good AV in general should have a fast signature
the Hub tests every AV in the worst case where malwares come from a flash drive and web filter is bypassed
 
Last edited:

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,040
for example NSS Labs gave TM a detection rate of 99,8% in Quarter 1 2018 and 100% in Quarter 4 2017.
I will look the link up. MRG test i just dont believe when testing.
That is normal in all such tests. The number of samples is too small, so the statistical error is important and hard to calculate. The above scores should be presented as (0.3 is the example of statistical error):
99,8% +- 0.3 and 100% +- 0.3 .
The 0.3 represents the fact that the samples used in the test did not reflect the real world scenario. You can make the error smaller only if you consider many tests.
 

motox781

Level 10
Verified
Well-known
Apr 1, 2015
483
for example NSS Labs gave TM a detection rate of 99,8% in Quarter 1 2018 and 100% in Quarter 4 2017.
I will look the link up. MRG test i just dont believe when testing.

Anything above 95% is pretty damn good. I'd take those chances.
 
5

509322

I truly don't understand why they have a "blocked in 24 hours" category. It's absurd. If malware executed and performed malicious activity then it should be counted as a miss. Detecting the sample in X number of hours after the infection took place is irrelevant if, like you said, your bank account was drained or all you valuable files were encrypted.

If AV test labs performed tests and made accurate, revealing statements that explained the truth of how atrocious some protections are, then they would have zero customers and quickly go out of business.

An AV test lab that never certifies any products has no customers. However, MRG Effitas has gone on record outside its routine certification testing and called the AV publishers on their weaknesses. Or at least one of their staff has gone on record regarding specific attacks which MRG does not test AVs against. LOL...

You're right - those yellow sections in the bars are meaningless. Most any user that has half a clue should be thinking "Those are all misses. Who cares that the system was remediated within 24 hours because the system is already compromised ?"
 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,040
MRGeffitas's results do reflect some findings in MT hub test
- trend micro has extremely poor signatures for new malwares, even when they are not truely zero-day. It was an everyday story when TM was tested in the hub
...
MRG Effitas tests show more statistics than MalwareTips tests. In the case of MRG Effitas is evident that TM signatures are not fast and additionally the cloud reaction is too slow to recompensate not fast signatures.
.
The MalwareTips tests miss the second factor, because detection test is not repeated.
.
In theory, the AV with slow signatures but very fast Cloud Reaction, can be better than another AV with faster signatures and very slow Cloud Reaction.
This is the same as in the below (extreme) analogy:
  1. The police in New York cannot disarm shooters until they kill someone (0% detection). But, then they can disarm the shooter immediately (one victim per one shooter).
  2. The police in Butt Town (1000 living people) can disarm 99% of shooters before they start killing (99% detection), but it takes 24 hours to disarm the shooter missed by them.
So if we consider 100 shooters, then it will be 100 victims in New York. But, one missed shooter in Butt Town can probably kill 50% of people in the town during 24 hours (500 victims).
The police in New York will score 0% shooter detection (in the standard test), but will have better protection as compared to Butt Town with 99% detection.:devil:
 
Last edited:
5

509322

MRG Effitas tests show more statistics than MalwareTips tests. In the case of MRG Effitas is evident that TM signatures are not fast and additionally the cloud reaction is too slow to recompensate not fast signatures.
.
The MalwareTips tests miss the second factor, because detection test is not repeated.
.
In theory, the AV with slow signatures but very fast Cloud Reaction, can be better than another AV with faster signatures and very slow Cloud Reaction.
This is the same as in the below (extreme) analogy:
  1. The police in New York cannot disarm shooters until they kill someone (0% detection). But, then they can disarm the shooter immediately (one victim per one shooter).
  2. The police in Butt Town (1000 living people) can disarm 99% of shooters before they start killing (99% detection), but it takes 24 hours to disarm the shooter missed by them.
So if we consider 100 shooters, then it will be 100 victims in New York. But, one missed shooter in Butt Town can probably kill 50% of people in the town during 24 hours (500 victims).
The police in New York will score 0% shooter detection (in the standard test), but will have better protection as compared to Butt Town with 99% detection.:devil:

There are even worse test methodologies. All anyone need do is take a close look at some of the commissioned tests performed by all the AV test labs. Having the customer dictate the terms of testing, in other industries, is not only considered a conflict of interest but also a lot of other things.
 
  • Like
Reactions: harlan4096
F

ForgottenSeer 58943

MRGeffitas's results do reflect some findings in MT hub test
- trend micro has extremely poor signatures for new malwares, even when they are not truely zero-day. It was an everyday story when TM was tested in the hub
- TM has great BB, no doubt so this can compensate for its slowly reactive signatures
- still, TM was frequently infected because its BB couldn't block everything. 20 malwares tested, 0-4/20 by signatures, ~13-14/20 by BB

I do find other MRG's results match the hub test, except for a few vendors

for me, according to all tests I have seen, TM is meh. a good AV in general should have a fast signature
the Hub tests every AV in the worst case where malwares come from a flash drive and web filter is bypassed

This is been my experience, which is why I said:

Trend's weakness I believe is the fact they can be very slow to respond to outbreaks and submissions. I've had to manually open a ticket and submit threats to them, then follow up over 3 days before it was detected. Trend is also poor with specific types of malware and some riskware. Otherwise, Trend is generally considered a solid product, just not the speediest reaction time. I'd almost always augment Trend with something like VoodooShield or OSArmor.

Trend is probably one of the most abysmal firms I have seen in outbreak response and submissions.. Slow as molasses, even when I send them a major new outbreak sample, it's often days before they even respond. Even Sophos responds to me within 60 minutes usually, but not Trend.

Trend is still the one I recommend to friends/family because it just works.. It's stable as hell. It's cheap, and the BB is usually more than sufficient for the average home user. Would I run Trend? Absolutely not. Period. Won't happen. Not because of the CIA thing, but because I think they are glacially slow in outbreak response and I do not like slow. They are getting better though, the machine learning they added a few months ago is starting to get effective.
 
  • Like
Reactions: Der.Reisende
D

Deleted Member 3a5v73x

Eh, I am grateful that G Data doesn't participate into every rigmarole test by independent AV labs, but I sometimes miss those charts by G Data, just for my own information. Based on this MRG Effitas test, I like that F-Secure been solid for a long time, but it would fail almost every VB100 test because most likely they will atleast flag 1 FP and that's already a failed test, well, at least in consumer versions it's not hard to exclude FP's, so it shouldn't be a problem for those running F-Secure and new 17.3 version is great. (y) There's not many F-Secure users in MalwareTips though, maybe because of it's simplicity? :unsure: I still consider F-Secure to be better than WD in all aspects. As long as you have backups/system image stored, it doesn't matter which reputable AV you are using if you are fine with companies privacy policy/price of the products, at the end of the day, as @Umbra said, it's just a "seat belt".
 
Last edited by a moderator:

Windows Defender Shill

Level 7
Verified
Well-known
Apr 28, 2017
326
On page four Webroot is not included in the certified level 2 list, but on page 13 it's listed as level 2.

Bitdefender, Avira and Kaspersky just had all the signatures out right.
 
  • Like
Reactions: Brie

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,040
The only one intriguing thing for me is the result of TrendMicro. From MalwareHub observation made by @evjls-rain and also MRG Effitas test Q1 2018, one can conclude that TrendMicro generally does not have the fastest signatures.
Yet, If one looks at results of AV-Comparatives Real-World tests and AV-TEST results, the opposite can be concluded. Both AV-Comparatives and AV-TEST methodology for the 0-day detection is based on the web threats like in MRG Effitas Q1 2018.
But, the situation is much clearer if we look at the previous TrendMicro results: Q4 2017, Q3 2017, Q2 2017, Q1 2017. In all those tests TrendMicro scored very high.
So, something went wrong for TrendMicro in Q1 2018 (caught by MRG Effitas and missed by other testing labs) or it had the bad luck with the samples used by MRG Effitas.
 

L0ckJaw

Level 19
Verified
Content Creator
Well-known
Feb 17, 2018
870
Exactly my point, thats why i do not trust the MRG test if all the other AV test companies give it high scores in Q1 2018.
Every Malware sample ( new ) i throw at it , it blocks it.
 

L0ckJaw

Level 19
Verified
Content Creator
Well-known
Feb 17, 2018
870
This is been my experience, which is why I said:



Trend is probably one of the most abysmal firms I have seen in outbreak response and submissions.. Slow as molasses, even when I send them a major new outbreak sample, it's often days before they even respond. Even Sophos responds to me within 60 minutes usually, but not Trend.

Trend is still the one I recommend to friends/family because it just works.. It's stable as hell. It's cheap, and the BB is usually more than sufficient for the average home user. Would I run Trend? Absolutely not. Period. Won't happen. Not because of the CIA thing, but because I think they are glacially slow in outbreak response and I do not like slow. They are getting better though, the machine learning they added a few months ago is starting to get effective.
Strange words @ForgottenSeer 58943 you recommend it to your family but you will not use it yourself ? I only recommend things i would use to my family otherwise if something happend i get the blame. In another post you say that you used to use TM before.
And ditched it due to the CIA thing. It looks to me you are running in circles and dont know it anymore ?
 
D

Deleted Member 3a5v73x

Exactly my point, thats why i do not trust the MRG test if all the other AV test companies give it high scores in Q1 2018.
Every Malware sample ( new ) i throw at it , it blocks it.
It doesn't take more than 10mins of work in Hybrid-Analysis to get some new malware slip by Trend Micro, of course nothing major like Ransomware, but still. Personally, I agree that TM is stable and does pro-actively catch more nasties than with sigs. I am also skeptic about this MRG test's TM charts, TM isn't that bad.
 
  • Like
Reactions: Der.Reisende
F

ForgottenSeer 58943

Strange words @ForgottenSeer 58943 you recommend it to your family but you will not use it yourself ? I only recommend things i would use to my family otherwise if something happend i get the blame. In another post you say that you used to use TM before.
And ditched it due to the CIA thing. It looks to me you are running in circles and dont know it anymore ?

I also recommend Dell to my family, but I would never buy a Dell. The reason is simple, Trend works, and wouldn't give them hassles, popups or an overabundance of false positives and it has a 'friendly' interface for simple people. Everything that would suit them fine and when it suits them it means less complaints to me. They can readily get support from Trend which keeps them from reaching out to me.

I have a lot of reasons I wouldn't run Trend myself though. Some of which I've already pointed out here and in other threads and don't care to rehash.
 
  • Like
Reactions: Der.Reisende

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top