MRG Effitas MRG Effitas 360 Degree Assessment & Certification – Q3 2020

[SeriousHoax]

This Programme is called “360 Assessment & Certification” since it tests the security applications capabilities with a full spectrum of attack vectors. In the 360 Assessment, trojans, backdoors, spyware, financial malware, ransomware and “other” malicious applications are all used. Along side the traditional In-The-Wild (ITW) file-based attacks, our evaluation also contains scenarios where fileless cases and exploitation techniques, live botnets and financial malware simulators are also applied. Besides the malicious attacks, in order to evaluate the practical accuracy of AV products, they were exposed to potentially unwanted applications (PUA or Greyware) and clean files (FP) as well. Additionally, besides security capabilities tests, our assessment measured the footprint each security software on a computer’s performance.

Read the full report here

 

[fabiobr]

ESET is doing very well.

 

[Vitali Ortzi]

ESET is doing very well.

i love eset but no kaspersky

 

[starsfighter]

Nice Eset as expected using their product for a long time never fail me

 

[Pat MacKnife]

MS defender also doing very well, much better on Exploit/Fileless than 8 months ago and a 100 % for ransomware.

 

[Andy Ful]

MRG Effitas tests (Q4 2019-Q3 2020)
Cumulative results of all 4 tests from 2 last years:
Microsoft Word - MRG_Effitas_2019Q4_360.docx (mrg-effitas.com) (mrg-effitas.com)
Microsoft Word - MRG_Effitas_2020Q1_360.docx (mrg-effitas.com) (mrg-effitas.com)
Microsoft Word - MRG_Effitas_2020Q2_360.docx (mrg-effitas.com) (mrg-effitas.com)
Microsoft Word - MRG_Effitas_2020Q3_360.docx (mrg-effitas.com) (mrg-effitas.com)

Exploit/Fileless
-----AV vendor-------------Missed samples----------------------------------------------
Bitdefender Endpoint Security.....................0+0+0+0.
Eset Endpoint Security.................................0+0+0+0
Microsoft Defender......................................1+1+0+0 ASR rules enabled
*Kaspersky Small Office Security................0+1+*0+*1
F-Secure Computer Protection Premium......2+2+0+0.
Symantec Endpoint Protection Cloud...........0+3+1+1
Avast Business Antivirus...............................1+1(2)+2+1(1)
Trend Micro Security......................................0+4+3+2
Avira Antivirus Pro..........................................2+4+3+3
*McAfee Endpoint Security............................4+3+3+*3

The AVs are sorted by the total number of missed samples. The numbers in brackets for Avast are related to detected samples that were not blocked/quarantined. Kaspersky did not participate in the last two tests, so I included the average of the two first tests (labeled by asterisks). McAfee did not participate in the last test, so the average from the first 3 tests was added.

MRG Effitas tests (Q4 2019-Q3 2020)
In the Wild + PUA + Ransomware + Financial
--------AV vendor-------------------------------------- Missed samples
Eset Endpoint Security.................................=2(3)
*Kaspersky Small Office Security................=*2(4)
Bitdefender Endpoint Security.....................=3(5)
Symantec Endpoint Protection Cloud.........=7(2) <--------- corrected missed PUA
Microsoft Defender......................................=8(5) ASR rules enabled
Avast Business Antivirus...............................=10(17)
F-Secure Computer Protection Premium.....=27(14)
*McAfee Endpoint Security............................=*25(30)
Avira Antivirus Pro....................... ..................=51(24)
Trend Micro Security..................... ................=113(9)


The numbers in the brackets are the samples missed but blocked in 24 hours. Kaspersky did not participate in the last two tests test, but I added the average from the two first tests. McAfee did not participate in the last test, so the average from the first three tests was added. The AVs are sorted by the sum of missed samples: not blocked in 24 hours + blocked in 24 hours.

Warning!
The results from the particular quarter do not reflect the real protection in the wild related to this quarter. This follows from a too-small number of tested samples. The AVs in green cannot be probably distinguished from each other for the same reason. For example in the first 3 tests (Q4 2019-Q2 2020), Bitdefender was after Microsoft Defender (In The Wild + PUA + Ransomware + Financial).

Edit.
Calculating the total number of missed samples from different categories is not a proper statistical method. I tried to use it to see any difference between top AVs. But, this difference is too small anyway.

 

[Lenny_Fox]

I run Kaspersky Cloud Free with WD Attack Surface Reduction Rules and extra hardening of Edge with WD Exploit Protection (@oldschool's settings copied)

 

[Andy Ful]

I run Kaspersky Cloud Free with WD Attack Surface Reduction Rules and extra hardening of Edge with WD Exploit Protection (@oldschool's settings copied)

WD Attack Surface Reduction Rules can work only when WD real-time protection is enabled.

 

[MacDefender]

Wow, this sure looks very promising for ESET! Congrats to them

 

[Lenny_Fox]

WD Attack Surface Reduction Rules can work only when WD real-time protection is enabled.

My bad I had read that exploit protection still worked except for protected folders. Since all 3 are part of exploit protection in GPO, I assumed ASR worked even when disabling Microsoft Defender.

 

[SeriousHoax]

Quoting user "itman" from Wilder forum. He could be right about why some vendors have not participated in this test after Q1.

As I see a lot of vendors pulled out after Q1 tests. I guess that there was something we don't know about.

Tight budgets due to COVID crisis would be one reason.

Absent in current test are Kaspersky, CloudStrike, Sophos, and McAfee. Aside from Kaspersky, the rest were ranked in the bottom tier protection-wise. No Endpoint AV vendor is going to stay in a public comparative where they rank poorly. That results in a $$$ hit for lost sales revenue for their "cash cow;" commercial software sales.

 

[Andy Ful]

Quoting user "itman" from Wilder forum. He could be right about why some vendors have not participated in this test after Q1.

I am not convinced by his arguments. COVID-19 forced many people to work at home for companies. This requires more comprehensive security, so the AV vendors probably do not suffer much from the pandemic event.
Furthermore, such a situation is the best moment to show that the product is strong. So, the absence of Kaspersky is still strange to me.

 

[Minimalist]

Maybe they have yearly "subscription" for their tests and they raised prices for this year, so many vendors decided it was not worth participating any more. Who knows.

 

[upnorth]

I am not convinced by his arguments. COVID-19 forced many people to work at home for companies. This requires more comprehensive security, so the AV vendors probably do not suffer much from the pandemic event.

I 100% agree. Simply ignore it.

@Andy Ful , Appendix 2 on page 36 in the pdf, was any of the vendors testing with that?

 

[Digmor Crusher]

I am not convinced by his arguments. COVID-19 forced many people to work at home for companies. This requires more comprehensive security, so the AV vendors probably do not suffer much from the pandemic event.
Furthermore, such a situation is the best moment to show that the product is strong. So, the absence of Kaspersky is still strange to me.

You would think that would be the case.

 

[Andy Ful]

I 100% agree. Simply ignore it.

@Andy Ful , Appendix 2 on page 36 in the pdf, was any of the vendors testing with that?

These non-default settings were used in MRG Eeffitas 360 degree assessment tests, but I do not know if any other AV Lab could use these particular settings. The config WD + ASR was tested on MH for several months.