MRG Effitas MRG Effitas 360 Degree Assessment & Certification – Q3 2020

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

SeriousHoax

Level 41
Thread author
Verified
Top poster
Well-known
Mar 16, 2019
3,095
This Programme is called “360 Assessment & Certification” since it tests the security applications capabilities with a full spectrum of attack vectors. In the 360 Assessment, trojans, backdoors, spyware, financial malware, ransomware and “other” malicious applications are all used. Along side the traditional In-The-Wild (ITW) file-based attacks, our evaluation also contains scenarios where fileless cases and exploitation techniques, live botnets and financial malware simulators are also applied. Besides the malicious attacks, in order to evaluate the practical accuracy of AV products, they were exposed to potentially unwanted applications (PUA or Greyware) and clean files (FP) as well. Additionally, besides security capabilities tests, our assessment measured the footprint each security software on a computer’s performance.
3.PNG
1.PNG
2.PNG

Read the full report here
 

Andy Ful

Level 79
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
6,834
MRG Effitas tests (Q4 2019-Q3 2020)
Cumulative results of all 4 tests from 2 last years:
Microsoft Word - MRG_Effitas_2019Q4_360.docx (mrg-effitas.com) (mrg-effitas.com)
Microsoft Word - MRG_Effitas_2020Q1_360.docx (mrg-effitas.com) (mrg-effitas.com)
Microsoft Word - MRG_Effitas_2020Q2_360.docx (mrg-effitas.com) (mrg-effitas.com)
Microsoft Word - MRG_Effitas_2020Q3_360.docx (mrg-effitas.com) (mrg-effitas.com)

Exploit/Fileless
-----AV vendor-------------Missed samples----------------------------------------------
Bitdefender Endpoint Security.....................0+0+0+0.
Eset Endpoint Security.................................0+0+0+0
Microsoft Defender......................................1+1+0+0 ASR rules enabled
*Kaspersky Small Office Security................0+1+*0+*1

F-Secure Computer Protection Premium......2+2+0+0.
Symantec Endpoint Protection Cloud...........0+3+1+1
Avast Business Antivirus...............................1+1(2)+2+1(1)
Trend Micro Security......................................0+4+3+2
Avira Antivirus Pro..........................................2+4+3+3
*McAfee Endpoint Security............................4+3+3+*3

The AVs are sorted by the total number of missed samples. The numbers in brackets for Avast are related to detected samples that were not blocked/quarantined. Kaspersky did not participate in the last two tests, so I included the average of the two first tests (labeled by asterisks). McAfee did not participate in the last test, so the average from the first 3 tests was added.

MRG Effitas tests (Q4 2019-Q3 2020)
In the Wild + PUA + Ransomware + Financial

--------AV vendor-------------------------------------- Missed samples
Eset Endpoint Security.................................=2(3)
*Kaspersky Small Office Security................=*2(4)
Bitdefender Endpoint Security.....................=3(5)

Symantec Endpoint Protection Cloud.........=7(2) <--------- corrected missed PUA
Microsoft Defender......................................=8(5) ASR rules enabled
Avast Business Antivirus...............................=10(17)
F-Secure Computer Protection Premium.....=27(14)
*McAfee Endpoint Security............................=*25(30)
Avira Antivirus Pro....................... ..................=51(24)
Trend Micro Security..................... ................=113(9)


The numbers in the brackets are the samples missed but blocked in 24 hours. Kaspersky did not participate in the last two tests test, but I added the average from the two first tests. McAfee did not participate in the last test, so the average from the first three tests was added. The AVs are sorted by the sum of missed samples: not blocked in 24 hours + blocked in 24 hours.

Warning!
The results from the particular quarter do not reflect the real protection in the wild related to this quarter. This follows from a too-small number of tested samples. The AVs in green cannot be probably distinguished from each other for the same reason. For example in the first 3 tests (Q4 2019-Q2 2020), Bitdefender was after Microsoft Defender (In The Wild + PUA + Ransomware + Financial).

Edit.
Calculating the total number of missed samples from different categories is not a proper statistical method. I tried to use it to see any difference between top AVs. But, this difference is too small anyway.
 
Last edited:

SeriousHoax

Level 41
Thread author
Verified
Top poster
Well-known
Mar 16, 2019
3,095
Quoting user "itman" from Wilder forum. He could be right about why some vendors have not participated in this test after Q1.
As I see a lot of vendors pulled out after Q1 tests. I guess that there was something we don't know about.
Tight budgets due to COVID crisis would be one reason.

Absent in current test are Kaspersky, CloudStrike, Sophos, and McAfee. Aside from Kaspersky, the rest were ranked in the bottom tier protection-wise. No Endpoint AV vendor is going to stay in a public comparative where they rank poorly. That results in a $$$ hit for lost sales revenue for their "cash cow;" commercial software sales.

 

Andy Ful

Level 79
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
6,834
Quoting user "itman" from Wilder forum. He could be right about why some vendors have not participated in this test after Q1.
I am not convinced by his arguments. COVID-19 forced many people to work at home for companies. This requires more comprehensive security, so the AV vendors probably do not suffer much from the pandemic event.
Furthermore, such a situation is the best moment to show that the product is strong. So, the absence of Kaspersky is still strange to me.
 

upnorth

Moderator
Verified
Staff member
Malware Hunter
Well-known
Jul 27, 2015
4,783
I am not convinced by his arguments. COVID-19 forced many people to work at home for companies. This requires more comprehensive security, so the AV vendors probably do not suffer much from the pandemic event.
I 100% agree. Simply ignore it.

@Andy Ful , Appendix 2 on page 36 in the pdf, was any of the vendors testing with that? :unsure:
 

Digmor Crusher

Level 13
Verified
Top poster
Well-known
Jan 27, 2018
647
I am not convinced by his arguments. COVID-19 forced many people to work at home for companies. This requires more comprehensive security, so the AV vendors probably do not suffer much from the pandemic event.
Furthermore, such a situation is the best moment to show that the product is strong. So, the absence of Kaspersky is still strange to me.
You would think that would be the case.
 

Andy Ful

Level 79
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
6,834
I 100% agree. Simply ignore it.

@Andy Ful , Appendix 2 on page 36 in the pdf, was any of the vendors testing with that? :unsure:
These non-default settings were used in MRG Eeffitas 360 degree assessment tests, but I do not know if any other AV Lab could use these particular settings. The config WD + ASR was tested on MH for several months.