AV-Comparatives Advanced Threat Protection Test 2022 – Consumer

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

SeriousHoax

Level 47
Thread author
Well-known
Mar 16, 2019
3,630
“Advanced persistent threat” is a term commonly used to describe a targeted cyber-attack that employs a complex set of methods and techniques to penetrate information system(s). Different aims of such attacks could be stealing/substituting/damaging confidential information, or establishing sabotage capabilities, the last of which could lead to financial and reputational damage of the targeted organisations. Such attacks are very purposeful, and usually involve highly specialised tools. The tools employed include heavily obfuscated malicious code, the malicious use of benign system tools, and non-file-based malicious code.
In our Advanced Threat Protection Test (Enhanced Real-World Test), we use hacking and penetration techniques that allow attackers to access internal computer systems. These attacks can be broken down into Lockheed Martin’s Cybersecurity Kill Chain, and seven distinct phases – each with unique IOCs (Indicators of Compromise) for the victims. All our tests use a subset of the TTP (Tactics, Techniques, Procedures) listed in the MITRE ATT&CK® framework. A false alarm test is also included in the report. The tests use a range of techniques and resources, mimicking malware used in the real world. Some examples of these are given here. We make use of system programs, in an attempt to bypass signature-based detection. Popular scripting languages (JavaScript, batch files, PowerShell, Visual Basic scripts, etc.) are used. The tests involve both staged and non-staged malware samples, and deploy obfuscation and/or encryption of malicious code before execution (Base64, AES). Different C2 channels are used to connect to the attacker (HTTP, HTTPS, TCP). Use is made of known exploit frameworks (Metasploit Framework, PowerShell Empire, commercial frameworks, etc.).......................................

Tested Products​

The following vendors participated in the Advanced Threat Protection Test. These are the vendors who were confident enough in the protection capabilities of their products against targeted attacks to take part in this public test. All other vendors in the Consumer Main-Test Series opted out of the test.

All consumer products were tested with default settings.

1.png2.png3.png
Read the full report here:
 

SeriousHoax

Level 47
Thread author
Well-known
Mar 16, 2019
3,630
Happy to see Microsoft didn't chicken out of this Advanced Threat test this time, and surprised to see how well it did in default settings. I guess, which shows that they are getting better and have more confidence in their base product.
 
Last edited:

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,040
According to Microsoft Defender.
Scenarios nr 14 and 15 could be blocked by the ASR rules for USB and MS Office. Many malware introduced in scenario nr 12 can be detected/blocked by other ASR rules or a higher Cloud Protection Level. Anyway, other AVs can also block more scenarios after some tweaking. :)
 
Last edited:

Anthony Qian

Level 9
Verified
Well-known
Apr 17, 2021
448
Very interesting results.

Some products (McAfee and Avira) perform well in AV-C's malware protection test but very poorly in this APT test. Some product (ESET) fails to get a 3-star certificate in malware protection tests, but turns out to be the best product in this APT test. Other products (Kaspersky and Bitdefender) generally perform well in both regular malware protection tests and this APT test. Anybody know why?

My thoughts on this seemingly irrational results:

According to my observation, some products like McAfee and Avira are continuously monitoring new samples uploaded to VirusTotal and immidiately block the samples that are not detected by themselves but detected by other trusted vendors. This strategy works out well in AV-C's malware protection tests as most samples may have been uploaded to VirusTotal. But it didn't work out well in the APT test which is focused on fileless attack.

Perhaps, something like this APT test can best indicate a product's true ability to block malware.
 

franz

Level 8
Verified
Well-known
May 29, 2021
383
I don't trust this test to be representative of real-life antivirus protection in general. In a test, one program is at the top, the next test at the bottom. I would like to see cruelsister and Shadowra do a similar test that was done here, but with their own traps. I would also like to hear what cruelsister and Shadowra think about this?

We are rooting for them both (y) :)
 

Anthony Qian

Level 9
Verified
Well-known
Apr 17, 2021
448
I don't trust this test to be representative of real-life antivirus protection in general. In a test, one program is at the top, the next test at the bottom. I would like to see cruelsister and Shadowra do a similar test that was done here, but with their own traps. I would also like to hear what cruelsister and Shadowra think about this?

We are rooting for them both (y) :)
Unlike malware sample tests, doing an APT test requires profound cybersecurity knowledge, some coding skills and a dedicated computer lab environment.
Anyway, I also hope someone here can do a similar test.
 

franz

Level 8
Verified
Well-known
May 29, 2021
383
Unlike malware sample tests, doing an APT test requires profound cybersecurity knowledge, some coding skills and a dedicated computer lab environment.
Anyway, I also hope someone here can do a similar test.
Considering what you say, isn't it strange that test results change the way they do if such tests give us correct answers? Maybe you only get out what you put in? :unsure:
 
  • Like
Reactions: simmerskool

Kubla

Level 8
Verified
Jan 22, 2017
355
ESET looks good on everything but Spear phishing and from what I have read most successful data breaches start with a spear phishing attack, I am wondering if you pair it with something like Blackfog it would seal that phishing hole so to speak?

 
Last edited:

monkeylove

Level 10
Verified
Well-known
Mar 9, 2014
489
If one wants something that does well overall and free, then I suppose it will be a choice between Bitdefender, Kaspersky, Avast, and AVG.

I'm not sure if the anti-ransomware features are complete for the first, but there were no popup notifications for upgrades.

For the second, when I open around 50 web pages simultaneously, not only of them are loaded fully. Also, there were popup notifications every month or so even if all notification settings I can think of have been disabled.

The third is the best so far, but it has to be run in silent mode so that no notifications show up.

I can't remember much about the fourth, except that Avast was slightly lighter than it.

Finally, I used Novabench free on all of them and Windows Security, and the latter was the heaviest.
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top