AV-Comparatives Advanced Threat Protection Test 2022 – Consumer

[SeriousHoax]

“Advanced persistent threat” is a term commonly used to describe a targeted cyber-attack that employs a complex set of methods and techniques to penetrate information system(s). Different aims of such attacks could be stealing/substituting/damaging confidential information, or establishing sabotage capabilities, the last of which could lead to financial and reputational damage of the targeted organisations. Such attacks are very purposeful, and usually involve highly specialised tools. The tools employed include heavily obfuscated malicious code, the malicious use of benign system tools, and non-file-based malicious code.
In our Advanced Threat Protection Test (Enhanced Real-World Test), we use hacking and penetration techniques that allow attackers to access internal computer systems. These attacks can be broken down into Lockheed Martin’s Cybersecurity Kill Chain, and seven distinct phases – each with unique IOCs (Indicators of Compromise) for the victims. All our tests use a subset of the TTP (Tactics, Techniques, Procedures) listed in the MITRE ATT&CK® framework. A false alarm test is also included in the report. The tests use a range of techniques and resources, mimicking malware used in the real world. Some examples of these are given here. We make use of system programs, in an attempt to bypass signature-based detection. Popular scripting languages (JavaScript, batch files, PowerShell, Visual Basic scripts, etc.) are used. The tests involve both staged and non-staged malware samples, and deploy obfuscation and/or encryption of malicious code before execution (Base64, AES). Different C2 channels are used to connect to the attacker (HTTP, HTTPS, TCP). Use is made of known exploit frameworks (Metasploit Framework, PowerShell Empire, commercial frameworks, etc.).......................................

Tested Products​

The following vendors participated in the Advanced Threat Protection Test. These are the vendors who were confident enough in the protection capabilities of their products against targeted attacks to take part in this public test. All other vendors in the Consumer Main-Test Series opted out of the test.

All consumer products were tested with default settings.

Read the full report here:

Advanced Threat Protection Test 2022 - Consumer

AV-Comparatives' Advanced Threat Protection Test 2022 for Consumer security products released, testing the protection against advanced attacks

www.av-comparatives.org

 

[Moonhorse]

Congratulations eset fans, its seem to be shining now

 

[SeriousHoax]

Happy to see Microsoft didn't chicken out of this Advanced Threat test this time, and surprised to see how well it did in default settings. I guess, which shows that they are getting better and have more confidence in their base product.

 

[Andy Ful]

According to Microsoft Defender.
Scenarios nr 14 and 15 could be blocked by the ASR rules for USB and MS Office. Many malware introduced in scenario nr 12 can be detected/blocked by other ASR rules or a higher Cloud Protection Level. Anyway, other AVs can also block more scenarios after some tweaking.

 

[simmerskool]

Congratulations eset fans, its seem to be shining now

happy days are here again

 

[Anthony Qian]

Very interesting results.

Some products (McAfee and Avira) perform well in AV-C's malware protection test but very poorly in this APT test. Some product (ESET) fails to get a 3-star certificate in malware protection tests, but turns out to be the best product in this APT test. Other products (Kaspersky and Bitdefender) generally perform well in both regular malware protection tests and this APT test. Anybody know why?

My thoughts on this seemingly irrational results:

According to my observation, some products like McAfee and Avira are continuously monitoring new samples uploaded to VirusTotal and immidiately block the samples that are not detected by themselves but detected by other trusted vendors. This strategy works out well in AV-C's malware protection tests as most samples may have been uploaded to VirusTotal. But it didn't work out well in the APT test which is focused on fileless attack.

Perhaps, something like this APT test can best indicate a product's true ability to block malware.

 

[franz]

I don't trust this test to be representative of real-life antivirus protection in general. In a test, one program is at the top, the next test at the bottom. I would like to see cruelsister and Shadowra do a similar test that was done here, but with their own traps. I would also like to hear what cruelsister and Shadowra think about this?

We are rooting for them both

 

[Anthony Qian]

I don't trust this test to be representative of real-life antivirus protection in general. In a test, one program is at the top, the next test at the bottom. I would like to see cruelsister and Shadowra do a similar test that was done here, but with their own traps. I would also like to hear what cruelsister and Shadowra think about this?

We are rooting for them both

Unlike malware sample tests, doing an APT test requires profound cybersecurity knowledge, some coding skills and a dedicated computer lab environment.
Anyway, I also hope someone here can do a similar test.

 

[franz]

Unlike malware sample tests, doing an APT test requires profound cybersecurity knowledge, some coding skills and a dedicated computer lab environment.
Anyway, I also hope someone here can do a similar test.

Considering what you say, isn't it strange that test results change the way they do if such tests give us correct answers? Maybe you only get out what you put in?

 

[ForgottenSeer 69673]

When are we going to see some SRP tests?

 

[Kubla]

ESET looks good on everything but Spear phishing and from what I have read most successful data breaches start with a spear phishing attack, I am wondering if you pair it with something like Blackfog it would seal that phishing hole so to speak?

​

 

[monkeylove]

If one wants something that does well overall and free, then I suppose it will be a choice between Bitdefender, Kaspersky, Avast, and AVG.

I'm not sure if the anti-ransomware features are complete for the first, but there were no popup notifications for upgrades.

For the second, when I open around 50 web pages simultaneously, not only of them are loaded fully. Also, there were popup notifications every month or so even if all notification settings I can think of have been disabled.

The third is the best so far, but it has to be run in silent mode so that no notifications show up.

I can't remember much about the fourth, except that Avast was slightly lighter than it.

Finally, I used Novabench free on all of them and Windows Security, and the latter was the heaviest.