AV-Comparatives Advanced Threat Protection Test 2021 – Consumer

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Andy Ful

Level 79
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
6,829
If you look at the Detection/Blocking stages table, you will find Avast (AVG) does not simply block all obfuscated scripts. In case 7 and 10, Avast blocked immediately after the threat has been run. In case 9, Avast blocked after the threat has been run, and its actions have been recognised.
The case 7 behaves just like Avast blocked obfuscated script run via Office macro.
The case 10 has nothing to do with scripts.
In the case 9 the CmdLine in .lnk file managed to download the PowerShell payload (obfuscated script) and the payload was blocked by Avast.(y)

Edit.
There is an error in the online report (wrong numeration of scenarios). So my answer about scenarios 9 and 10 is in fact about scenarios 10 and 11. The corrected answer is here:
https://malwaretips.com/threads/advanced-threat-protection-test-2021-–-consumer.110873/post-964448
 
Last edited:

Andy Ful

Level 79
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
6,829
AVG (and probably Avast) simply blocks all obfuscated scripts and this helped to get the best result.
Among 15 testing scenarios, there is an interesting sample nr 14 (corrected numbering) that managed to bypass the protection of 4 AVs. In the case of Avast, it was most probably blocked by Cyber Capture.
The attack was performed via spearphishing link, so the EXE file (masquerading werfault.exe) was downloaded/executed (MOTW added by the web browser). The MOTW is required for an EXE file to trigger Cyber Capture, so the file was uploaded and detonated in the cloud sandbox.

Another interesting example is sample nr 12 (corrected numbering). This sample could bypass 5 AVs (including Avast). The Avast CyberCapture could not block it. This attack is similar to scenario 14 (which was blocked by Avast). There is a note about this in the report:

Avast, AVG: In one case, the Sandbox came back with the verdict that the file was safe. Then, while the threat was already running in memory, it was detected, but a stable C2 connection remained open anyway, and the attack continued without restrictions.

Edit.
Post edited to include the corrections due to scenario numbering error in the online report.
I removed the note about scenario 12 (incorrect numbering) because, in fact, it was about scenario 13 (correct numbering). This scenario was blocked by Avast (script was blocked). If this script was not blocked then CyberCapture could not detect the DLL sideloading technique.
 
Last edited:

Gandalf_The_Grey

Level 59
Thread author
Verified
Helper
Top poster
Content Creator
Well-known
Apr 24, 2016
4,864

The_King

Level 12
Verified
Top poster
Well-known
Aug 2, 2020
557

Andy Ful

Level 79
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
6,829
Probably they're not confident about their product doing well in this test.
I found this in the report:
However, precisely the same product and configuration is used for all the tests in the series.
When we look into the Business Security Test March-June 2021 (full report) we can see for Defender:
Microsoft: Google Chrome extension “Windows Defender Browser Protection” installed and enabled; “CloudBlockLevel” set to “High”.

It means that Microsoft is tested without ASR rules and scored very well. This can be the reason why Microsoft does not want to participate in Advanced Threat Protection tests that would require enabling ASR rules.
 

The_King

Level 12
Verified
Top poster
Well-known
Aug 2, 2020
557
I was really disappointed with Bitdefender's result. It is clear that it does not scan scripts by default, but would it really improve the result? And why did they disable script protection in the default settings at all, if this is one of the most popular attack vectors? :(
There are a two things I have done to improve BD protection on my system.

1. Turn scan scripts on. (this may or may not have helped in the tests carried out here)
2. Under Firewall Turn on Alert mode. This will be a slight annoyance at first however after a day or two
it will only alert you if a new program or a changed/updated program is trying to connect to the internet.

I only get alerts now after Edge and Firefox update or after doing a windows security update.
This should improve protection from default.

Also keep in mind this is one set of tests and does not mean the product is completely usless. :LOL:
Some products that scored well here do not score has well in other malware related tests carried out by the same site.

Edit.
I also blocked a few LOLbins in BD firewall from accessing the internet.
 
Last edited:

Andy Ful

Level 79
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
6,829
If you look at the Detection/Blocking stages table, you will find Avast (AVG) does not simply block all obfuscated scripts. In case 7 and 10, Avast blocked immediately after the threat has been run. In case 9, Avast blocked after the threat has been run, and its actions have been recognised.
There is an error in the online report. Scenarios 9 and 10 are glued together and scenarios 10-14 are in fact 11-15. The PDF report is OK.
It seems that in scenario 10 (after correction) Avast blocked the non-obfuscated VBS script.
In scenario 9 the CmdLine in .LNK shortcut was used to download an obfuscated PowerShell payload which was blocked by Avast. (y)
 
Last edited:

Gandalf_The_Grey

Level 59
Thread author
Verified
Helper
Top poster
Content Creator
Well-known
Apr 24, 2016
4,864
I have both BIS and BTS subs.
I can confirm its off by default. I clicked reset settings and the option is disabled
View attachment 261855
From IBK:
Instead of looking at that setting (which also here is disabled, as it is all default with consumer products), please try malicious scripts and look whether they are scanned (they are, otherwise it would have missed several more).
 

The_King

Level 12
Verified
Top poster
Well-known
Aug 2, 2020
557

SeriousHoax

Level 41
Verified
Top poster
Well-known
Mar 16, 2019
3,092
I am on Wilders seen his reply.
I have the same handle there.
I am not a malware tester so not going to run malcious scripts on my PC to test. :LOL:
BD scans and block scripts just fine in the default settings after execution by signatures as well as by the behaviour blocker. But I don't know what extra thing it does when that option is enabled.
 

ExecutiveOrder

Level 2
Sep 21, 2021
56
There are some interesting points to note in their testing.
By default BD does not scan scripts.
Would BD have blocked these scripts if scan scripts option was turned on?
Not sure if Bitdefender changed its product behavior regarding script detection in the past 2 years.

It's a bit late to reply, but...
In ATP 2020 test, with the same "default" settings, it successfully blocked a similar "script" threat to number (3):
2) This threat is introduced via Trusted Relationship. A PowerShell script containing an AMSI bypass and a PowerShell Empire stager was executed.

In ATP 2019 test, it successfully blocked a "script" threat scenario:
9) This threat is introduced via Removable Media (USB). A PowerShell script executes a PowerShell payload into memory. This test case was created with Unicorn.
But also failed to another "script" threat scenario:
15) This threat is introduced via Spearphishing Link. A PowerShell script injects an obfuscated PowerShell payload into memory. This test case was created with Metasploit Meterpreter.

I have both BIS and BTS subs.
I can confirm its off by default. I clicked reset settings and the option is disabled
Based on this user guide:
The Scan scripts feature allows Bitdefender to scan powershell scripts and office documents that could contain script-based malware.
Not sure if it means only applicable to scripts (including PowerShell) inside macro files (like in office documents), it didn't mention if this setting is turned on by default. None of the "script" test scenarios were using macros as an attack vector. Also, this user guide is similar to the website support information, published in late 2019, before ATP 2020 and 2021 tests were started.
In this Bitdefender community post, look like it is just an UI issue, the original post said that after installation it's OFF but turned ON after reset (contrary to your experience, 2 years apart though, different versions), another user replied that the feature didn't appear in UI and have to contact support based on moderator's suggestion.

Do it only in guest VM if you want to test it against any malicious script.
Anyway, at the end of the day, we can conclude it's turned ON based on a statement by AV-Comparatives themselves:
According to IBK from AV-Comparatives Bitdefender does scan scripts by default:

Honestly, I also expect better from Bitdefender and am rather disappointed by these results, but anything can be turned out unexpected.
 

SeriousHoax

Level 41
Verified
Top poster
Well-known
Mar 16, 2019
3,092

Thanks to user @South Park's comment on the above thread, I got interested and checked whether he's right or not. Turns out the "Scan Script" feature in Bitdefender which is off by default (for most if not all) is AMSI. When it's enabled, AMSI is on, otherwise it's off. I checked this using three methods in the latest Bitdefender Total Security 26.0.3.29.
So this could be the reason why Bitdefender performed worse than expected in this test. Bitdefender free have almost no configuration options, so AMSI is not present in that product. Looks like the feature being off in the paid versions is not a bug, it's by choice. Though I don't know why.
All paid Bitdefender user should manually enable the feature to increase protection against scripts.