45AVs VS CobaltStrike

ShenguiTurmi · Mar 1, 2023

This is the test I did in China security forum from January 18-23, and I'm carrying it over so that more people can see it.

In 2020 we tested mainstream security software using the Empire framework (not carried over, I'm just paraphrasing the original) and some of them were able to defend well, while others failed miserably.
Now, almost three years later, we are going to do it again, this time using the more advanced and popular CobaltStrike framework.

Our test will do the following:
1. download the payload to local machine
2. start the payload (may have a loader)
3. payload establish c2 connection (target server is public cloud server)
4. target machine online
5. screenshot the target machine and send to cobaltstrike teamserver
6. teamserver send a command to obtain a txt file in the c drive (simulated data theft)
If the security software blocks any step of the following process in any way (including static scanning/bb/firewall blocking c2, etc.), the defense is considered successful, and only after all steps are executed and still no action is considered a failure.

This time we prepared 10 samples for testing purposes, all using the CobaltStrike framework.
Sample 1: CobaltStrike HTA (VBA) Payload, which will call VBA to run Payload after execution.
Sample 2: CobaltStrike Powershell Payload, built using the self-contained phishing function, no file landing.
Sample 3: CobaltStrike Bitsadmin Payload, built with its own phishing function, after running, it uses Bitsadmin to download the malicious Payload and combine it into an exe.
Sample 4: CobaltStrike StageLess EXE, as the most basic form of CS backdoor, to examine the strength of security software features.
Sample 5: In sample 4 EXE based on the addition of Themida protection, to examine whether the detection capabilities of security software will be affected by protection.
Sample 6: Payload loaded by NIM generated loader on top of CobaltStrike C Payload, inject to notepad launch Payload.
Sample 7: Payload loaded by NIM generated loader on top of CobaltStrike C Payload, using 3DES encryption.
Sample 8: Payload loaded by Veil generated Go language loader on top of CobaltStrike Powershell Payload.
Sample 9: Payload loaded by Veil-generated Python language loader on top of CobaltStrike Powershell Payload, using DES encryption.
Sample 10: Payload loaded by Go language loader provided by TideSec on top of CobaltStrike C language Payload, using XOR encryption.

Although I am now the only one involved in the test in MalwareTips, it is right to leave their names:
Kafan Malware Test Group: @ShenguiTurmi
Kafan BangBangTuan: @隔山打空气 @呵呵大神001
Participants without team: @東雪蓮Official

Because the workload of carrying screenshots one by one from other forums is a bit too much, I will first publish the test results, if you want to see which security software test screenshots, please reply to tell me, I will carry the screenshots over.

Test result:

Huorong

:
√ × √ √ √ √ √ √ √ √
Failed(9/10)

Tencent PC Manager(China TAV ver. not BD engine global ver.)

:
√ × √ √ × × × √ × ×
Failed(4/10)

Qihoo 360(China ver. not 360TS)

:
√ √ √ √ × √ √ × × √
Failed(7/10)

WiseVector StopX

:
√ √ √ √ √ √ √ √ √ √
Approved

Kingsoft Duba(China ver. not Kingsoft IS Pro)

:
√ √ √ × × × × × √ ×
Failed(4/10)

Qi-AnXin TianShou

:
√ √ √ √ √ √ √ √ √ √
Approved

Rising V17

:
√ × √ √ × × × √ × ×
Failed(4/10)

HitmanPro.Alert

:
√ √ √ √ √ √ √ √ √ √
Approved

Microsoft Defender

:
√ × √ √ √ × × √ √ ×
Failed(6/10)

Avast One

:
√ √ √ √ √ √ √ √ √ √
Approved

Heimdal

:
× × √ √ √ √ √ √ √ √
Failed(8/10)

F-Secure

:
√ √ √ √ √ √ √ √ √ √
Approved

Norton 360

:
√ √ √ √ √ √ √ √ √ √
Approved

Ikarus

:
√ × √ √ × √ √ √ × √
Failed(7/10)

Kaspersky IS

:
√ √ √ √ √ √ √ √ √ √
Approved

Avira

:
√ √ √ √ √ √ √ √ √ √
Approved

Bitdefender

:
√ √ √ √ √ √ √ √ √ √
Approved

Ahnlab V3 Lite

:
√ √ √ √ √ √ √ √ √ √
Approved

McAfee

:
√ √ √ √ √ × × √ √ ×
Failed(7/10)

Malwarebytes

:
√ × √ √ √ √ √ √ √ √
Failed(9/10)

Panda Dome

:
× × √ × × × × × × ×
Failed(1/10)

TrendMicro

:
√ √ √ √ √ × √ √ √ √
Failed(9/10)

ESET

:
√ √ √ √ √ √ √ √ √ √
Approved

QuickHeal

:
√ × √ √ × × × × × ×
Failed(3/10)

Webroot

:
× × √ √ × × × √ × ×
Failed(3/10)

ZoneAlarm Next-Gen

:
√ × √ √ √ √ √ √ √ ×
Failed(8/10)

Arconis

:
√ √ √ √ √ √ √ √ √ √
Approved

Cisco Immunet

:
× × × × × × × × × ×
Failed

Vibranium

:
× × √ √ √ √ √ √ √ ×
Failed(7/10)

Drweb AVDesk

:
√ √ √ √ √ √ √ √ √ √
Approved

K7

:
√ × √ √ √ √ √ √ √ √
Failed(9/10)

GDATA

:
√ √ √ √ √ √ √ √ √ √
Approved

Emsisoft

:
√ × √ √ √ √ √ √ √ √
Failed(9/10)

VIPRE

:
√ √ √ √ × √ √ √ √ √
Failed(9/10)

TotalDefense

:
√ √ √ √ × √ √ √ √ √
Failed(9/10)

eScan

:
√ × √ √ √ √ √ √ √ √
Failed(9/10)

Adaware Free

:
√ × √ √ × √ √ √ √ √
Failed(8/10)

Comodo IS Pro

:
? × ? ? ? ? ? ? ? ?
Failed(9/10)
NOTE: comodo did not detect any malicious behavior from start to finish, and the testing process was carried out completely, but comodo put 9 samples in a sandbox run automatically, so no screenshots or files of the real system were stolen (except sample 2).

Watchdog Anti-Malware

:
× × × × × × × × × ×
Failed

Zemana Anti-Malware

:
× × × × √ × × × × ×
Failed(1/10)

Zillya

:
× × × × × × × × × ×
Failed

Protegent

:
× × √ × × × × × × ×
Failed(1/10)

Bkav Free

:
× × × × × × × × × ×
Failed
NOTE: During testing, I found that the bkav free database has not been updated for a long time, and I'm not sure if they still maintain the free version.

MaxSecure

:
× × √ √ √ √ √ √ √ ×
Failed(7/10)

Catchpulse Lite

:
√ √ √ √ √ √ √ √ √ √
Failed(FP)
NOTE:catchpulse prevented the full attack, but it FP some normal files in my VM, which shouldn't have happened, so I counted it as a failure.

Source Test Link (chinese, maybe login require to show screenshot):【毒组x帮帮团】如果用攻击企业的方法攻击个人安全软件会怎么样呢？第三期_国外杀毒软件_安全区卡饭论坛 - 互助分享 - 大气谦和!

silversurfer · Mar 1, 2023

ShenguiTurmi said:
This is the test I did in China security forum from January 18-23, and I'm carrying it over so that more people can see it.

In 2020 we tested mainstream security software using the Empire framework (not carried over, I'm just paraphrasing the original) and some of them were able to defend well, while others failed miserably.
Now, almost three years later, we are going to do it again, this time using the more advanced and popular CobaltStrike framework.

Our test will do the following:
1. download the payload to local machine
2. start the payload (may have a loader)
3. payload establish c2 connection (target server is public cloud server)
4. target machine online
5. screenshot the target machine and send to cobaltstrike teamserver
6. teamserver send a command to obtain a txt file in the c drive (simulated data theft)
If the security software blocks any step of the following process in any way (including static scanning/bb/firewall blocking c2, etc.), the defense is considered successful, and only after all steps are executed and still no action is considered a failure.

This time we prepared 10 samples for testing purposes, all using the CobaltStrike framework.
Sample 1: CobaltStrike HTA (VBA) Payload, which will call VBA to run Payload after execution.
Sample 2: CobaltStrike Powershell Payload, built using the self-contained phishing function, no file landing.
Sample 3: CobaltStrike Bitsadmin Payload, built with its own phishing function, after running, it uses Bitsadmin to download the malicious Payload and combine it into an exe.
Sample 4: CobaltStrike StageLess EXE, as the most basic form of CS backdoor, to examine the strength of security software features.
Sample 5: In sample 4 EXE based on the addition of Themida protection, to examine whether the detection capabilities of security software will be affected by protection.
Sample 6: Payload loaded by NIM generated loader on top of CobaltStrike C Payload, inject to notepad launch Payload.
Sample 7: Payload loaded by NIM generated loader on top of CobaltStrike C Payload, using 3DES encryption.
Sample 8: Payload loaded by Veil generated Go language loader on top of CobaltStrike Powershell Payload.
Sample 9: Payload loaded by Veil-generated Python language loader on top of CobaltStrike Powershell Payload, using DES encryption.
Sample 10: Payload loaded by Go language loader provided by TideSec on top of CobaltStrike C language Payload, using XOR encryption.

Although I am now the only one involved in the test in MalwareTips, it is right to leave their names:
Kafan Malware Test Group: @ShenguiTurmi
Kafan BangBangTuan: @隔山打空气 @呵呵大神001
Participants without team: @東雪蓮Official

Because the workload of carrying screenshots one by one from other forums is a bit too much, I will first publish the test results, if you want to see which security software test screenshots, please reply to tell me, I will carry the screenshots over.

Test result:

Huorong :
√ × √ √ √ √ √ √ √ √
Failed(9/10)

Tencent PC Manager(China TAV ver. not BD engine global ver.) :
√ × √ √ × × × √ × ×
Failed(4/10)

Qihoo 360(China ver. not 360TS) :
√ √ √ √ × √ √ × × √
Failed(7/10)

WiseVector StopX :
√ √ √ √ √ √ √ √ √ √
Approved

Kingsoft Duba(China ver. not Kingsoft IS Pro) :
√ √ √ × × × × × √ ×
Failed(4/10)

Qi-AnXin TianShou :
√ √ √ √ √ √ √ √ √ √
Approved

Rising V17 :
√ × √ √ × × × √ × ×
Failed(4/10)

HitmanPro.Alert :
√ √ √ √ √ √ √ √ √ √
Approved

Microsoft Defender :
√ × √ √ √ × × √ √ ×
Failed(6/10)

Avast One :
√ √ √ √ √ √ √ √ √ √
Approved

Heimdal :
× × √ √ √ √ √ √ √ √
Failed(8/10)

F-Secure :
√ √ √ √ √ √ √ √ √ √
Approved

Norton 360 :
√ √ √ √ √ √ √ √ √ √
Approved

Ikarus :
√ × √ √ × √ √ √ × √
Failed(7/10)

Kaspersky IS :
√ √ √ √ √ √ √ √ √ √
Approved

Avira :
√ √ √ √ √ √ √ √ √ √
Approved

Bitdefender :
√ √ √ √ √ √ √ √ √ √
Approved

Ahnlab V3 Lite :
√ √ √ √ √ √ √ √ √ √
Approved

McAfee :
√ √ √ √ √ × × √ √ ×
Failed(7/10)

Malwarebytes :
√ × √ √ √ √ √ √ √ √
Failed(9/10)

Panda Dome :
× × √ × × × × × × ×
Failed(1/10)

TrendMicro :
√ √ √ √ √ × √ √ √ √
Failed(9/10)

ESET :
√ √ √ √ √ √ √ √ √ √
Approved

QuickHeal :
√ × √ √ × × × × × ×
Failed(3/10)

Webroot :
× × √ √ × × × √ × ×
Failed(3/10)

ZoneAlarm Next-Gen :
√ × √ √ √ √ √ √ √ ×
Failed(8/10)

Arconis :
√ √ √ √ √ √ √ √ √ √
Approved

Cisco Immunet :
× × × × × × × × × ×
Failed

Vibranium :
× × √ √ √ √ √ √ √ ×
Failed(7/10)

Drweb AVDesk :
√ √ √ √ √ √ √ √ √ √
Approved

K7 :
√ × √ √ √ √ √ √ √ √
Failed(9/10)

GDATA :
√ √ √ √ √ √ √ √ √ √
Approved

Emsisoft :
√ × √ √ √ √ √ √ √ √
Failed(9/10)

VIPRE :
√ √ √ √ × √ √ √ √ √
Failed(9/10)

TotalDefense :
√ √ √ √ × √ √ √ √ √
Failed(9/10)

eScan :
√ × √ √ √ √ √ √ √ √
Failed(9/10)

Adaware Free :
√ × √ √ × √ √ √ √ √
Failed(8/10)

Comodo IS Pro :
? × ? ? ? ? ? ? ? ?
Failed(9/10)
NOTE: comodo did not detect any malicious behavior from start to finish, and the testing process was carried out completely, but comodo put 9 samples in a sandbox run automatically, so no screenshots or files of the real system were stolen (except sample 2).

Watchdog Anti-Malware :
× × × × × × × × × ×
Failed

Zemana Anti-Malware :
× × × × √ × × × × ×
Failed(1/10)

Zillya :
× × × × × × × × × ×
Failed

Protegent :
× × √ × × × × × × ×
Failed(1/10)

Bkav Free :
× × × × × × × × × ×
Failed
NOTE: During testing, I found that the bkav free database has not been updated for a long time, and I'm not sure if they still maintain the free version.

MaxSecure :
× × √ √ √ √ √ √ √ ×
Failed(7/10)

Catchpulse Lite :
√ √ √ √ √ √ √ √ √ √
Failed(FP)
NOTE:catchpulse prevented the full attack, but it FP some normal files in my VM, which shouldn't have happened, so I counted it as a failure.

Source Test Link (chinese, maybe login require to show screenshot):【毒组x帮帮团】如果用攻击企业的方法攻击个人安全软件会怎么样呢？第三期_国外杀毒软件_安全区卡饭论坛 - 互助分享 - 大气谦和!

Well, no offense, please try to improve your presentation of testing results, all looks like a mess as probably just copy/paste including signs for passed or failed what looks almost illegible... Not to mention when browsing on mobile it's like scrolling down endless to read the full content of this thread.
The forum software offers the function of spoilers what should help to improve

ShenguiTurmi · Mar 1, 2023

silversurfer said:
Well, no offense, please try to improve your presentation of testing results, all looks like a mess as probably just copy/paste including signs for passed or failed what looks almost illegible... Not to mention when browsing on mobile it's like scrolling down endless to read the full content of this thread.
The forum software offers the function of spoilers what should help to improve

Thank you very much. Next time I will try to make it more readable.
Because the original test results are basically shown in screenshots, and it is really difficult to move 450 pictures here at one time. I will learn the use of spoilers. xD

Digmor Crusher · Mar 1, 2023

Very interesting, surprised at Emsisoft's result.

silversurfer · Mar 1, 2023

ShenguiTurmi said:
Because the workload of carrying screenshots one by one from other forums is a bit too much, I will first publish the test results, if you want to see which security software test screenshots, please reply to tell me, I will carry the screenshots over.

Test result:

Emsisoft :
√ × √ √ √ √ √ √ √ √
Failed(9/10)

@ShenguiTurmi please could you share here the screenshot of Emsisoft test result? as far as I understood, Emsisoft failed on the sample 2, correct?
Most big players passed this test, so no need to ask even for other AV, at least from my interest... Let's wait for other people

I'm sorry for forgot to mention on my first post #2 that watching from mobile phone and with MT forum dark theme makes it for me difficult to interpret all signs correct,
but now on my laptop, I see you probably added for each AV this info like: Failed(9/10)

Digmor Crusher · Mar 1, 2023

I think I misinterpreted , Emsisoft failed but it blocked 9 out of 10, I originally thought it failed on 9 out of 10. My bad.

SeriousHoax · Mar 1, 2023

I saw this test when it was published thanks to @Anthony Qian. This is a very well done realistic test that you won't see often, excluding professional lab tests.
Apart from the products that most of us expect to do well, quite a few AV vendors missed sample number 2 which is a fileless malware. It could mean that these products are overall less effective against fileless threats. Fileless threats are probably less of a worry for home users, but it's worth noting.

mlnevese · Mar 1, 2023

Digmor Crusher said:
I think I misinterpreted , Emsisoft failed but it blocked 9 out of 10, I originally thought it failed on 9 out of 10. My bad.

I had to reread the test results after your comment because I thought the same...

franz · Mar 1, 2023

mlnevese said:
I had to reread the test results after your comment because I thought the same...

Same here

Zero Knowledge · Mar 1, 2023

Great contribution! Interesting test results. Thank you for your effort!

ShenguiTurmi · Mar 1, 2023

silversurfer said:
@ShenguiTurmi please could you share here the screenshot of Emsisoft test result? as far as I understood, Emsisoft failed on the sample 2, correct?
Most big players passed this test, so no need to ask even for other AV, at least from my interest... Let's wait for other people

I'm sorry for forgot to mention on my first post #2 that watching from mobile phone and with MT forum dark theme makes it for me difficult to interpret all signs correct,
but now on my laptop, I see you probably added for each AV this info like: Failed(9/10)