Battle Ransomware simulator vs 10 AVs

[miguelang611]

Environment:
VMWare
Windows 10 LTSC 21H2
Windows Defender fully disabled (except for Windows Defender tests ofc)
4 GB RAM
No Internet

Ransomware simulator KnowBe4​

1. BitDefender Total Security 2022
Just Ransomware shield. ATP identifies the simulator, not the ransomwares itself, so we will just base the test on real behavioural shield, the ransomware remediation:

Spoiler

Test is pretty fast. About 1:30 min, and result is pretty good: 17/23 non vulnerable.
Resource usage however can be tremendous, having few moments when BD takes up over the ransomwares themselves.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


2. Eset Internet Security

Spoiler

8 and 9 keep stuck. If we have a look at their folders, we didn't get ransomed. However, I have made this test a couple of times and results differ... Usually they do get ransomed, but hey, we got lucky this time.

We got popups, but we didn't interact with them.
Note: with custom settings from RoboMan (Q&A - Configure ESET Antivirus for Maximum Security (by RoboMan)), you would be overwhelmed by pop-ups. Will depend if you allow or block... But nothing automatic. And yeah, I forgot to disable Internet, but results are same for Eset.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


3. WiseVector StopX v306

Spoiler

Depending on the ransom, it will autoremediate (16 as shown up here) or ask for recover (20 as shown down here):

Test ends up soon, also about 1:30. Real-time protection and advanced malware protection are disabled. We just leave ransomware shields enabled and with default settings.
Result: 11 non vulnerable.
Note: very light on resources, similar to ESET.
Note 2: great improvement from V305 (WiseVector StopX vs 0-day ransomware (KnowBe4)). On equivalent conditions we were just protected of 4 ransom and just few days after, we got 11!!

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


4. Kaspersky Free 21.5

Spoiler

Let's disable AMSI and rely on system watcher:

Surpringsily I can't find which component remediated the behaviour:

Threats aren't loaded in memory, system is clean, but is it really remediated or is it preblocked? On the past I remember I got system watcher popups and I could see cxp loading, but not anymore...
Result: FULLY CLEAN

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


5. F-Secure Safe 18.5

Spoiler

Results speak for themselves... 0 non vulnerable

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


6. Avira Prime

Spoiler

Note: Fastest test, although very bad results as well...

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


7. Microsfot Defender Default:

Spoiler

Well done Defender!! 1 popup 1 protected!! No extra hassle!!

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


8. Hardened Microsoft Defender

Spoiler

Hardened Defender (security level: high/interactive/max - no difference)

With Interactive/Max, the script is blocked (similar behaviour as if we keep enabled signatures on BD/ESET or the 2 other engines in WVX). This is not ransomware remediation but raw blocking --> Not valid for the test:

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


9. G-Data Internet Security 2022

Spoiler

Very heavy on resources... Killed our test:

But only 1 ransom catched:

Let's reboot... Should we hope?

Nah, we shouldn't!!

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


10. Avast Free

Spoiler

 

[miguelang611]

Results:

BitDefender Total Security --> 17/23. Great results. With all shields enabled we would have got 23/23 because it would be pre-blocked via signatures.
WiseVector StopX v306 --> 11/23. Great results as well.
ESET Internet Security 15.1.2.0 --> 3/23. Pretty much 0 behavioural shield. With custom settings and manual interaction we could get better protection... If we click on the proper button.
Kaspersky Free (UK version) 21.5 --> 23/23. Clear winner
MS Defender --> 1/23. Poor results
MS Defender Hardened --> on high we get 1 more ransom catched. On interactive/max we get the whole script blocked. Again, no real behavioural shield. Same as ESET.
F-Secure Safe 18.5 --> 0/23 DeepGuard fails to prevent the attack.
Avira Prime --> 0/23. Bad results again
G-Data Total Security 2022 --> 1/23. The rest are unidentified, and although test is killed, damage is already done.
Avast Free --> 1/23. Again same thing. We got a pop-up and it is the only one protected.

Winners:

1. Kaspersky --> even on its free version we get max protection out of the box.
2. BD Total Security --> great reactive protection against unknown threats. Not present on Free (which fails the test).
3. WiseVector StopX v306 --> version after version it is improving. From 4 to 11 in just one version jump! Well done!

Conclussions:
Protected folders or similar thing is in general a needed thing for your beloved data... Avast gets 23/23 with them enabled for instance...

 

[miguelang611]

Reserved 2

 

[miguelang611]

Reserved 3

 

[blackice]

Turning off real time scanning is not an accurate reflection for some AV vendors as they have engine interdependencies. Also KnowB4 has been known to have variable and unreliable results for years. I’ve never considered it a reliable tool.

 

[Nightwalker]

Turning off real time scanning is not an accurate reflection for some AV vendors as they have engine interdependencies. Also KnowB4 has been known to have variable and unreliable result for years. I’ve never considered it a reliable tool.

Another thing is that some security vendors whitelisted the tool, so it wont be monitored at all by the behavior blocker component.

Unfortunately, this specific test is useless ...

 

[blackice]

Another thing is that some security vendors whitelisted the tool, so it wont be monitored at all by the behavior blocker component.

That would be my guess for the ones with 0/23 detected.

 

[Trooper]

Turning off real time scanning is not an accurate reflection for some AV vendors as they have engine interdependencies. Also KnowB4 has been known to have variable and unreliable results for years. I’ve never considered it a reliable tool.

Was about to say something similar.

 

[blackice]

By the way, I appreciate the effort put into this, even if I think the methodology is flawed.

 

[Nightwalker]

By the way, I appreciate the effort put into this, even if I think the methodology is flawed.

 

[miguelang611]

Turning off real time scanning is not an accurate reflection for some AV vendors as they have engine interdependencies. Also KnowB4 has been known to have variable and unreliable results for years. I’ve never considered it a reliable tool.

Unfortunately if AV vendors aren't clear about this, I can't do anything. Anyways, everybody has paused for a moment some shield on an AV... The clearer you have the things the better chance you won't " mess" up because of dependencies.

Another thing is that some security vendors whitelisted the tool, so it wont be monitored at all by the behavior blocker component.

Unfortunately, this specific test is useless ...

Mostly if you run the test with signatures enabled, it is blocked from the begining (not allowing cxp to run). So I doubt it is whitelisted.

By the way, I appreciate the effort put into this, even if I think the methodology is flawed.

Thank you!! I am open to other methodologies I can make simple changes to it!!

By the way, I added some small conclussions on 2nd post!

See you!

 

[blackice]

ESET has been pretty clear that their modules are interdependent. Though they don’t have a great BB anyway. As for deep guard since the test is a test and not malicious we cannot draw any conclusions really. It’s possible the ones who did well have coded to do well in the test. Without actual ransomware we are still not very enlightened by this test software.

As for users turning off signature while executing potentially malicious files that would have been caught otherwise…that’s on the user.

 

[Trooper]

ESET has been pretty clear that their modules are interdependent. Though they don’t have a great BB anyway. As for deep guard since the test is a test and not malicious we cannot draw any conclusions really. It’s possible the ones who did well have coded to do well in the test. Without actual ransomware we are still not very enlightened by this test software.

As for users turning off signature while executing potentially malicious files that would have been caught otherwise…that’s on the user.

Spot on. Would love to see each of these tested like this but with actual ransomware.

 

[MacDefender]

I think the verbal descriptions from KnowBe4 are pretty good in terms of describing a valid ransomware algorithm. But as others have said, KnowBe4's binaries have frequently either been added as signatures or whitelisted from behavior blocking. Remember that almost all vendors use behavior blocker hits to feed into their cloud reporting, and not all of them want their samples polluted with proof of concepts.

I've found that writing your own hand-rolled malware doing the things described is a better custom zero-day test, and will get you results closer to what you expect.

 

[HarborFront]

Why Avast performs poorly? I thought it has behavior shield, ransomware shield, cybercapture and hardened mode?

 

[Andy Ful]

Results:
...
MS Defender Hardened --> on high we get 1 more ransom catched. On interactive/max we get the whole script blocked. Again, no real behavioural shield. Same as ESET.
...

If I have correctly understood, you suggest that the script was blocked by the ASR rule. But, ASR rules usually do not block scripts - they focus on blocking the suspicious actions (behavioral blocks). Could you post the alert related to this block?

I agree with the conclusion that Defender could identify not the ransomware attack but the suspicious actions of the simulator. But, this does not mean that there is no real behavioral shield - this also follows from my tests based on custom-made samples. The correct test should be done on the custom simulator that is undetected by Defender.

If you cannot make the simulator undetected by Defender, then the test can be enhanced by repeating it with a disabled ASR rule that blocked the simulator. But you also have to remove all samples dependent on this particular rule (could be remediated if this ASR rule was active).

 

[Shadowra]

Kaspersky proves once again its performance against Ransomware!

Too bad, no Norton in your test, I would have liked to see the SONAR in activity

 

[Andrew3000]

Why Avast performs poorly? I thought it has behavior shield, ransomware shield, cybercapture and hardened mode?

AVAST/AVG protect from ransomware only folders that have been specified by the user (excluding folders: documents, images, videos etc that are included by default)

 

[Andy Ful]

If I have correctly understood, you suggest that the script was blocked by the ASR rule. But, ASR rules usually do not block scripts - they focus on blocking the suspicious actions (behavioral blocks). Could you post the alert related to this block?

I agree with the conclusion that Defender could identify not the ransomware attack but the suspicious actions of the simulator. But, this does not mean that there is no real behavioral shield - this also follows from my tests based on custom-made samples. The correct test should be done on the custom simulator that is undetected by Defender.

If you cannot make the simulator undetected by Defender, then the test can be enhanced by repeating it with a disabled ASR rule that blocked the simulator. But you also have to remove all samples dependent on this particular rule (could be remediated if this ASR rule was active).

I have done the test myself. I had to deactivate the ASR rule "Block process creations originating from PSExec and WMI commands".

1. ConfigureDefender MAX + deactivated one ASR rule ---> all blocked (2 false positives)
2. ConfigureDefender Default + Cloud Block Level = Block ----> all blocked (2 false positives)
3. ConfigureDefender Default + Cloud Block Level = Highest ----> all blocked (2 false positives)
4. ConfigureDefender Default ---> all blocked (2 false positives)

These results should be repeated by someone else because running the same test 4 times on one computer could cause the Defender to learn about the dangerous actions.
So, 4 different computers should be used to test these four settings separately.

Edit.
In my opinion, using such a tool as KnowBe4 is not especially useful, because many AVs can learn via post-execution telemetry. So, the results depend much on the previous tests done by anonymous users before the test is done and reported on Malware Tips.

 

[Andy Ful]

I have repeated the tests (Windows 10 Home 21H2), but now I started from Default settings and removed dynamic signatures after each test:

1. Defender Default ---> 12 vulnerable, 2 false positives.
2. Defender Default + Cloud Block Level = Highest ----> 5 vulnerable, 2 false positives.
3. Defender Default + Cloud Block Level = Block ----> 0 vulnerable, 2 false positives.

Microsoft recommends Cloud Block Level = Highest, as a strong anti-ransomware protection.
As I wrote in my previous test, it is very probable that Defender learned much from post-execution telemetry compared to the test done by @miguelang611. This confirms my opinion that the results do not show the real capabilities of tested AVs, but rather show if someone else used KnowBe4 in the past to test the particular AV.