Battle Ransomware simulator vs 10 AVs

Software comparison
BitDefender Total Security 2022
WiseVector StopX v306
ESET Internet Security 15.1.2.0
Kaspersky Free (UK version) 21.5.11.384 (Patch B)
MS Defender
MS Defender Hardened
F-Secure Safe 18.5
Avira Prime
G-Data Total Security 2022
Avast Free
Feature comparison
  1. Ransomware protection

miguelang611

Level 2
Thread author
Apr 13, 2020
99
Environment:
VMWare
Windows 10 LTSC 21H2
Windows Defender fully disabled (except for Windows Defender tests ofc)
4 GB RAM
No Internet

Ransomware simulator KnowBe4


1. BitDefender Total Security 2022
Just Ransomware shield. ATP identifies the simulator, not the ransomwares itself, so we will just base the test on real behavioural shield, the ransomware remediation:
1649971890294.png

1649971941096.png


Test is pretty fast. About 1:30 min, and result is pretty good: 17/23 non vulnerable.
Resource usage however can be tremendous, having few moments when BD takes up over the ransomwares themselves.

1649972110943.png


------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


2. Eset Internet Security
1649972358906.png


1649972414437.png

8 and 9 keep stuck. If we have a look at their folders, we didn't get ransomed. However, I have made this test a couple of times and results differ... Usually they do get ransomed, but hey, we got lucky this time.
1649972916689.png

We got popups, but we didn't interact with them.
Note: with custom settings from RoboMan (Q&A - Configure ESET Antivirus for Maximum Security (by RoboMan)), you would be overwhelmed by pop-ups. Will depend if you allow or block... But nothing automatic. And yeah, I forgot to disable Internet, but results are same for Eset.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


3. WiseVector StopX v306
1649973164685.png

1649973186367.png


1649973251824.png

Depending on the ransom, it will autoremediate (16 as shown up here) or ask for recover (20 as shown down here):
1649973415383.png

Test ends up soon, also about 1:30. Real-time protection and advanced malware protection are disabled. We just leave ransomware shields enabled and with default settings.
Result: 11 non vulnerable.
Note: very light on resources, similar to ESET.
Note 2: great improvement from V305 (WiseVector StopX vs 0-day ransomware (KnowBe4)). On equivalent conditions we were just protected of 4 ransom and just few days after, we got 11!!


------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


4. Kaspersky Free 21.5
1649974126494.png

1649973778206.png

1649973837658.png

Let's disable AMSI and rely on system watcher:
1649974215285.png

1649974376277.png

1649974422287.png

Surpringsily I can't find which component remediated the behaviour:
1649974494524.png

Threats aren't loaded in memory, system is clean, but is it really remediated or is it preblocked? On the past I remember I got system watcher popups and I could see cxp loading, but not anymore...
Result: FULLY CLEAN


------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


5. F-Secure Safe 18.5
1649973943321.png

1649973962184.png

1649974307175.png

Results speak for themselves... 0 non vulnerable

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


6. Avira Prime
1649974975481.png

1649975030794.png

1649975074990.png

Note: Fastest test, although very bad results as well...


------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


7. Microsfot Defender Default:
1649975260000.png

1649975305885.png

1649975344500.png

Well done Defender!! 1 popup 1 protected!! No extra hassle!!

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


8. Hardened Microsoft Defender
Hardened Defender (security level: high/interactive/max - no difference)
1649976942087.png

1649976992977.png

With Interactive/Max, the script is blocked (similar behaviour as if we keep enabled signatures on BD/ESET or the 2 other engines in WVX). This is not ransomware remediation but raw blocking --> Not valid for the test:
1649977113541.png

1649977197919.png

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


9. G-Data Internet Security 2022
1649975623322.png

1649975651537.png

Very heavy on resources... Killed our test:
1649975697983.png

But only 1 ransom catched:
1649975727367.png

Let's reboot... Should we hope?
1649975896955.png

Nah, we shouldn't!!



------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


10. Avast Free
1649976107072.png

1649976160194.png

1649976188089.png

1649976492126.png
 

Attachments

  • 1649975990058.png
    1649975990058.png
    246.7 KB · Views: 65
  • 1649976440841.png
    1649976440841.png
    495.9 KB · Views: 72

miguelang611

Level 2
Thread author
Apr 13, 2020
99
Results:

BitDefender Total Security --> 17/23. Great results. With all shields enabled we would have got 23/23 because it would be pre-blocked via signatures.
WiseVector StopX v306 --> 11/23. Great results as well.
ESET Internet Security 15.1.2.0 --> 3/23. Pretty much 0 behavioural shield. With custom settings and manual interaction we could get better protection... If we click on the proper button.
Kaspersky Free (UK version) 21.5 --> 23/23. Clear winner
MS Defender --> 1/23. Poor results
MS Defender Hardened --> on high we get 1 more ransom catched. On interactive/max we get the whole script blocked. Again, no real behavioural shield. Same as ESET.
F-Secure Safe 18.5 --> 0/23 DeepGuard fails to prevent the attack.
Avira Prime --> 0/23. Bad results again
G-Data Total Security 2022 --> 1/23. The rest are unidentified, and although test is killed, damage is already done.
Avast Free --> 1/23. Again same thing. We got a pop-up and it is the only one protected.

Winners:

1. Kaspersky --> even on its free version we get max protection out of the box.
2. BD Total Security --> great reactive protection against unknown threats. Not present on Free (which fails the test).
3. WiseVector StopX v306 --> version after version it is improving. From 4 to 11 in just one version jump! Well done!

Conclussions:
Protected folders or similar thing is in general a needed thing for your beloved data... Avast gets 23/23 with them enabled for instance...
 
Last edited:

Nightwalker

Level 23
Verified
Helper
Top poster
Content Creator
Well-known
May 26, 2014
1,297
Turning off real time scanning is not an accurate reflection for some AV vendors as they have engine interdependencies. Also KnowB4 has been known to have variable and unreliable result for years. I’ve never considered it a reliable tool.

Another thing is that some security vendors whitelisted the tool, so it wont be monitored at all by the behavior blocker component.

Unfortunately, this specific test is useless ...
 

miguelang611

Level 2
Thread author
Apr 13, 2020
99
Turning off real time scanning is not an accurate reflection for some AV vendors as they have engine interdependencies. Also KnowB4 has been known to have variable and unreliable results for years. I’ve never considered it a reliable tool.
Unfortunately if AV vendors aren't clear about this, I can't do anything. Anyways, everybody has paused for a moment some shield on an AV... The clearer you have the things the better chance you won't " mess" up because of dependencies.
Another thing is that some security vendors whitelisted the tool, so it wont be monitored at all by the behavior blocker component.

Unfortunately, this specific test is useless ...
Mostly if you run the test with signatures enabled, it is blocked from the begining (not allowing cxp to run). So I doubt it is whitelisted.
By the way, I appreciate the effort put into this, even if I think the methodology is flawed.
Thank you!! I am open to other methodologies I can make simple changes to it!!

By the way, I added some small conclussions on 2nd post! :)

See you!
 
Last edited by a moderator:

blackice

Level 36
Verified
Top poster
Well-known
Apr 1, 2019
2,577
ESET has been pretty clear that their modules are interdependent. Though they don’t have a great BB anyway. As for deep guard since the test is a test and not malicious we cannot draw any conclusions really. It’s possible the ones who did well have coded to do well in the test. Without actual ransomware we are still not very enlightened by this test software.

As for users turning off signature while executing potentially malicious files that would have been caught otherwise…that’s on the user.
 

Trooper

Level 16
Well-known
Aug 28, 2015
755
ESET has been pretty clear that their modules are interdependent. Though they don’t have a great BB anyway. As for deep guard since the test is a test and not malicious we cannot draw any conclusions really. It’s possible the ones who did well have coded to do well in the test. Without actual ransomware we are still not very enlightened by this test software.

As for users turning off signature while executing potentially malicious files that would have been caught otherwise…that’s on the user.

Spot on. Would love to see each of these tested like this but with actual ransomware.
 

MacDefender

Level 16
Verified
Top poster
Oct 13, 2019
786
I think the verbal descriptions from KnowBe4 are pretty good in terms of describing a valid ransomware algorithm. But as others have said, KnowBe4's binaries have frequently either been added as signatures or whitelisted from behavior blocking. Remember that almost all vendors use behavior blocker hits to feed into their cloud reporting, and not all of them want their samples polluted with proof of concepts.

I've found that writing your own hand-rolled malware doing the things described is a better custom zero-day test, and will get you results closer to what you expect.
 

Andy Ful

From Hard_Configurator Tools
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
7,067
Results:
...
MS Defender Hardened --> on high we get 1 more ransom catched. On interactive/max we get the whole script blocked. Again, no real behavioural shield. Same as ESET.
...

If I have correctly understood, you suggest that the script was blocked by the ASR rule. But, ASR rules usually do not block scripts - they focus on blocking the suspicious actions (behavioral blocks). Could you post the alert related to this block?

I agree with the conclusion that Defender could identify not the ransomware attack but the suspicious actions of the simulator. But, this does not mean that there is no real behavioral shield - this also follows from my tests based on custom-made samples. The correct test should be done on the custom simulator that is undetected by Defender.

If you cannot make the simulator undetected by Defender, then the test can be enhanced by repeating it with a disabled ASR rule that blocked the simulator. But you also have to remove all samples dependent on this particular rule (could be remediated if this ASR rule was active).
 
Last edited:

Andy Ful

From Hard_Configurator Tools
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
7,067
If I have correctly understood, you suggest that the script was blocked by the ASR rule. But, ASR rules usually do not block scripts - they focus on blocking the suspicious actions (behavioral blocks). Could you post the alert related to this block?

I agree with the conclusion that Defender could identify not the ransomware attack but the suspicious actions of the simulator. But, this does not mean that there is no real behavioral shield - this also follows from my tests based on custom-made samples. The correct test should be done on the custom simulator that is undetected by Defender.

If you cannot make the simulator undetected by Defender, then the test can be enhanced by repeating it with a disabled ASR rule that blocked the simulator. But you also have to remove all samples dependent on this particular rule (could be remediated if this ASR rule was active).
I have done the test myself. I had to deactivate the ASR rule "Block process creations originating from PSExec and WMI commands".
  1. ConfigureDefender MAX + deactivated one ASR rule ---> all blocked (2 false positives)
  2. ConfigureDefender Default + Cloud Block Level = Block ----> all blocked (2 false positives)
  3. ConfigureDefender Default + Cloud Block Level = Highest ----> all blocked (2 false positives)
  4. ConfigureDefender Default ---> all blocked (2 false positives)
These results should be repeated by someone else because running the same test 4 times on one computer could cause the Defender to learn about the dangerous actions.
So, 4 different computers should be used to test these four settings separately.

Edit.
In my opinion, using such a tool as KnowBe4 is not especially useful, because many AVs can learn via post-execution telemetry. So, the results depend much on the previous tests done by anonymous users before the test is done and reported on Malware Tips.(y)
 
Last edited:

Andy Ful

From Hard_Configurator Tools
Verified
Helper
Top poster
Developer
Well-known
Dec 23, 2014
7,067
I have repeated the tests (Windows 10 Home 21H2), but now I started from Default settings and removed dynamic signatures after each test:

1650134123106.png

  1. Defender Default ---> 12 vulnerable, 2 false positives.
  2. Defender Default + Cloud Block Level = Highest ----> 5 vulnerable, 2 false positives.
  3. Defender Default + Cloud Block Level = Block ----> 0 vulnerable, 2 false positives.
Microsoft recommends Cloud Block Level = Highest, as a strong anti-ransomware protection.
As I wrote in my previous test, it is very probable that Defender learned much from post-execution telemetry compared to the test done by @miguelang611. This confirms my opinion that the results do not show the real capabilities of tested AVs, but rather show if someone else used KnowBe4 in the past to test the particular AV.
 
Last edited: