Battle Ransomware simulator vs 10 AVs

[miguelang611]

Environment:
VMWare
Windows 10 LTSC 21H2
Windows Defender fully disabled (except for Windows Defender tests ofc)
4 GB RAM
No Internet

Ransomware simulator KnowBe4​

1. BitDefender Total Security 2022
Just Ransomware shield. ATP identifies the simulator, not the ransomwares itself, so we will just base the test on real behavioural shield, the ransomware remediation:

Spoiler

Test is pretty fast. About 1:30 min, and result is pretty good: 17/23 non vulnerable.
Resource usage however can be tremendous, having few moments when BD takes up over the ransomwares themselves.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


2. Eset Internet Security

Spoiler

8 and 9 keep stuck. If we have a look at their folders, we didn't get ransomed. However, I have made this test a couple of times and results differ... Usually they do get ransomed, but hey, we got lucky this time.

We got popups, but we didn't interact with them.
Note: with custom settings from RoboMan (Q&A - Configure ESET Antivirus for Maximum Security (by RoboMan)), you would be overwhelmed by pop-ups. Will depend if you allow or block... But nothing automatic. And yeah, I forgot to disable Internet, but results are same for Eset.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


3. WiseVector StopX v306

Spoiler

Depending on the ransom, it will autoremediate (16 as shown up here) or ask for recover (20 as shown down here):

Test ends up soon, also about 1:30. Real-time protection and advanced malware protection are disabled. We just leave ransomware shields enabled and with default settings.
Result: 11 non vulnerable.
Note: very light on resources, similar to ESET.
Note 2: great improvement from V305 (WiseVector StopX vs 0-day ransomware (KnowBe4)). On equivalent conditions we were just protected of 4 ransom and just few days after, we got 11!!

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


4. Kaspersky Free 21.5

Spoiler

Let's disable AMSI and rely on system watcher:

Surpringsily I can't find which component remediated the behaviour:

Threats aren't loaded in memory, system is clean, but is it really remediated or is it preblocked? On the past I remember I got system watcher popups and I could see cxp loading, but not anymore...
Result: FULLY CLEAN

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


5. F-Secure Safe 18.5

Spoiler

Results speak for themselves... 0 non vulnerable

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


6. Avira Prime

Spoiler

Note: Fastest test, although very bad results as well...

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


7. Microsfot Defender Default:

Spoiler

Well done Defender!! 1 popup 1 protected!! No extra hassle!!

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


8. Hardened Microsoft Defender

Spoiler

Hardened Defender (security level: high/interactive/max - no difference)

With Interactive/Max, the script is blocked (similar behaviour as if we keep enabled signatures on BD/ESET or the 2 other engines in WVX). This is not ransomware remediation but raw blocking --> Not valid for the test:

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


9. G-Data Internet Security 2022

Spoiler

Very heavy on resources... Killed our test:

But only 1 ransom catched:

Let's reboot... Should we hope?

Nah, we shouldn't!!

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


10. Avast Free

Spoiler

 

[Shadowra]

With Norton

 

[miguelang611]

Kaspersky proves once again its performance against Ransomware!

Too bad, no Norton in your test, I would have liked to see the SONAR in activity

With Norton

When I first did these tests in the past Kaspersky was already the most advantaged pupil, both Kaspersky and BD improved however! And as far as I see as you tested, Norton does a pretty fair job as well!!

AVAST/AVG protect from ransomware only folders that have been specified by the user (excluding folders: documents, images, videos etc that are included by default)

Correct! I tried KnowBe4 on Documents folder and avast would prevent anything to run not trusted, so would have worked!

I have repeated the tests (Windows 10 Home 21H2), but now I started from Default settings and removed dynamic signatures after each test:

View attachment 265938

1. Defender Default ---> 12 vulnerable, 2 false positives.
2. Defender Default + Cloud Block Level = Highest ----> 5 vulnerable, 2 false positives.
3. Defender Default + Cloud Block Level = Block ----> 0 vulnerable, 2 false positives.

Microsoft recommends Cloud Block Level = Highest, as a strong anti-ransomware protection.
As I wrote in my previous test, it is very probable that Defender learned much from post-execution telemetry compared to the test done by @miguelang611. This confirms my opinion that the results do not show the real capabilities of tested AVs, but rather show if someone else used KnowBe4 in the past to test the particular AV.

Well wow, that is a huge increase in protection from what I got. However I gotta say that I repeated these tests with 12 days between my first tests and when I took these with screenshots. They were pretty much equal between both timestamps... However I didn't try Defender (neither stock or hardened) on my first test (just the other 8 options) so that might be the reason!

Very useful comment!

 

[Andy Ful]

I have repeated the tests (Windows 10 Home 21H2), but now I started from Default settings and removed dynamic signatures after each test:

View attachment 265938

1. Defender Default ---> 12 vulnerable, 2 false positives.
2. Defender Default + Cloud Block Level = Highest ----> 5 vulnerable, 2 false positives.
3. Defender Default + Cloud Block Level = Block ----> 0 vulnerable, 2 false positives.

...

I am not sure if the results included in this thread for tested AVs are truly correct.

I repeated the test 25 times for Defender at default settings and the results were as follows:
12, 12, 12, 11, 10, 6, 9, 8, 7, 5, 7, 7, 7, 7, 3, 6, 7, 18, 17, 20, 16, 18, 20, 16, 16 ~ on average 11 vulnerables per 1 test.

So, it is possible that in a single test one could get the result 20 vulnerables and also 3 vulnerables.

To be sure, one should repeat the test several times for each AV.

 

[miguelang611]

I am not sure if the results included in this thread for tested AVs are truly correct.

I repeated the test 25 times for Defender at default settings and the results were as follows:
12, 12, 12, 11, 10, 6, 9, 8, 7, 5, 7, 7, 7, 7, 3, 6, 7, 18, 17, 20, 16, 18, 20, 16, 16 ~ on average 11 vulnerables per 1 test.

So, it is possible that in a single test one could get the result 20 vulnerables and also 3 vulnerables.

To be sure, one should repeat the test several times for each AV.

Wew, that is so much of a change!! Defender I just tested once, but the others there was like 10/15 days between my first test and when I did it properly for uploading it here and nothing really changed... But well, that's 2/3 times, not 10 hehe

By the way, thanks to the mod who edited my OP and added spoilers. It is so much better now!

See you!