Q&A My "Unknown" Ransomware Test | How do you test your Kaspersky?

How much protection rate of "Unknown" Ransomware* do you think in this test?

  • No Protection

    Votes: 0 0.0%
  • Between 1-24%

    Votes: 2 9.1%
  • Between 25-49%

    Votes: 1 4.5%
  • Between 50-74%

    Votes: 3 13.6%
  • Between 75-99%

    Votes: 12 54.5%
  • Complete Protection

    Votes: 3 13.6%
  • Accidentally Clicked the Results First (Spoiler)

    Votes: 1 4.5%

  • Total voters
    22

ExecutiveOrder

Level 1
Verified
Sep 21, 2021
27
140
Background
While independent AV testing organizations like AV-Comparatives, SE Labs, AV-Test, etc have their own well-proven, high-quality, certified, and award-winning testing methodology including large scale, up-to-date sample collections backed with years of experience from experts that will help people choose the right AV solution, some might want to test themself mostly just to have a hands-on experience watching AV product at work against malware samples or just want to have fun with it, in a VM or secured environment, of course even if they can just watch some of the tests from Youtube channels.

It's very difficult to find how well an AV product fares against an unknown threat, especially ransomware. In some testing might simply disable everything except System Watcher (found in Kaspersky Internet/Total/Cloud Security and others) but this will make the System Watcher weaker because, in the documentation of the particular component, it works together with other components to exchange data with each other, although latest BSS can make that weakness nothing much to worry about (if the test is using the latest version of Kaspersky against "known" ransomware), also cloud/KSN (or even internet) is off to make sure it won't fetch data from the cloud.
Or even the PoC test by AV-Test here, still based on known samples:
This test scenario reveals the readiness of security solutions to protect against ransomware attacks where malware developers start looking for alternative encryption methods or techniques. Even when some techniques are not popular or common, or even generally in use, this doesn’t mean they are not already used in very narrow targeted attacks, or will not become popular in the future. Security solutions are expected to be ready to protect users in all conditions, no matter what the current threat landscape looks like

So I was thinking that I will test an old Kaspersky against the latest threats, the major flaw of my test, perhaps the product cannot be compared to the current one against future threats (didn't include how far Kaspersky components improved or evolved ever since compared to threats), also testing each newer sample with new Kaspersky version is kind of impossible for me at least (like 1st sample from May vs Kaspersky from March, 2nd sample from June vs Kaspersky from May, and so on). The other thing, some new samples might have similarities to old samples, thus behavior stream signature able to catch some of them (but BSS is not necessarily always linked to ransomware every single behavior signature, rather a complete set that can be improved over time especially after some very unique ransomware emerged, to detect ransomware better). So here's my (another) amateur "for fun" test:

* My Test: Kaspersky vs 66 Samples of "Unknown" Ransomware [LINK HERE] *link spoiler
- Kaspersky Total Security 19.0.0.1088(d) released on 2018/12/03, database date: 2019/02/17, cloud/internet off, no updates, all relevant components active and set to default.
- Ransomware: 66 "unknown" samples (2019/03/01 to 2021/09/28) and 39 "known" samples (2007/01/05 to 2018/11/19) [other two were omitted, details in the test link]
Samples were scanned first (include opening the folder) until the AV was not detected anything and then the undetected samples were executed.
Note, some samples simply when it appears, not necessarily always "when it first appeared", and some consideration from me. Example: STOP/DJVU was first detected in 2018, but there's a "new variant" with different stuff like "super difficult" to get free decryptor, samples were from 2021 but the creation date 2020, so I selected that 2020 for this one, Krotten ransomware was first discovered in 2005 but my sample was in 2007, etc (individual date in the link). All samples already tested around Sept-Oct in the latest Windows 10 without AV, if they work (without "run as", just double click and UAC disabled) = they are included in this test. One sample I need to activate internet (refreshed by VM every sample) because it won't work without C2, another I have to install VC++ 13, details in the link.
- User data used: Documents (.txt, .rtf, .docx), Music (.mp3), Video (.srt, .mp4), picture (.jpg, .png), archive (.zip, .7z), program (.exe) placed in their own folder (Documents, Downloads, Musics, Pictures, Videos) and a full copy of all of them in Desktop, a folder directly in OS drive (C) named "Work", and new drive (D) named "Work".
- Up to date Windows 10, default, VMWare. Official Kaspersky Offline Installer from HERE.

Please answer the poll what your thought the detection is gonna be before clicking on the spoiler.
RESULTS
"Unknown Samples" (66):
- 18.18% detection rate during the on-demand scan, without cloud/internet.
- 15.15% no detection, ransomware encryption process completed.
- 18.18% partial protection rate after execution, w/o cloud/internet, some .mp3 files were sacrificed (all detected by behavioral components).
- 66.67% complete protection rate after execution, w/o cloud/internet (all detected by behavioral components).

"Known Samples" (39):
- 84.61% detection rate during the on-demand scan, without cloud/internet.
- 5.13% no detection, ransomware encryption process completed.
- 94.87% complete protection rate after execution, without cloud/internet.

After considering what I've explained in "Background" above, I think 66.67% complete (+18.18% partial) protection rate isn't that bad.

What do you guys think about my test especially the results? Any questions or feedback?

I think that's all. Not sure where should I post this, I don't think this is feedback for security programs, "Security Programs Tested" is archived, not part of test "Videos", and this forum is specifically for Q&A and product updates, but I saw a similar thread "malware test" from MT software forum for another vendor. I apologize if this isn't the correct place to post.
I would like to see your (other MT Kaspersky user) test results if you don't mind, how did you guys test your Kaspersky, "for fun" test or even serious test, especially if the result is not yet shared with the public. Thanks in advance.
 
Last edited:

dabluez98

Level 3
Oct 2, 2018
140
288
my guess was between 50-74 , i am glad it was no lower than 68 though cause thats a bit scary as to how much protection one can really expect but then again this test tested ‘unknown malware’ which average joe should not run into - hence protection is in fact 90ish i would guess for average user based on this result
 
Last edited:

ExecutiveOrder

Level 1
Verified
Sep 21, 2021
27
140
my gues was between 50-74 , i am glad it was no lower than 68 though cause thats a bit scary as to how much protection kne can really expect but then again this test tesrs ‘unknown malware’ which average joe should not run into - hence protection is in fact 90ish i would guess fir average user based on this result
Yeah, if it's possible to test with something like 7 days difference between each sample discovery date and the Kaspersky database date, I think it will be close to a 90% complete protection rate. If (old version of) Kaspersky in my test doesn't need to sacrifice few files (.mp3), the protection rate is almost 85%.
"Real-World" 0-day is most likely close to 100% with the latest version of Kaspersky (and using KSN).
 
Top