- Sep 21, 2021
- 47
Background
While independent AV testing organizations like AV-Comparatives, SE Labs, AV-Test, etc have their own well-proven, high-quality, certified, and award-winning testing methodology including large scale, up-to-date sample collections backed with years of experience from experts that will help people choose the right AV solution, some might want to test themself mostly just to have a hands-on experience watching AV product at work against malware samples or just want to have fun with it, in a VM or secured environment, of course even if they can just watch some of the tests from Youtube channels.
It's very difficult to find how well an AV product fares against an unknown threat, especially ransomware. In some testing might simply disable everything except System Watcher (found in Kaspersky Internet/Total/Cloud Security and others) but this will make the System Watcher weaker because, in the documentation of the particular component, it works together with other components to exchange data with each other, although latest BSS can make that weakness nothing much to worry about (if the test is using the latest version of Kaspersky against "known" ransomware), also cloud/KSN (or even internet) is off to make sure it won't fetch data from the cloud.
Or even the PoC test by AV-Test here, still based on known samples:
So I was thinking that I will test an old Kaspersky against the latest threats, the major flaw of my test, perhaps the product cannot be compared to the current one against future threats (didn't include how far Kaspersky components improved or evolved ever since compared to threats), also testing each newer sample with new Kaspersky version is kind of impossible for me at least (like 1st sample from May vs Kaspersky from March, 2nd sample from June vs Kaspersky from May, and so on). The other thing, some new samples might have similarities to old samples, thus behavior stream signature able to catch some of them (but BSS is not necessarily always linked to ransomware every single behavior signature, rather a complete set that can be improved over time especially after some very unique ransomware emerged, to detect ransomware better). So here's my (another) amateur "for fun" test:
* My Test: Kaspersky vs 66 Samples of "Unknown" Ransomware [LINK HERE] *link spoiler
- Kaspersky Total Security 19.0.0.1088(d) released on 2018/12/03, database date: 2019/02/17, cloud/internet off, no updates, all relevant components active and set to default.
- Ransomware: 66 "unknown" samples (2019/03/01 to 2021/09/28) and 39 "known" samples (2007/01/05 to 2018/11/19) [other two were omitted, details in the test link]
Samples were scanned first (include opening the folder) until the AV was not detected anything and then the undetected samples were executed.
Note, some samples simply when it appears, not necessarily always "when it first appeared", and some consideration from me. Example: STOP/DJVU was first detected in 2018, but there's a "new variant" with different stuff like "super difficult" to get free decryptor, samples were from 2021 but the creation date 2020, so I selected that 2020 for this one, Krotten ransomware was first discovered in 2005 but my sample was in 2007, etc (individual date in the link). All samples already tested around Sept-Oct in the latest Windows 10 without AV, if they work (without "run as", just double click and UAC disabled) = they are included in this test. One sample I need to activate internet (refreshed by VM every sample) because it won't work without C2, another I have to install VC++ 13, details in the link.
- User data used: Documents (.txt, .rtf, .docx), Music (.mp3), Video (.srt, .mp4), picture (.jpg, .png), archive (.zip, .7z), program (.exe) placed in their own folder (Documents, Downloads, Musics, Pictures, Videos) and a full copy of all of them in Desktop, a folder directly in OS drive (C) named "Work", and new drive (D) named "Work".
- Up to date Windows 10, default, VMWare. Official Kaspersky Offline Installer from HERE.
- 18.18% detection rate during the on-demand scan, without cloud/internet.
- 15.15% no detection, ransomware encryption process completed.
- 18.18% partial protection rate after execution, w/o cloud/internet, some .mp3 files were sacrificed (all detected by behavioral components).
- 66.67% complete protection rate after execution, w/o cloud/internet (all detected by behavioral components).
"Known Samples" (39):
- 84.61% detection rate during the on-demand scan, without cloud/internet.
- 5.13% no detection, ransomware encryption process completed.
- 94.87% complete protection rate after execution, without cloud/internet.
After considering what I've explained in "Background" above, I think 66.67% complete (+18.18% partial) protection rate isn't that bad.
What do you guys think about my test especially the results? Any questions or feedback?
I think that's all. Not sure where should I post this, I don't think this is feedback for security programs, "Security Programs Tested" is archived, not part of test "Videos", and this forum is specifically for Q&A and product updates, but I saw a similar thread "malware test" from MT software forum for another vendor. I apologize if this isn't the correct place to post.
I would like to see your (other MT Kaspersky user) test results if you don't mind, how did you guys test your Kaspersky, "for fun" test or even serious test, especially if the result is not yet shared with the public. Thanks in advance.
While independent AV testing organizations like AV-Comparatives, SE Labs, AV-Test, etc have their own well-proven, high-quality, certified, and award-winning testing methodology including large scale, up-to-date sample collections backed with years of experience from experts that will help people choose the right AV solution, some might want to test themself mostly just to have a hands-on experience watching AV product at work against malware samples or just want to have fun with it, in a VM or secured environment, of course even if they can just watch some of the tests from Youtube channels.
It's very difficult to find how well an AV product fares against an unknown threat, especially ransomware. In some testing might simply disable everything except System Watcher (found in Kaspersky Internet/Total/Cloud Security and others) but this will make the System Watcher weaker because, in the documentation of the particular component, it works together with other components to exchange data with each other, although latest BSS can make that weakness nothing much to worry about (if the test is using the latest version of Kaspersky against "known" ransomware), also cloud/KSN (or even internet) is off to make sure it won't fetch data from the cloud.
Or even the PoC test by AV-Test here, still based on known samples:
So I was thinking that I will test an old Kaspersky against the latest threats, the major flaw of my test, perhaps the product cannot be compared to the current one against future threats (didn't include how far Kaspersky components improved or evolved ever since compared to threats), also testing each newer sample with new Kaspersky version is kind of impossible for me at least (like 1st sample from May vs Kaspersky from March, 2nd sample from June vs Kaspersky from May, and so on). The other thing, some new samples might have similarities to old samples, thus behavior stream signature able to catch some of them (but BSS is not necessarily always linked to ransomware every single behavior signature, rather a complete set that can be improved over time especially after some very unique ransomware emerged, to detect ransomware better). So here's my (another) amateur "for fun" test:
* My Test: Kaspersky vs 66 Samples of "Unknown" Ransomware [LINK HERE] *link spoiler
- Kaspersky Total Security 19.0.0.1088(d) released on 2018/12/03, database date: 2019/02/17, cloud/internet off, no updates, all relevant components active and set to default.
- Ransomware: 66 "unknown" samples (2019/03/01 to 2021/09/28) and 39 "known" samples (2007/01/05 to 2018/11/19) [other two were omitted, details in the test link]
Samples were scanned first (include opening the folder) until the AV was not detected anything and then the undetected samples were executed.
Note, some samples simply when it appears, not necessarily always "when it first appeared", and some consideration from me. Example: STOP/DJVU was first detected in 2018, but there's a "new variant" with different stuff like "super difficult" to get free decryptor, samples were from 2021 but the creation date 2020, so I selected that 2020 for this one, Krotten ransomware was first discovered in 2005 but my sample was in 2007, etc (individual date in the link). All samples already tested around Sept-Oct in the latest Windows 10 without AV, if they work (without "run as", just double click and UAC disabled) = they are included in this test. One sample I need to activate internet (refreshed by VM every sample) because it won't work without C2, another I have to install VC++ 13, details in the link.
- User data used: Documents (.txt, .rtf, .docx), Music (.mp3), Video (.srt, .mp4), picture (.jpg, .png), archive (.zip, .7z), program (.exe) placed in their own folder (Documents, Downloads, Musics, Pictures, Videos) and a full copy of all of them in Desktop, a folder directly in OS drive (C) named "Work", and new drive (D) named "Work".
- Up to date Windows 10, default, VMWare. Official Kaspersky Offline Installer from HERE.
Please answer the poll what your thought the detection is gonna be before clicking on the spoiler.
RESULTS
"Unknown Samples" (66):RESULTS
- 18.18% detection rate during the on-demand scan, without cloud/internet.
- 15.15% no detection, ransomware encryption process completed.
- 18.18% partial protection rate after execution, w/o cloud/internet, some .mp3 files were sacrificed (all detected by behavioral components).
- 66.67% complete protection rate after execution, w/o cloud/internet (all detected by behavioral components).
"Known Samples" (39):
- 84.61% detection rate during the on-demand scan, without cloud/internet.
- 5.13% no detection, ransomware encryption process completed.
- 94.87% complete protection rate after execution, without cloud/internet.
After considering what I've explained in "Background" above, I think 66.67% complete (+18.18% partial) protection rate isn't that bad.
What do you guys think about my test especially the results? Any questions or feedback?
I think that's all. Not sure where should I post this, I don't think this is feedback for security programs, "Security Programs Tested" is archived, not part of test "Videos", and this forum is specifically for Q&A and product updates, but I saw a similar thread "malware test" from MT software forum for another vendor. I apologize if this isn't the correct place to post.
I would like to see your (other MT Kaspersky user) test results if you don't mind, how did you guys test your Kaspersky, "for fun" test or even serious test, especially if the result is not yet shared with the public. Thanks in advance.
Last edited:
