- Oct 13, 2019
- 779
This kind of evolved out of a few discussions (ESET, Emsisoft, etc) where the abilities of their behavior blocker came to question. As a part of this, I made two pieces of zero-day "malware". For now I'm avoiding their distribution because of all the legal implications of writing and distributing malware. The goal here was to come up with something that has ZERO known signatures... I compile it against a slightly different toolchain for each test and opted out of cloud submission as long as it does not affect detection.
The two samples are:
Sample 1: Simulated PUA/Replication: Copies itself to the temp directory under a brand new name. Registers that copy as a startup item via the registry.
Sample 2: Simulated Ransomware: Goes into "My Documents\test", and encrypts each file with a randomly generated AES256 key in a ".encrypted" file, deletes the original.
Result definitions:
Protected: Threat blocked with default settings
Protected (Folder Protection Only): For the ransomware sample, requires a specific anti-ransomware feature (like protected folders) to be turned on
Infected: For the startupware sample, AV suite had no reaction to it.
Partial: For the startupware sample, AV suite had a late reaction (such as after a reboot). This is overall acceptable since the sample has no malicious behavior, catching it after a reboot is perfectly fine.
Encrypted: For the ransomware sample, no matter what settings were tried, all files ended up encrypted.
Overall Conclusion:
F-Secure, Emsisoft, and Kaspersky provided the best protection. Overall, F-Secure and sorta Emsisoft won by a thin margin, because of a bug I hit in Kaspersky (see below)
Additional Notes:
I'm curious if this kind of testing is considered valuable. Happy to take suggestions of what other behaviors are common for in the wild malware samples and worthwhile of simulating. Remember that this is solely meant as a behavior blocker test, not an overall AV test. Behavior blocking is just one layer of protection that an AV provides.
The two samples are:
Sample 1: Simulated PUA/Replication: Copies itself to the temp directory under a brand new name. Registers that copy as a startup item via the registry.
Sample 2: Simulated Ransomware: Goes into "My Documents\test", and encrypts each file with a randomly generated AES256 key in a ".encrypted" file, deletes the original.
Result definitions:
Protected: Threat blocked with default settings
Protected (Folder Protection Only): For the ransomware sample, requires a specific anti-ransomware feature (like protected folders) to be turned on
Infected: For the startupware sample, AV suite had no reaction to it.
Partial: For the startupware sample, AV suite had a late reaction (such as after a reboot). This is overall acceptable since the sample has no malicious behavior, catching it after a reboot is perfectly fine.
Encrypted: For the ransomware sample, no matter what settings were tried, all files ended up encrypted.
Sample | Kaspersky Antivirus | F-Secure SAFE 17.7 | Windows Defender | ESET NOD32 13 | Norton 360 | Emsisoft Antimalware |
Simulated PUA/Replicator | Partial * | Protected | Infected | Infected | Protected + Partial | Protected |
Simulated Ransomware | Protected * | Protected | Protected (Folder Protection Only) | Encrypted | Encrypted | Protected |
Overall Conclusion:
F-Secure, Emsisoft, and Kaspersky provided the best protection. Overall, F-Secure and sorta Emsisoft won by a thin margin, because of a bug I hit in Kaspersky (see below)
Additional Notes:
- Kaspersky: Did a great job cleaning up the startup items and rolling back changes done. For the simulated ransomware, it reacted differently to each file being renamed (protected) vs just putting all the encrypted files into one password-protected ZIP file and then deleting all the files (no reaction). WARNING: Kaspersky's behavior blocker did not activate until after first reboot. Even though it appears to be functional after a fresh install, always reboot the machine after installing Kaspersky or you might not be protected. Since this only affects initial installation, I don't find it very concerning.
- F-Secure: All detections came from DeepGuard. Sample 1 had a disappointingly generic detection name but Sample 2 performance was impressive -- even disabling "Ransomware" (protected folder) detection, DeepGuard still thinks it's ransomware behavior.
- Windows Defender: Turning on the Ransomware Guard feature resulted in the only behavior block during this test. I should say, though, Windows Defender's ransomware protection feature is extremely sensitive and FP prone. For example, it also blocked me from running the Norton 360 installer because the installer creates a Norton Downloaded Files directory on the desktop.
- ESET: As discussed thoroughly in the ESET thread, did not react at all. According to their forums, their HIPS is more for detection of variants of existing malware, not entirely new breeds.
- Emsisoft: Did great against both tests with default settings. No complaints (other than a hang when running the binaries off the network)
- Norton: SONAR picked up the startup sample right away but it cleaned up the original file as opposed to the replicated binary, SONAR picked it up again on reboot and this time it cleaned it up completely. No reaction against the ransomware simulator though.
I'm curious if this kind of testing is considered valuable. Happy to take suggestions of what other behaviors are common for in the wild malware samples and worthwhile of simulating. Remember that this is solely meant as a behavior blocker test, not an overall AV test. Behavior blocking is just one layer of protection that an AV provides.