App Review Ransomware Test: Cylance, Sophos, VoodooShield | by VoodooShield

[danb]

PUP detection isn't a "thing" in Cylance's configuration settings. They rate based on files that are "Unsafe" or "Abnormal." This is an internal rating system.

Yeah, I am not at all familiar with the Cylance options... Dan and Joel configured everything for me. For some reason, I thought they asked if I wanted PUP's to be detected... it was just over a year ago, so maybe I am confused.

So when a customer wants to either enable or disable PUP detection, is there not a settings for that?

I believe there is a setting for script control, right? Which basically just disables every script unless you manually whitelist it? The reason I remember that is because Dan, Joel and I had a long discussion about this feature.

 

[danb]

Sure, I can do that. About to hit the road, so I may not be able to until tomorrow.

Great, I just emailed you a link to the samples used in our test.

Since you are unable to send me your 100 samples until tomorrow, can you at least give me an approximate date and time that you performed the test, so I can look them up in the database? Thank you!

 

[simmerskool]

Great, I just emailed you a link to the samples used in our test.

Since you are unable to send me your 100 samples until tomorrow, can you at least give me an approximate date and time that you performed the test, so I can look them up in the database? Thank you!

looking forward to updates / discoveries from you both (& others) on this.

 

[David R]

Great, I just emailed you a link to the samples used in our test.

Since you are unable to send me your 100 samples until tomorrow, can you at least give me an approximate date and time that you performed the test, so I can look them up in the database? Thank you!

I ran the test yesterday in an offline lab environment. I did not connect to the internet with VoodooShield. Both tests (VS and Cylance) were performed in a lab where once the AV was installed, it was disconnected from the network so as not to produce any live traffic in the event malware was not blocked.

In this test, 99 of the samples were not run with VS in AutoPilot mode, while one escaped and ran anyways. Cylance AV blocked/quarantined all 100 samples while offline.

 

[David R]

Yeah, I am not at all familiar with the Cylance options... Dan and Joel configured everything for me. For some reason, I thought they asked if I wanted PUP's to be detected... it was just over a year ago, so maybe I am confused.

So when a customer wants to either enable or disable PUP detection, is there not a settings for that?

I believe there is a setting for script control, right? Which basically just disables every script unless you manually whitelist it? The reason I remember that is because Dan, Joel and I had a long discussion about this feature.

No, there's no "PUP" setting. They are probably referring to the "Abnormal" alert option. Something I would never disable.

There is a setting for script control yes. It gives you decent control over running scripts (could use some improvements).

 

[David R]

Great, I just emailed you a link to the samples used in our test.

Since you are unable to send me your 100 samples until tomorrow, can you at least give me an approximate date and time that you performed the test, so I can look them up in the database? Thank you!

Thanks, I'm building a new lab environment and am about to test the 147 samples (in Cylance) you sent with the Efficacy Test tool as well. Will reply to this thread with my findings...

 

[danb]

I ran the test yesterday in an offline lab environment. I did not connect to the internet with VoodooShield. Both tests (VS and Cylance) were performed in a lab where once the AV was installed, it was disconnected from the network so as not to produce any live traffic in the event malware was not blocked.

In this test, 99 of the samples were not run with VS in AutoPilot mode, while one escaped and ran anyways. Cylance AV blocked/quarantined all 100 samples while offline.

Cool, thank you for the 100 samples!

I noticed that there are A LOT of duplicates in the 100 samples you sent me. Typically, I think it is important to remove duplicates before testing, simply because if there are 10 samples that are duplicates, in a test of only 100 samples, it is going to skew the results heavily one way or the other. For example, if the software you are testing misses that sample, it skews the results so that the reported efficacy is much lower than the true efficacy. Conversely, if the software you are testing nails the sample, it skews the results so that the reported efficacy is much higher than the true efficacy. If you are testing 1,000-3,000 samples, it does not matter quite as much, but either way, it is best to remove the dups. Does that make sense?

I have a small utility I wrote that removes the duplicates, I will paste a link to it once I find it. It is not super advanced, but it does a pretty good job of removing the duplicates.

Also, when testing VS, it really does require an internet connection... for the blacklist scan and for the VoodooAi scan. Our ML/Ai models are in the Azure ML platform... we do not have local models. We might one day, but since VS does not require updates that often (in theory ), it is nice to have our ML models in the cloud, so we can update them without updating the client software.

If you test VS without an internet connection, it should block every single one of them, and tell you that an internet connection is not found. If you experience something different from that, please let me know . Thank you!

 

[danb]

No, there's no "PUP" setting. They are probably referring to the "Abnormal" alert option. Something I would never disable.

There is a setting for script control yes. It gives you decent control over running scripts (could use some improvements).

That makes sense now, thank you!

 

[danb]

Thanks, I'm building a new lab environment and am about to test the 147 samples (in Cylance) you sent with the Efficacy Test tool as well. Will reply to this thread with my findings...

Very cool! Yeah, the EfficacyTest app was something that I just kind of threw together, but it makes everything a lot easier when testing. Sure, you can use a command prompt or whatever, but it is nice to have a little more control over the test (with the built in options), and it also provides you with a detailed report when the test is complete. If you think of any new features that you want me to add to EfficacyTest that will make testing even easier, please let me know. The source code is on github if you want to modify it yourself. I actually have TONS of little testing apps that I have created the last few years (similar to the EfficacyTest and the dup finder apps)... I will look through them all and see if there are any others that you might be interested in, since you are building a new testing lab .

BTW, I actually think Cylance has a great product... it usually performs pretty well:





And I was actually surprised that it performed like it did in this test (the one for this thread)... but it is what it is.

They probably have updated their models since this test, so I will be curious to see the results, assuming that you are using the new models (and assuming that there are newer models). Thank you, have fun with your new lab!

 

[danb]

Oops, forget what I said about the 100 samples not being executable... I was doing something wrong (long story). Basically, if the folder name has "VoodooShield" in it, then VS will block the file automatically, without prompting the user. It is a bug in a silly security feature that we might remove one day.

 

[simmerskool]

BTW, I actually think Cylance has a great product... it usually performs pretty well:

Dan (& DavidR) gotta ask. I used "personal" cylance via malwaremanaged for awhile but drifted away, lack of user control, and not running on optimum hardware.... Now new hardware and running Deep Armor (beta), and its gui is minimal too, but not trying to compare as I don't exactly recall. The downside I see with DA is it slowing the pc down. IIRC cylance did not seem to slow down even the older hardware very much. In terms of my "personal" need to have something AI running, I'm finding VS is great. and perhaps it was combo of DA with VS running too that impacted DA speed? I have not been running DA the past couple of weeks, but interested in it. Have either of you used it or interested in testing it too. Would like to see that!
DeepArmor – SparkCognition Inc

 

[danb]

Dan (& DavidR) gotta ask. I used "personal" cylance via malwaremanaged for awhile but drifted away, lack of user control, and not running on optimum hardware.... Now new hardware and running Deep Armor (beta), and its gui is minimal too, but not trying to compare as I don't exactly recall. The downside I see with DA is it slowing the pc down. IIRC cylance did not seem to slow down even the older hardware very much. In terms of my "personal" need to have something AI running, I'm finding VS is great. and perhaps it was combo of DA with VS running too that impacted DA speed? I have not been running DA the past couple of weeks, but interested in it. Have either of you used it or interested in testing it too. Would like to see that!
DeepArmor – SparkCognition Inc

Thank you! DA looks cool, but I am guessing that they probably will not give me a license . I think VS probably combos extremely well with any of the ML/Ai products. Since they have realtime scans, their models / algos cannot be quite as aggressive as the ones we use for pre-execution, so it really should be a great combo either way.

 

[simmerskool]

Thank you! DA looks cool, but I am guessing that they probably will not give me a license . I think VS probably combos extremely well with any of the ML/Ai products. Since they have realtime scans, their models / algos cannot be quite as aggressive as the ones we use for pre-execution, so it really should be a great combo either way.

thanks

EDIT are you and DavidR going to post any more results after sharing malware or going PM?

 

[danb]

Sure, thank you! I tested and can confirm his result... VoodooAi missed one of the files. The blacklist scan would probably have nailed the file though. That is the cool thing about the blacklist and VoodooAi combo... the blacklist is great for known files, and VoodooAi is great for new and zero days. So even running on AutoPilot, the chances of something actually slipping through is pretty small. Then again, that is why we need to lock our computer when they are at risk . But AutoPilot is cool if you have a great AV or ML/Ai running with it.

He emailed me earlier and said he was sick over the weekend, but will be testing soon. It will be cool to see his results! I like it when people test VS .

BTW, I was mistaken about the duplicate utility I was going to post... I use one by Nirsoft (I believe)... but there are tons of them out there. Nirsoft has tons of really cool utilities.

 

[simmerskool]

Sure, thank you! I tested and can confirm his result... VoodooAi missed one of the files. The blacklist scan would probably have nailed the file though. That is the cool thing about the blacklist and VoodooAi combo... the blacklist is great for known files, and VoodooAi is great for new and zero days. So even running on AutoPilot, the chances of something actually slipping through is pretty small. Then again, that is why we need to lock our computer when they are at risk . But AutoPilot is cool if you have a great AV or ML/Ai running with it.

He emailed me earlier and said he was sick over the weekend, but will be testing soon. It will be cool to see his results! I like it when people test VS .

BTW, I was mistaken about the duplicate utility I was going to post... I use one by Nirsoft (I believe)... but there are tons of them out there. Nirsoft has tons of really cool utilities.

great, thanks!

 

[Windows_Security]

Thanks,

At #2 ease of use and #3 compatibility
At the moment we are discussing a local white-list based on program signatures determined when VS does its post-install system snapshot. Because companies often use standardized images, this would imply a "no actions required / automatic deployment" for system admins/IT managers. This local hash/signature whitelist guarantees offline protection (no internet connection available) and can be changed through central management (e.g. in response to malware outbreak). Central Admins even can put on a 'lock' when endpoint has admin rights (tightly limited to local white-list managed by company or relaxed using the cloud whitelist).

Dan emailed me that he had replaced the allow by whitlisted parent process with the allow by whitlisted vendor signature. Can anyone onfirm?

 

[David R]

Dan (& DavidR) gotta ask. I used "personal" cylance via malwaremanaged for awhile but drifted away, lack of user control, and not running on optimum hardware.... Now new hardware and running Deep Armor (beta), and its gui is minimal too, but not trying to compare as I don't exactly recall. The downside I see with DA is it slowing the pc down. IIRC cylance did not seem to slow down even the older hardware very much. In terms of my "personal" need to have something AI running, I'm finding VS is great. and perhaps it was combo of DA with VS running too that impacted DA speed? I have not been running DA the past couple of weeks, but interested in it. Have either of you used it or interested in testing it too. Would like to see that!
DeepArmor – SparkCognition Inc

My personal opinion: I wouldn't go with Cylance if I don't have management/control to tweak and blacklist/whitelist. It might be good for people who aren't tech savvy at all. Something like VS could be a decent alternative (I don't have a lot of experience, but I'm sure danb could answer all your questions there), but I'm more comfortable with Malwarebyte's endpoint suite, ESET or even BitDefender (the newer version seems to perform better). Again, it very much does depend on 1) cost, and 2) your hardware. If your hardware isn't up to snuff, you'll definitely notice performance issues, particularly when a full scan is running.

 

[David R]

thanks

EDIT are you and DavidR going to post any more results after sharing malware or going PM?

Definitely... Coming right up.

 

[David R]

Dan (& DavidR) gotta ask. I used "personal" cylance via malwaremanaged for awhile but drifted away, lack of user control, and not running on optimum hardware.... Now new hardware and running Deep Armor (beta), and its gui is minimal too, but not trying to compare as I don't exactly recall. The downside I see with DA is it slowing the pc down. IIRC cylance did not seem to slow down even the older hardware very much. In terms of my "personal" need to have something AI running, I'm finding VS is great. and perhaps it was combo of DA with VS running too that impacted DA speed? I have not been running DA the past couple of weeks, but interested in it. Have either of you used it or interested in testing it too. Would like to see that!
DeepArmor – SparkCognition Inc

I tried downloading the Home version, but it requires a corporate email... how'd you get the home version?

 

[David R]

@danb so I finally got around to the test.

Scan 1 (completely offline): 35 files, ranging from not-so-malicious malware to ransomware, were missed of the 147.

Scan 2 (with internet connectivity): all but 8 of the 147 files were quarantined (incl. the ransomware). See attachment.