App Review Ransomware Test: Cylance, Sophos, VoodooShield | by VoodooShield

[erreale]

http://www.voodooshield.com/artwork/EfficacyTest.exe

many thanks

 

[Windows_Security]

@Zero Knowledge

First thank you for giving insightfull answers. The log really is a good tip.To answer your question.

Hey Kees!

Just a simple question, Kees are you now a business partner of VoodooSheild?

No, the WE in the VS video is Dan and his loyal Golden Retriever

VS is around a few years and from posting in the VS thread with Dan, we started to go into private discussion, PM's turned into emailing each other which over time turned into skype calls. Then beginning 2016 Dan had hired someone for marketing and sales and asked me to review the plans since I have a background of Marketing & Sales in IT and Dan is a developer in heart and mind.

So I am not a business partner, more like a pen pal who signed a non-disclosure before reviewing the plans. The reason I asked you for more info, is because I had the simular reservations when I advised Dan to spend his money on maturing VS first in stead of jumping on the Next Gen wagon with a product that works great but needs a little 'ease of use' polishing for both consumer and corporate market.

The magic and charm of VS is that AI/ML component of one man band solution performs better as NextGen solutions, while those NextGen solutions have raised between 45 to 140 million dollar to develop their AI/ML engine. It is a bit like David against a team of Goliath's or Asterix and Obelix against the Roman empire

 

[ForgottenSeer 55778]

Voodooshield is amazing! However, I take this with a grain of salt, as Voodooshield was the tester. For a fact, I know that Cylance was overhyped and not at all ready to compete with the likes of Emsisoft, Kaspersky, Bitdefender, Eset, etc.

 

[Windows_Security]

Voodooshield is amazing! However, I take this with a grain of salt, as Voodooshield was the tester. For a fact, I know that Cylance was overhyped and not at all ready to compete with the likes of Emsisoft, Kaspersky, Bitdefender, ESET, etc.

That is why Dan (VoodooShield) released his test sample randomizer as open source software, so that anyone could replicate the test with a malware set. See Evjl's Rain post#20

 

[Zero Knowledge]

Windows_Security/Kees

Now I understand your role in VoodooSheild. Thanks for clearing that up.

It sounds like you have given VoodooSheild some good advice. Make the product great then worry about the marketing.

 

[woodrowbone]

Windows_Security/Kees

Now I understand your role in VoodooSheild. Thanks for clearing that up.

It sounds like you have given VoodooSheild some good advice. Make the product great then worry about the marketing.

Yes, great advice!
No Comodo strategi in that advice

/W

 

[Windows_Security]

No, the WE in the VS video is Dan and his loyal Golden Retriever

EDIT Molly is a Yellow Labrador, not a Golden Retriever

Apolgies to Molly

 

[erreale]

@Zero Knowledge

First thank you for giving insightfull answers. The log really is a good tip.To answer your question.



No, the WE in the VS video is Dan and his loyal Golden Retriever

VS is around a few years and from posting in the VS thread with Dan, we started to go into private discussion, PM's turned into emailing each other which over time turned into skype calls. Then beginning 2016 Dan had hired someone for marketing and sales and asked me to review the plans since I have a background of Marketing & Sales in IT and Dan is a developer in heart and mind.

So I am not a business partner, more like a pen pal who signed a non-disclosure before reviewing the plans. The reason I asked you for more info, is because I had the simular reservations when I advised Dan to spend his money on maturing VS first in stead of jumping on the Next Gen wagon with a product that works great but needs a little 'ease of use' polishing for both consumer and corporate market.

The magic and charm of VS is that AI/ML component of one man band solution performs better as NextGen solutions, while those NextGen solutions have raised between 45 to 140 million dollar to develop their AI/ML engine. It is a bit like David against a team of Goliath's or Asterix and Obelix against the Roman empire

I think VS is a great piece of software at a low cost compared to other software. A price that everyone can afford.

 

[David R]

Having had hands-on experience with the Cylance and Sophos Home products, and done similar testing, I would be hard pressed to believe the VoodooShield did their due diligence in configuration of the Sophos Home and Cylance settings. You can easily create a passive policy, and force global quarantine of certain files, which would cause Cylance or Sophos to "trip up." So, unless VoodooShield comes out with a video showing their configuration of CylancePROTECT and Sophos Home, I wouldn't put much weight on this test.

EDIT: I just reviewed VoodooShield's home/free version. Take away: it's basically an app whitelisting program with cloud-based signatures (or scores) for malware. It seems useless without an internet connection. out of 100 samples I tried to run, it actually missed one and let it install.

Cylance also has an app whitelisting option, which I'm sure was not used in this video test.

Anyone else have feedback on this?

EDIT 2: Just ran my own test on a protected system (lab) with Cylance (offline, no internet connection) on 100 samples of ransomware and other kinds of random malware and all 100 pieces were quarantined with AND without execution.

Moral: Do your own tests. Take advantage of demo/lab environments vendors offer and test, test, test... then come to your own conclusions. Don't rely on any vendor's analysis/comparison of their own + other's solutions.

 

[danb]

Having had hands-on experience with the Cylance and Sophos Home products, and done similar testing, I would be hard pressed to believe the VoodooShield did their due diligence in configuration of the Sophos Home and Cylance settings. You can easily create a passive policy, and force global quarantine of certain files, which would cause Cylance or Sophos to "trip up." So, unless VoodooShield comes out with a video showing their configuration of CylancePROTECT and Sophos Home, I wouldn't put much weight on this test.

EDIT: I just reviewed VoodooShield's home/free version. Take away: it's basically an app whitelisting program with cloud-based signatures (or scores) for malware. It seems useless without an internet connection. out of 100 samples I tried to run, it actually missed one and let it install.

Cylance also has an app whitelisting option, which I'm sure was not used in this video test.

Anyone else have feedback on this?

EDIT 2: Just ran my own test on a protected system (lab) with Cylance (offline, no internet connection) on 100 samples of ransomware and other kinds of random malware and all 100 pieces were quarantined with AND without execution.

Moral: Do your own tests. Take advantage of demo/lab environments vendors offer and test, test, test... then come to your own conclusions. Don't rely on any vendor's analysis/comparison of their own + other's solutions.

This is Dan with VoodooShield... I am not going to get into a long argument with you, but just to explain a couple of things...

The CylancePROTECT configuration was configured manually by Dan and Joel of MalwareManaged for testing purposes with PUP detection enabled... they fully understood that I was testing their product. They actually emailed me out of the blue when they realized that I was testing so much malware, and asked if they could custom configure their product specifically for testing. They can confirm this, and if not, I still have all of the emails.

Sophos was tested with all default settings.

Sure, there are a lot of enterprise products that have an application whitelisting component... but according to MalwareManaged and Black Cipher (another Cylance reseller) and other endpoint protection vendors, smb and enterprise customers almost always disable the application whitelisting component. The reality is... unless application whitelisting is made user-friendly, it will NEVER be adopted by the masses... either consumers, SMB or enterprise. Everyone... well, most people understand this.

Yeah, if you only tested 100 samples with VS on AutoPilot, that is pretty much the exact same result I get every time I test... around 99-99.5%, so that sounds about right to me. If you tested with VS in Smart or Always ON mode, there is not a chance that something slipped by it, unless you clicked allow. If this is what you are suggesting, then you obviously have not tested VS in Smart or Always ON mode.

Sure, VS works much better when it has an internet connection... but then again, how is the computer going to become infected if it is not on the internet? USB drive? That is covered too.

I totally agree... people should test for themselves, I have been saying that for years now. However, I highly recommend that they not download test samples from malware repositories that have connections to ANY security software vendors.

 

[TheMalwareMaster]

This is Dan with VoodooShield... I am not going to get into a long argument with you, but just to explain a couple of things...

The CylancePROTECT configuration was configured manually by Dan and Joel of MalwareManaged for testing purposes with PUP detection enabled... they fully understood that I was testing their product. They actually emailed me out of the blue when they realized that I was testing so much malware, and asked if they could custom configure their product specifically for testing. They can confirm this, and if not, I still have all of the emails.

Sophos was tested with all default settings.

Sure, there are a lot of enterprise products that have an application whitelisting component... but according to MalwareManaged and Black Cipher (another Cylance reseller) and other endpoint protection vendors, smb and enterprise customers almost always disable the application whitelisting component. The reality is... unless application whitelisting is made user-friendly, it will NEVER be adopted by the masses... either consumers, SMB or enterprise. Everyone... well, most people understand this.

Yeah, if you only tested 100 samples with VS on AutoPilot, that is pretty much the exact same result I get every time I test... around 99-99.5%, so that sounds about right to me. If you tested with VS in Smart or Always ON mode, there is not a chance that something slipped by it, unless you clicked allow. If this is what you are suggesting, then you obviously have not tested VS in Smart or Always ON mode.

Sure, VS works much better when it has an internet connection... but then again, how is the computer going to become infected if it is not on the internet? USB drive? That is covered too.

I totally agree... people should test for themselves, I have been saying that for years now. However, I highly recommend that they not download test samples from malware repositories that have connections to ANY security software vendors.

A small question Dan. I've just seen again the video, and noticed that in the Cylance part internet connection was disabled. Why?

 

[David R]

This is Dan with VoodooShield... I am not going to get into a long argument with you, but just to explain a couple of things...

The CylancePROTECT configuration was configured manually by Dan and Joel of MalwareManaged for testing purposes with PUP detection enabled... they fully understood that I was testing their product. They actually emailed me out of the blue when they realized that I was testing so much malware, and asked if they could custom configure their product specifically for testing. They can confirm this, and if not, I still have all of the emails.

Sophos was tested with all default settings.

Sure, there are a lot of enterprise products that have an application whitelisting component... but according to MalwareManaged and Black Cipher (another Cylance reseller) and other endpoint protection vendors, smb and enterprise customers almost always disable the application whitelisting component. The reality is... unless application whitelisting is made user-friendly, it will NEVER be adopted by the masses... either consumers, SMB or enterprise. Everyone... well, most people understand this.

Yeah, if you only tested 100 samples with VS on AutoPilot, that is pretty much the exact same result I get every time I test... around 99-99.5%, so that sounds about right to me. If you tested with VS in Smart or Always ON mode, there is not a chance that something slipped by it, unless you clicked allow. If this is what you are suggesting, then you obviously have not tested VS in Smart or Always ON mode.

Sure, VS works much better when it has an internet connection... but then again, how is the computer going to become infected if it is not on the internet? USB drive? That is covered too.

I totally agree... people should test for themselves, I have been saying that for years now. However, I highly recommend that they not download test samples from malware repositories that have connections to ANY security software vendors.

I am not associated with the Cylance company in any way, but have had the opportunity to test first hand. I'm very familiar with their product and the ways you can configure it. The video you that was posted by VS seems really sketchy, I'll have to be honest. Can you provide the source of the malware samples that were used? I'd like to verify them myself.

The application whitelisting option is also disabled. Never use that for reasons you mentioned: it's not practical or user-friendly in any way.

The test I did with 100 samples earlier was not using script, memory or application control/blocking. The lab I had set up was completely offline (one advantage to Cylance over other products) and it worked brilliantly.

I have definitely found some false negatives myself (as with pretty much every other vendor), but not even close to the efficacy rating in the video.

 

[David R]

A small question Dan. I've just seen again the video, and noticed that in the Cylance part internet connection was disabled. Why?

The Cylance agent does not require an internet connection because the engine itself contains the IOCs, etc. It doesn't use signatures for it's AI engine. I'm suspicious of VS's video demo because they don't provide the configuration settings that were used, nor can confirm whether or not it had a "passive" policy applied to it with some of the hashes (that were caught) in Global Quarantine.

Something else to keep in mind: they did not have the Cylance UI open during the test, so you can't very/confirm what actions were being taken and whether or not a file might have been specifically waived.

 

[danb]

A small question Dan. I've just seen again the video, and noticed that in the Cylance part internet connection was disabled. Why?

Cool... yeah, they always recommend to test their product offline, so they can demonstrate the efficacy of their ML/Ai models alone, and also to be certain there is no funny business going on .

This is similar to the reason I recommend that people test VS on AutoPilot. If you test VS with it in AlwaysON or Smart mode, it will be a very, very boring test because it will block 100% of the samples.

So if I am aware that a vendor recommends that we test their product a certain way, it is best to follow their recommendation (in my opinion).

 

[danb]

I am not associated with the Cylance company in any way, but have had the opportunity to test first hand. I'm very familiar with their product and the ways you can configure it. The video you that was posted by VS seems really sketchy, I'll have to be honest. Can you provide the source of the malware samples that were used? I'd like to verify them myself.

The application whitelisting option is also disabled. Never use that for reasons you mentioned: it's not practical or user-friendly in any way.

The test I did with 100 samples earlier was not using script, memory or application control/blocking. The lab I had set up was completely offline (one advantage to Cylance over other products) and it worked brilliantly.

I have definitely found some false negatives myself (as with pretty much every other vendor), but not even close to the efficacy rating in the video.

Hehehe, David... I posted all of the hashes and offered to provide all of the samples to anyone who wanted them... has ANY vendor EVER done that before? How more open and honest could a test possibly be? I would be happy to send you the files. Please email me at support at voodooshield.com and I will send them to you.

You need to be very careful here... there is at least one indication that tells me that there is a high probability that you have never ran VS...
"The application whitelisting option is also disabled" is not a this is not a "thing" in VS!!!

Can you please provide the approximate date and time that you ran the tests of 100 samples? I would like to look them up in the database to see how VS and VoodooAi performed.

On a side note... if you tried VS for a few days, you would realize that we have reduced the number of "safe" blocks to next to nothing. I know this because I have to reset my whitelist almost daily for dev reasons, and I cannot remember the last time I had a block. I also know this because when I am performing maintenance for a client, they might ask me about a block, so I will look at the logs and notice that there are not that many... every time.

I completely agree about the false negatives in AutoPilot mode... it is difficult or impossible for ANY traditional AV or ML/Ai engine to consistently achieve an efficacy above 95% (or whatever number you want to use).

But then again, that is just further proof that the endpoint should be locked when it is at risk... you will never convince me otherwise .

 

[danb]

The Cylance agent does not require an internet connection because the engine itself contains the IOCs, etc. It doesn't use signatures for it's AI engine. I'm suspicious of VS's video demo because they don't provide the configuration settings that were used, nor can confirm whether or not it had a "passive" policy applied to it with some of the hashes (that were caught) in Global Quarantine.

Something else to keep in mind: they did not have the Cylance UI open during the test, so you can't very/confirm what actions were being taken and whether or not a file might have been specifically waived.

In case you missed it the first time: The CylancePROTECT configuration was configured manually by Dan and Joel of MalwareManaged for testing purposes with PUP detection enabled... they fully understood that I was testing their product. They actually emailed me out of the blue when they realized that I was testing so much malware, and asked if they could custom configure their product specifically for testing. They can confirm this, and if not, I still have all of the emails.

I was not aware that their GUI displays the ransomware that was missed. It did however, display the files that were quarantined... which is further proof that Dan and Joel configured the policy correctly. .

 

[David R]

"The application whitelisting option is also disabled"

This is in reference to the Cylance configuration for the test I ran with the 100 samples. I'll email you for the samples you used.

 

[danb]

This is in reference to the Cylance configuration for the test I ran with the 100 samples. I'll email you for the samples you used.

Great, thank you...I will send them to you either way... but can you also send me the 100 samples that you recently tested?

 

[David R]

PUP detection

PUP detection isn't a "thing" in Cylance's configuration settings. They rate based on files that are "Unsafe" or "Abnormal." This is an internal rating system.

 

[David R]

Great, thank you...I will send them to you either way... but can you also send me the 100 samples that you recently tested?

Sure, I can do that. About to hit the road, so I may not be able to until tomorrow.