App Review 1,000 Malware Sample Pre-Execution Efficacy Test - Malware Test

[Deleted member 2913]

A test done by Voodoo Shield Dev.


1000 random malware samples (Details in the video)
More details in the video

Products Tested
Voodoo Shield
Cylance
Avira
Sophos
AVG
Bitdefender
Kaspersky
Eset
McAfee
Avast
Norton

 

[Atlas147]

That was really interesting, would love to see the same malware pack go against other multi-engine cloud based AVs like Zemana Antimalware or SecureAplus

 

[Piteko21]

Great test, some AV surprise me like AVG and Sophos. I expected more from Bitdefender and Kaspersky.
Seems that free AV are doing a good

 

[DardiM]

Doesn't it depend a lot on the settings ?
For example, my Kaspersky Total Security settings (modified after read a @harlan4096 post), doesn't let any file to run (at least the first time) without my consent (even ZAM, MBAM, etc...) by Application Control : I set it to put in Untrusted group all unknown files (even with a digital signature)

Pingouin critical sense - ON
I saw It's a test done by Voodoo Shield Dev ...
=> Voodoo Shield on AutoPilot Efficacity Test =>100 %
It's not a test by independent entity : sure that they made the settings to disadvantage the others (even before recording this video )
BTW, they should have put safes files => 100 % blocked too ?
Pingouin critical sense - OFF

In any case, thanks for sharing it

 

[DJ Panda]

This test kind of shocked me. I set Avast to be almost like Voodosheild and a little more I have had it since 2006 and won't give it up now.

 

[harlan4096]

I test many samples almost every day and many of them even are not detected but running in low restricted group/privileges (KTS/KIS Application Control) and system is not affect, so that means sometimes not detected is not equal than system is affected/infected...

 

[Nightwalker]

What about antivirus settings? Was PUP detection enabled in Kaspersky/Eset for example?

 

[harlan4096]

As you can see in MWHub section, Default Settings are mandatory to test, so PUP setting is disabled in my KTS2016, anyway even with PUP setting enabled in Kaspersky products some samples may be flagged as Trusted, so Kaspersky products should be tweaked for better blocking protection... in Kaspersky section in this forum there are some threads reffering about it

 

[DJ Panda]

Hey so I got a contact from Voodosheild support and they told me they wanted me to post this. Voodosheild Dev response for the comment by @DardiM I quote


"This is no big deal at all, but we need to make something very, very clear...


Video Review - 1,000 Malware Sample Pre-Execution Efficacy Test - Malware Test


VS was on AutoPilot during the test (the lock was OFF), so any clean / safe sample would have been automatically allowed (see 44:54 in the video)... where I installed VLC media player to simply show that the good stuff was still being allowed.


All of the software was tested using the default settings, and all of the definitions were updated. I had to disable VS's Automatically Allow by Parent Process feature, otherwise, since all of the malware files were child processes of the test app would have been allowed. I also disabled "Deny by Default" so we could see the full user prompts, and also adjusted the countdown timer and the number of VS flashes so that the executions would happen quicker. Other then that, VS was using all default settings, and was on AutoPilot.


Also, every prompt clearly says "Threat Detected by VoodooAi!", or often times "Threats Detected by Blacklist Scan and VoodooAi". One thing I totally forgot to mention is that the blacklist scan is only analyzing some of the files because it will only allow you to upload a few at a time... basically it limits the number of files you can upload at any given time. If we delayed each execution by 5-10 seconds, the blacklist scan would analyze every single file.


Also, keep in mind, we can easily test with the stand alone version of VoodooAi, which does not have the blacklist scan… that is the whole purpose of the app. Either way, the result is the same.


If you get a chance, do you mind posting this? Thank you!"

 

[DardiM]

Hey so I got a contact from Voodosheild support and they told me they wanted me to post this. Voodosheild Dev response for the comment by @DardiM
"...
All of the software was tested using the default settings, and all of the definitions were updated.
...

They could directly contact me !? I'm a friendly person
A tool that make 100 % on 1000 samples isn't too good to be true !?
Can they give us a link to their 1000 samples ? To make independent tests (Why do you say that I am dreaming ?)

 

[DJ Panda]

A tool that make 100 % on 1000 samples is too good to be true !?
Can they give us a link to their 1000 samples ? To make independent tests (Why do you say that I am dreaming ?)

Checking on that right now.

 

[Atlas147]

Also the 1000 samples might not have all been malicious, ESET and Kaspersky have been known to thoroughly check their samples before marking it as malicious unlike a lot of the other AV vendors. So the samples run by many of the AVs might have been clean.

 

[DardiM]

Checking on that right now.

And the result ? did they send you the samples ?

 

[TheMalwareMaster]

A lot of adware in this test! I can see some games (probably safe). Quite surprised by Avira. I'd like to know more about the tool used for the test (the one on the top right). For norton, I would say in this test scenario was disadvantaged. If the pack was downloaded from the internet, download insight would have worked and blocked most of the threats

 

[Logethica]

Excellent Video....
I cannot speak highly enough of VoodooShield. I absolutely consider it to be the best security software I have used.
IMO VS and VS-AI are Incredible....and in addition to this,the developer (Dan) not only appears to be an extremely ethical and principled guy, but also has more frequent,supportive,informative,& friendly interaction with VS users than any other dev that I have seen.
IMO a good Anti-EXE is the most important soft on a machine...and to anybody that doesn't already have one I Very,Very Strongly recommend VoodooShield.

 

[DJ Panda]

Excellent Video....
I cannot speak highly enough of VoodooShield. I absolutely consider it to be the best security software I have used.
IMO VS and VS-AI are Incredible....and in addition to this,the developer (Dan) not only appears to be an extremely ethical and principled guy, but also has more frequent,supportive,informative,& friendly interaction with VS users than any other dev that I have seen.
IMO a good Anti-EXE is the most important soft on a machine...and to anybody that doesn't already have one I Very,Very Strongly recommend VoodooShield.

Would it be ok running Voodosheild alongside other products like an AV or AM?

 

[DardiM]

I 've just done a Static test using the 1000 samples with KTS and custom settings.
Will test dynamically the sample not detected soon after my series

 

[Logethica]

Would it be ok running Voodosheild alongside other products like an AV or AM?

Yes...There is no evidence of incompatibility with any AV or AM.
The stable version is 2.86 i think from memory, but IMO the BETA Versions (Currently at approx 3.28-3.29) are equally stable.
There is currently a very,very small freeze issue that affects a small percentage of users & they are currently helping Dan to Iron this out (Which he definitely will)...
I (Like most) have not had this issue,and run VS alongside Avast Free,Sandboxie,ZoneAlarm Firewall,Crystal Security,MAE,Spyshelter,and with UAC set to max...
The support forum is on Wilders,...and (as I said) I think VS is Incredible

 

[DardiM]

Also the 1000 samples might not have all been malicious, ESET and Kaspersky have been known to thoroughly check their samples before marking it as malicious unlike a lot of the other AVs vendors. So the samples run by many of the AVs might have been clean.

Static scan with KTS => 166 files not detected => 81 files safe or unknown by VirusTotal => only 85 potential malware files not detected.

Détails :

Crystal security :

- White-listed: 102 files
=> 57 safe, 24 unknown (0/56 AVs) => 81
=> 21 suspicious ( <= 5% detection ratio - on 56 AVs)

- Blacklisted : 64 files ( >= 10 % detection - on 56 AVs)

My Personal Conclusion :
Only with static scan
=> KTS = (1000 - (166 - 81)) x 0.1 = 91.5 % (excluding the 81 samples that are not known as real malware)
=> Very far from the video result
=> It should be the same for a lot of AVs listed on this video.

=> Voodoo Shield is certainly a (very) good tool, but I think stopping 100 % with only 91,9 % real malware, makes the results of AVs tested lower than it must be in reality.

Doesn't it depend a lot on the settings ?
For example, my Kaspersky Total Security settings (modified after read a @harlan4096 post), doesn't let any file to run (at least the first time) without my consent (even ZAM, MBAM, etc...) by Application Control : I set it to put in Untrusted group all unknown files (even with a digital signature)

It would be interesting to see with dynamic test , but not with my custom setting, how many of the 85 only malware not detected by KTS in static scan, are stopped ... another day ...another thread )

 

[Deleted member 2913]

Doesn't it depend a lot on the settings ?
For example, my Kaspersky Total Security settings (modified after read a @harlan4096 post), doesn't let any file to run (at least the first time) without my consent (even ZAM, MBAM, etc...) by Application Control : I set it to put in Untrusted group all unknown files (even with a digital signature)

Pingouin critical sense - ON
I saw It's a test done by Voodoo Shield Dev ...
=> Voodoo Shield on AutoPilot Efficacity Test =>100 %
It's not a test by independent entity : sure that they made the settings to disadvantage the others (even before recording this video )
BTW, they should have put safes files => 100 % blocked too ?
Pingouin critical sense - OFF

In any case, thanks for sharing it

IMHO the test is good.

Default v/s Custom settings - I think one should take the results as per the settings used. Default settings are same for all the users whereas Custom settings are not same i.e depends on users.

Default settings tests gives you an idea how the products performed with the recommended settings used by majority.
Custom settings tests is not easy i.e not all users custom settings are same.

Like you mentioned your Custom settings. Some users also add Personal folders to Kaspersky to protect from Ransomware. So even if a test is done with Custom settings, some users will find some options not enabled/disabled & the product not tested properly.

Trust the test/settings, etc... - I will leave it to you.
For me, I know & trust him. Like I know many users here & trust them.