App Review 1,000 Malware Sample Pre-Execution Efficacy Test - Malware Test

[Deleted member 2913]

Static scan with KTS => 166 files not detected => 81 files safe or unknown by VirusTotal => only 85 potential malware files not detected.

Détails :

Crystal security :

- White-listed: 102 files
=> 57 safe, 24 unknown (0/56 AVs) => 81
=> 21 suspicious ( <= 5% detection ratio - on 56 AVs)

- Blacklisted : 64 files ( >= 10 % detection - on 56 AVs)

My Personal Conclusion :
Only with static scan
=> KTS = (1000 - (166 - 81)) x 0.1 = 91.5 % (excluding the 81 samples that are not known as real malware)
=> Very far from the video result
=> It should be the same for a lot of AVs listed on this video.

=> Voodoo Shield is certainly a (very) good tool, but I think stopping 100 % with only 91,9 % real malware, makes the results of AVs tested lower than it must be in reality.
It would be interesting to see with dynamic test , but not with my custom setting, how many of the 85 only malware not detected by KTS in static scan, are stopped ... another day ...another thread )

Files not detected at VT doesn't means not malicious, etc... May be adware, PUP, bundled adware & PUP, etc... And it depends on the users if he thinks those stuffs should be detected or not.

Dont know if you tested with VT or Crystal Security for VT results? I have noticed quite a few times Crystal Security gives no detection on VT but there are detection on VT for the sample. I admit haven't tested the latest Crystal Security. This was my experience with couple previous versions of CS. Just my experience & no offense here.

Undetected samples on VT may be adware, bundled stuffs, etc... or may be no PUP, bundled stuffs & safe?

And as per VS Dev, 954/1000 were quarantined by Zamana.

 

[DardiM]

Files not detected at VT doesn't means not malicious, etc... May be adware, PUP, bundled adware & PUP, etc... And it depends on the users if he thinks those stuffs should be detected or not.

Dont know if you tested with VT or Crystal Security for VT results? I have noticed quite a few times Crystal Security gives no detection on VT but there are detection on VT for the sample. I admit haven't tested the latest Crystal Security. This was my experience with couple previous versions of CS. Just my experience & no offense here.

Undetected samples on VT may be adware, bundled stuffs, etc... or may be no PUP, bundled stuffs & safe?

And as per VS Dev, 954/1000 were quarantined by Zamana.

"Files not detected at VT doesn't means not malicious" => and doesn't mean malicious

(none of the file 166 last files can run on my custom setting, then with a dynamic test => 100 %)

I tested with Crystal Security all engines only the 166 sample not detected by KTS.

- 57 reported as safe
- only 24 unknown

With today's update ZAM => 961/1000 detected

I'm really impressed by the 100 % (even if these are old samples). But I can't stop asking myself one thing :

What about "Also, keep in mind, all software was tested in their default settings… simply because the absolute vast majority of users run with all default settings" ?!

From VoodooShield (on their youtube)
"We did have to make some adjustments to VoodooShield so that it would act more like a traditional antivirus software as opposed to an application whitelister, and all of the adjustments are shown at the beginning of the VoodooShield test"

So, a "normal" user who purchase VS and install it whit default setting will it have the 100 % result ?

I will answer myself : purchased today

 

[DJ Panda]

@DardiM More messages from Voodoosheild

"Can you please explain to DardiM that 954 of the files were quarantined by Zemana. The rest were analyzed by Cuckoo, and all but a handful (2-5) were definitely malware. Sure, there was some adware, but adware IS malware… even AV testing labs agree and include adware in their samples, and often times it is more malicious then other types of malware. BTW, the 2-5 samples are probably malware as well… he has to remember that malware analysis is not perfect.



And actually, adware is more difficult to detect then most super bad malware, so it is best to use random samples with all different types of malware… ESPECIALLY since adware is more difficult to detect. That is, if you really want to know the true efficacy of a given product.



Sure, I could have cherry picked certain types of malware, but that would have skewed the results in favor of whatever engines detected the malware while pre-filtering the samples. Random is ALWAYS better… even if it does include a handful of questionable samples… but that is why you test with a larger sample size, like 1,000 or 3,000.



As far as the unknowns go… just because his test was unable to detect these files as malware, does not mean that they were not malware. In fact, most likely it is quite the opposite.



Thank you!"

 

[DardiM]

My main answer is on my post before yours, @J Gamez065
You can send it if you want

And , as I have purchased their tool (version pro), i'm making actually tests for the moment WITH DEFAULT SETTING (like the AVs they tested ), on the 166 files not detected by KTS

The major part is warning me to quarantine or block the files, but I've seen several time :


=> Allow "acceptable" blue : safe - purple (when 1 FP has been reported)



=> "Quarantine" suggested because unknown => suspicious

"I'm really impressed by the 100 % (even if these are old samples). But I can't stop asking myself one thing :

What about "Also, keep in mind, all software was tested in their default settings… simply because the absolute vast majority of users run with all default settings" ?!

From VoodooShield (on their youtube)
"We did have to make some adjustments to VoodooShield so that it would act more like a traditional antivirus software as opposed to an application whitelister, and all of the adjustments are shown at the beginning of the VoodooShield test"

So, a "normal" user who purchase VS and install it whit default setting will it have the 100 % result
=> no

Is it a good tool ?
=> yes (I purchased it) because I liked what I've seen, using it, but not for the 100 %.

WARNING THIS IS ONLY MY PERSONAL ANALYSE, FEELING - DON'T SEND ME PIT BULLS

 

[Deleted member 2913]

From VoodooShield (on their youtube)
"We did have to make some adjustments to VoodooShield so that it would act more like a traditional antivirus software as opposed to an application whitelister, and all of the adjustments are shown at the beginning of the VoodooShield test"

So, a "normal" user who purchase VS and install it whit default setting will it have the 100 % result ?

I will answer myself : purchased today

He didn't changed any settings that will change the outcome, the outcome will be same for the changed settings & default settings.

He changed a "Parent process" setting as the tool that was used to run malware/check efficacy, malware run through the tool would be considered the child process of the tool & allowed. So if you run the malware normal then no need to change the setting.

And other changed settings were just normal like instead of 20 secs for popup to remain on screen, he reduced the time & couple such settings that doesn't affect the outcome or protection.

 

[Deleted member 2913]

View attachment 105932
=> "Quarantine" suggested because unknown => suspicious

Quarantine is not suggested coz of Unknown.

The verdict on popup is result of Blacklist Scan + VoodooAi combined results.
The popup in question mention that the sample is unknown to blacklist scan which is suspicious i.e the sample is not available to 56 engines/blacklist scan.
And the popup also mention VoodooAi detected the sample as Unsafe.
So the combined verdict "Quarantine" for Unknown + Unsafe sample.

 

[DardiM]

He didn't changed any settings that will change the outcome, the outcome will be same for the changed settings & default settings.
He changed a "Parent process" setting as the tool that was used to run malware/check efficacy, malware run through the tool would be considered the child process of the tool & allowed. So if you run the malware normal then no need to change the setting.
And other changed settings were just normal like instead of 20 secs for popup to remain on screen, he reduced the time & couple such settings that doesn't affect the outcome or protection.

Quarantine is not suggested coz of Unknown.
The verdict on popup is result of Blacklist Scan + VoodooAi combined results.
The popup in question mention that the sample is unknown to blacklist scan which is suspicious i.e the sample is not available to 56 engines/blacklist scan.
And the popup also mention VoodooAi detected the sample as Unsafe.
So the combined verdict "Quarantine" for Unknown + Unsafe sample.

Thanks for these informations

I repeat, I liked their product when testing yesterday/today, I purchased it, but :
what about the several (few, but there was some) samples that make "Blue" and "Purple" warnings that suggested me to "Allow" => if the others AVs didn't detected these files as malware too, in their test, how was the impact on their % rate ?
(Because for me these popup show that there was not 1000 malware)

 

[Deleted member 2913]

Thanks for these informations

I repeat, I liked their product when testing, I bought it, but :
what about the several (few, but there was some) samples that make "Blue" and "Purple" warnings that suggested me to "Allow" => if the others AVs didn't detected these files as malware too, in their test, how was the impact on their % rate ?
(Because for me these popup show that there was not 1000 malware)

I think -
Blue popup - Blacklist scan "0" detection & VAi calculation "Safe" & combined verdict Allow
Purple popup - Blacklist scan FP & VAi calculated Safe & combined verdict Allow

As per VS Dev there is an FPs engine that decides FPs on blacklist scan detection based on good AVs with low FPs & FPs prone AVs.

"The blacklist scanner is overwhelmed when that many samples are thrown at it at once, and when this happens, the composite VoodooAi score can change, so it depends on a lot of factors" - This is mentioned in one of VS Devs posts on Wilders.

I will mention your test & request info from VS Dev.

 

[Deleted member 2913]

DardiM,

I mentioned your test to VS Dev.

His reply -
"Simple, here is a good example... please look at sample # 9 out of 1000 at 39:41 in the video (which I have already discussed on the wilders VoodooShield? thread). The blacklist false positive adjusted the composite VoodooAi score to 0.3608, which is just over 0.3333... the threshold for VoodooAi to allow an item when VS is on AutoPilot.

There are probably 4-5 samples that are similar to this, but there will be a logical explanation for them too.

Remember, VS has different thresholds depending on the mode, so if you test in a different mode, then you might receive a different result.

But that is why I always recommend people test with the stand alone version of VoodooAi, if they are interested in seeing how VoodooAi performs on its own. That is actually the main reason why I created the stand alone version... so people can test the efficacy of only VoodooAi, so they know there is no funny business going on with the blacklist scan. Fire up fiddler and test until your heart is content .

It will not be an issue either way now, since you can disable both the blacklist scan and VoodooAi, and you can test however you like".

 

[DardiM]

[

DardiM,

I mentioned your test to VS Dev.

His reply -
"Simple, here is a good example... please look at sample # 9 out of 1000 at 39:41 in the video (which I have already discussed on the wilders VoodooShield? thread). The blacklist false positive adjusted the composite VoodooAi score to 0.3608, which is just over 0.3333... the threshold for VoodooAi to allow an item when VS is on AutoPilot.

There are probably 4-5 samples that are similar to this, but there will be a logical explanation for them too.

Remember, VS has different thresholds depending on the mode, so if you test in a different mode, then you might receive a different result.

But that is why I always recommend people test with the stand alone version of VoodooAi, if they are interested in seeing how VoodooAi performs on its own. That is actually the main reason why I created the stand alone version... so people can test the efficacy of only VoodooAi, so they know there is no funny business going on with the blacklist scan. Fire up fiddler and test until your heart is content .

It will not be an issue either way now, since you can disable both the blacklist scan and VoodooAi, and you can test however you like".

Thanks a lot, it's now more clear for me on what their test means .
As I purchased it, I 'm going to "play" with it

 

[Deleted member 2913]

[

Thanks a lot, it's now more clear for me on what their test means .
As I purchased it, I 'm going to "play" with it

Are you trying latest beta 3.29?

 

[DardiM]

That's the 3.28 beta version (downloaded this afternoon when purchased, on their website - France time => 8h ago)

 

[Smith83]

Well, I tested many many many samples with VS and ZemanAntimalware Premium, and absolutely nothing was able to get by including ransom adware rootkits etc...

Best part is with the two enabled on a single machine, there is very minimal impact on system performance.