THERE’S TESTING, THEN THERE’S VIRUSTOTAL
They're not the same thing...
From time to time, I find myself having to rail against the misuse of VirusTotal’s service as a sort of surrogate AV product test or a system to determine a maliciousness of a sample. But I don't think I've done it here before, so here I go.
VirusTotal subjects the files people submit to a battery of command-line scanners, as a means of finding out whether or not it's malware. That’s fine for a rough assessment of the likelihood that a submitted file is malicious, but it isn't a detection test. On one hand, the fact that a scanner doesn't identify a file as malware does not mean it isn't malicious, of course, and we shouldn't forget that. On the other hand, if a file is identified as malicious by one or more scanners but not by others, it doesn't necessarily mean that the scanners that don’t flag it as malicious are incompetent or the file is real malware as FPs can be caused by packer detections.
Obviously, sometimes a file is misdiagnosed as malicious (a false positive) and sometimes that false positive may ‘cascade’ through more than one lab because of insufficient checking when samples are shared.
VirusTotal doesn’t exercise the whole functionality of an anti-virus product, still very less that of a security suite: it uses command-line program versions, rather than the interactive and/or real-time scanning functionality that even a simple commercial scanner has.
Generally, command-line scanners simply look at the code passively: unless they run the code in a safe environment (emulation or sandboxing) to see what it really does, they may not recognize some malcode that their on-access components would have recognized.
In other words, VirusTotal doesn’t tell you whether the whole product is capable of detecting a malicious file. At best, it tells you whether it’s sig engine capable of detecting it using that particular program module and configuration.
It does NOT cover Sandboxing,string detections,cloud,Behaviour Blocks,URL blocking.
There has to be some relation with the samples used in any sort of AV testing from what I can say:
1)You know the origin/behavior/way of spreading of the sample (it comes from a machine that you recently disinfected e.g.)
2)You are sure to properly filter the broken files/same malware but different file from your set.Whether or Not there is a VT Hit you should know whether or not the sample or URL does any harm to the system.You cannot call a miss just because your AV didn't block the URL.What if the site was cleaned earlier and then the blacklisters and you added it to your list later.
3. you're able to write related metadata either to VT comments or here
You should be very careful from what you read off virustotal results without prior analysis of a sample.
The reason the samples picked up from VT should be properly tested first is Because other AVs are unreliable as the source of the detection. Firstly, because they're FP infested and the second problem is that some vendors like to play games by creating innocent samples with their detections and then measuring how many other AVs are caught by the trap.See:
http://www.securelist.com/en/weblog?weblogid=208188011
Kaspersky defends false detection experiment
They're not the same thing...
From time to time, I find myself having to rail against the misuse of VirusTotal’s service as a sort of surrogate AV product test or a system to determine a maliciousness of a sample. But I don't think I've done it here before, so here I go.
VirusTotal subjects the files people submit to a battery of command-line scanners, as a means of finding out whether or not it's malware. That’s fine for a rough assessment of the likelihood that a submitted file is malicious, but it isn't a detection test. On one hand, the fact that a scanner doesn't identify a file as malware does not mean it isn't malicious, of course, and we shouldn't forget that. On the other hand, if a file is identified as malicious by one or more scanners but not by others, it doesn't necessarily mean that the scanners that don’t flag it as malicious are incompetent or the file is real malware as FPs can be caused by packer detections.
Obviously, sometimes a file is misdiagnosed as malicious (a false positive) and sometimes that false positive may ‘cascade’ through more than one lab because of insufficient checking when samples are shared.
VirusTotal doesn’t exercise the whole functionality of an anti-virus product, still very less that of a security suite: it uses command-line program versions, rather than the interactive and/or real-time scanning functionality that even a simple commercial scanner has.
Generally, command-line scanners simply look at the code passively: unless they run the code in a safe environment (emulation or sandboxing) to see what it really does, they may not recognize some malcode that their on-access components would have recognized.
In other words, VirusTotal doesn’t tell you whether the whole product is capable of detecting a malicious file. At best, it tells you whether it’s sig engine capable of detecting it using that particular program module and configuration.
- The intention was to ‘determine the effectiveness of on demand scanners with a random collection of malware’. This misses the point entirely: competent AV testing isn’t about random samples. On the contrary, samples have to be verified and correctly classified. (And scanners have to be configured accordingly in order to provide an accurate test.) VirusTotal didn’t do this (which is fine, because VT itself is very definite about the fact that its service is not suitable for product testing. Whether it’s a single product or all AV products "under test" is beside the point.
- Anti-virus companies use VirusTotal in their own blog posts to indicate how many manufacturers detect specific malware. And yes, actually, we do from time to time, normally as a rough guide to whether the industry as a whole is aware of a very new threat. And perhaps we should make it clearer when we do so that it is only a (very) rough guide, and not a poorly conceived or deliberately misleading attempt to compare the performance of other products to our own.
It does NOT cover Sandboxing,string detections,cloud,Behaviour Blocks,URL blocking.
There has to be some relation with the samples used in any sort of AV testing from what I can say:
1)You know the origin/behavior/way of spreading of the sample (it comes from a machine that you recently disinfected e.g.)
2)You are sure to properly filter the broken files/same malware but different file from your set.Whether or Not there is a VT Hit you should know whether or not the sample or URL does any harm to the system.You cannot call a miss just because your AV didn't block the URL.What if the site was cleaned earlier and then the blacklisters and you added it to your list later.
3. you're able to write related metadata either to VT comments or here
You should be very careful from what you read off virustotal results without prior analysis of a sample.
The reason the samples picked up from VT should be properly tested first is Because other AVs are unreliable as the source of the detection. Firstly, because they're FP infested and the second problem is that some vendors like to play games by creating innocent samples with their detections and then measuring how many other AVs are caught by the trap.See:
http://www.securelist.com/en/weblog?weblogid=208188011
Kaspersky defends false detection experiment
Last edited:
