- Apr 25, 2013
- 5,355
Many independent antivirus testing labs have taken to calling Microsoft Security Essentials their baseline, separate from the products undergoing testing. If an antivirus can't do better than Microsoft, it's a poor product indeed. However, Dennis Batchelder, director of the Microsoft Malware Protection Center (MMPC), contends that lab tests don't reflect the product's actual user protection, and that in the real world Microsoft is much more effective than the tests show. A recent test suggests that just might be true.
World-Wide Telemetry
The basis of Batchelder's claim is that Microsoft's researchers know more about the actual prevalence of specific malware families than just about anybody. Why? Because every Patch Tuesday the Malicious Software Removal Tool both deletes prevalent malware and reports a raft of non-personal information back to Microsoft. The returned telemetry includes what (if any) threats were neutralized, but also tells them the Windows version, the version of any installed antivirus software, whether that software is up to date, and more.
The MMPC website offers visitors a lightweight summary of current statistics. Under the hood, they've got vastly more data, and they use that data to prioritize protection against the most dangerous and most prevalent malware threats. Or so they say.
Considering Prevalence
Microsoft commissioned the well-known lab AV-Comparatives to re-evaluate a recent test taking prevalence of samples into account. This was a simple file detection test—run an antivirus scan with each product and note how many of over 100,000 samples it detects.
The samples are selected to represent malware prevalent in the wild and to avoid over-representation of any one malware family. However, in calculating the detection rate, every sample gets the same weight. The new report takes the same data and applies weighting based on Microsoft's reported prevalence. The results were vastly different from the original, as you can see in the chart below.
Weighting didn't change the top and bottom scores. Kaspersky Lab is still number one, and AhnLab is still in the cellar. But other rankings changed dramatically. Instead of being second-to-last, Microsoft ranked better than three-quarters of the competition. And aside from Kaspersky, all those ranked above Microsoft came up from lower rankings.
By the same token, most of the lowest-ranked products started off much higher. Bitdefender, Lavasoft, Kingsoft, Emsisoft, Qihoo, and BullGuard had originally tied for sixth place. After weighting for prevalence they're ranked from 15th place on down. Baidu took the biggest plunge, from second place to 22nd. Why? Because while it didn't miss many samples, the ones it did miss were extremely widespread.
The full report from AV-Comparatives describes the weighting scheme in detail and also offers country-specific analysis showing each product's performance on a global map. It states, "This report should be regarded as a prototype, the purpose of which is spark debate on the significance of prevalence data, and promote ideas for improving the method," and expresses a hope that other vendors will share telemetry data with Microsoft "in order to get a more significant and impartial customer-impact analysis."
That sounds like a good plan to me. The radically different results using Microsoft's prevalence data alone suggests that we need data from a broader set of sources.
World-Wide Telemetry
The basis of Batchelder's claim is that Microsoft's researchers know more about the actual prevalence of specific malware families than just about anybody. Why? Because every Patch Tuesday the Malicious Software Removal Tool both deletes prevalent malware and reports a raft of non-personal information back to Microsoft. The returned telemetry includes what (if any) threats were neutralized, but also tells them the Windows version, the version of any installed antivirus software, whether that software is up to date, and more.
The MMPC website offers visitors a lightweight summary of current statistics. Under the hood, they've got vastly more data, and they use that data to prioritize protection against the most dangerous and most prevalent malware threats. Or so they say.
Considering Prevalence
Microsoft commissioned the well-known lab AV-Comparatives to re-evaluate a recent test taking prevalence of samples into account. This was a simple file detection test—run an antivirus scan with each product and note how many of over 100,000 samples it detects.
The samples are selected to represent malware prevalent in the wild and to avoid over-representation of any one malware family. However, in calculating the detection rate, every sample gets the same weight. The new report takes the same data and applies weighting based on Microsoft's reported prevalence. The results were vastly different from the original, as you can see in the chart below.
Weighting didn't change the top and bottom scores. Kaspersky Lab is still number one, and AhnLab is still in the cellar. But other rankings changed dramatically. Instead of being second-to-last, Microsoft ranked better than three-quarters of the competition. And aside from Kaspersky, all those ranked above Microsoft came up from lower rankings.
By the same token, most of the lowest-ranked products started off much higher. Bitdefender, Lavasoft, Kingsoft, Emsisoft, Qihoo, and BullGuard had originally tied for sixth place. After weighting for prevalence they're ranked from 15th place on down. Baidu took the biggest plunge, from second place to 22nd. Why? Because while it didn't miss many samples, the ones it did miss were extremely widespread.
The full report from AV-Comparatives describes the weighting scheme in detail and also offers country-specific analysis showing each product's performance on a global map. It states, "This report should be regarded as a prototype, the purpose of which is spark debate on the significance of prevalence data, and promote ideas for improving the method," and expresses a hope that other vendors will share telemetry data with Microsoft "in order to get a more significant and impartial customer-impact analysis."
That sounds like a good plan to me. The radically different results using Microsoft's prevalence data alone suggests that we need data from a broader set of sources.
