Hi,
I have met Andreas and Peter from AV-Comparatives a few years ago at Prague when they were speakers at an Avast held event (because they had taken over AVG). Their real world test is based on infections delivered through browser. So when people start to download malware packs and execute them (allowing UAC elevation), then the result are different. Same applies to slightly changed samples which are executed from disk.
To the (valued and respected) forum members - who download malware packs and run them in virtual machine to find out how AV's do in such a settting - I can also say (tongue in cheek): there is no (software) medicine against user stupidity. So when people are so naief to run software from unknown sources, what would you expect? It is like igniting Chinese fireworks in a closed toilet and being surprised you got your @$$ burned.
Tests executed in a different setting than AV-Comparatives real world tests will always show different results. This does not say anything about the quality of the AV-C test method (which is ISO and EICAR certified). They collect samples over a period (usually a month) and run them (in batches) simultaneously on PC's in seperated environt each with their own seperated internet connection.
The only non-transparent part of their test setup is that they don't publish any information about the samples used (like age and malware families). When I do the math on their average sample size (on average around 150-200 collected in a month), this means that 5 to 7 samples are less than a day old. This means that any AV scoring less than 99.5% probably misses all those fresh samples.
With this in mind a protection percentage of 99.5% suddenly does not look that good. But that is only true for the first victim. Average chance of being first victim is less than 0.0002% to 0.0004% (I once did the math on wilders), so in real world situation this protection is not that bad. For comparison, the chances of being involved in a traffic incident are higher and the chances of surviving an aircraft incident are much lower. But I tend to look at the bright side of life (so run a default deny without AV
)
Regards Kees