F
ForgottenSeer 97327
Yes, that is right. But we know that if the statistical fluctuations are big, then the number of tests is insufficient.
This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
We encourage you to compare these results with others and take informed decisions on what security products to use.
Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.
Yes, that is right. But we know that if the statistical fluctuations are big, then the number of tests is insufficient.
The monthly changes have no impact on the statistic in this thread (it is a two-year cumulative statistic).Not when the population you are researching changes all the time, there you are wrong. The active in the wild-malware changes every month (and also their prevalence)
![]()
You claim that the changes in results are an indication of to small sample set. But this only applies for more or less stable populations. So taking two years of monthly results is arbitrary (since it is not applicable on changing populations).The monthly changes have no impact on the statistic in this thread (it is a two-year cumulative statistic).
fwiw, I took several university courses in statistics eons ago, nuances forgotten, and between you and @Andy Ful this discussion seems a little "apples and oranges" -- but friendly too!You claim that the changes in results are an indication of to small sample set. But this only applies for more or less stable populations. So taking two years of monthly results is arbitrary (since it is not applicable on changing populations).
When you are doing a poll on how many people have a valid driving license in Poland, the number of inhabitants only change marginally (people die and children are born). When you are testing 100 to 250 malware samples every month , the malware population of which this sample is taken changes every month (new malwares appear and malware prevelence change).
Some claim that half a million new malware appears every day. Researchers say that around half a million new malware binaries appear every day, of which only a fraction are unique (less than 0.5 percent) from even a lower number of families (less than 0,1 percent). This clearly shows that the 100 to 250 in the wild malware samples are probably taken from a heavily changed population, explaining the high difference in outcome.
This is different from the "do you have a driving license poll in Poland" where the number of people who could be potentially asked (the population) only marginally changes in a month. According to Goofle Poland jad 37.5 million inhabitants in 2021, with a total of 500.000 death and 300,000 born the total population in Poland only changes 2.1 percent per year of .2 per month.
So you are right that real world malware sample sets are small, but when there are 500.000 malware executables new per day of which only 0,35 % are unique, this would mean that around 1750 unique malware every day total up to around 60.000 every month. According to the sample test calculator a sample size of 382 would be sufficient.
When you take into account that unique malware can often be identified because they are part of a malware family (generic fingerprints), the sample set needed could further decrease. This would imply that the real world test samples (100 to 250 per month) used by the mainstream test labs could well be representative.
Not exactly. The fluctuations are always big if the number of tested samples in one test is very very small compared to the number of total samples. Even significant changes in the population cannot change it (until the number of tested samples is relatively small).You claim that the changes in results are an indication of to small sample set. But this only applies for more or less stable populations.
Yes, fwiw this is what I meant by my apples & oranges comment (even though you weren't replying directly to mine)It can be an example that significant changes in population can be not important for some statistical models.
From the testing methodology (for example AV-Comparatives), we know that one test (about 300 samples per month) cannot differentiate between AVs contained in the same award group (usually 2/3 of tested products). So, several tests are required.
This calculator uses a particular statistical model that is clearly incompatible with the data related to AV tests.
(1) Again that is not valid when the population itself changes a lot (new malwares every day). It is true for relatively stable populations (like demographic research)(1) The fluctuations are always big if the number of tested samples in one test is very very small compared to the number of total samples.
(2) Did you notice that this calculator is insensitive to the changes in the population? If you use a population size 100 times bigger (6000000), then you get the sample size 385. This calculator uses a particular statistical model that is clearly incompatible with the data related to AV tests.
It could help 50 years ago.Read this, it might help @Andy Ful to use the sample calculator correctly:
![]()
Population Proportion
Simple definition for the population proportion, written in plain English. Finding confidence intervals and sample proportions, step by steps plus videos.www.statisticshowto.com
To use your own words: "it is a very simple statistical model"![]()
As I read this thread, isn't there another variable, ie, eg, the Avast used in year2 or year3 is not necessarily the same Avast that was used in year1, ie, the vendors are "upgrading" or at least modifying their AV products every so often. Perhaps that is irrelevant, or is that just part of the averaging??The calculations with the "sample size calculator" partially confirm that a two-year cumulative statistic is usually required to find statistically significant differences between AVs. In some cases three-year (or more) testing period is needed. But, using a three-year or more, can be accepted only in the case when the AVs protection rate does not change significantly. So I probably stay with a two-year long-term period.
As I read this thread, isn't there another variable, ie, eg, the Avast used in year2 or year3 is not necessarily the same Avast that was used in year1, ie, the vendors are "upgrading" or at least modifying their AV products every so often. Perhaps that is irrelevant, or is that just part of the averaging??![]()