Reply to thread

Message

<blockquote data-quote="Andy Ful" data-source="post: 1022835" data-attributes="member: 32260">The readers probably noticed that I had a problem with using the sample size calculator in the real scenario of AV testing. What was the problem?The calculator uses only one parameter &quot;Population Proportion&quot; (which is easy to understand).[ATTACH=full]272620[/ATTACH]But, in the AV test, we have several AVs, and each AV can generate a different population proportion. Which one should be chosen? There is no natural choice. Should we use the average &quot;Population Proportion&quot; over all tested AVs? Another possibility would be using &quot;Population Proportion&quot; = 80% (or 20%) which would approximately describe truly never-before-seen malware which was not blocked by any AV.After a few hours, I realized that by using some trick the calculator can be used to compare the adjacent clusters, by selecting only 2 AVs (each from a different cluster) and using only the &quot;Population Proportion&quot; of one AV. The trick is that this will work only if the difference in the protection of AVs is not too big.In this post, I will improve this trick by using an average of the &quot;Population Proportion&quot; generated by both AVs. Here is an improved calculation for Norton and Microsoft from the two-year statistics:[ATTACH=full]272619[/ATTACH]For the Confidence Level 90% the sample size is 6632 (lower than in two-year statistics 2021-2022)For the Confidence Level 95% the sample size is 9323 (higher than in two-year statistics 2021-2022).In the two-year statistics (AV-Comparatives + AV-Test + SE Labs) the sample size is only 7548.So for the two-year statistics from OP, the Confidence Level is probably close to 92% (Norton is better than Microsoft).Edit, By using different absolute errors (6.18 for Norton and 9.32 for Microsoft) one can optimize the calculated sample size to 8950 with the Confidence Level 95%)</blockquote>

[QUOTE="Andy Ful, post: 1022835, member: 32260"] The readers probably noticed that I had a problem with using the sample size calculator in the real scenario of AV testing. What was the problem? The calculator uses only one parameter "Population Proportion" (which is easy to understand). [ATTACH type="full" alt="1675351940537.png"]272620[/ATTACH] But, in the AV test, we have several AVs, and each AV can generate a different population proportion. Which one should be chosen? There is no natural choice. Should we use the average "Population Proportion" over all tested AVs? Another possibility would be using "Population Proportion" = 80% (or 20%) which would approximately describe truly never-before-seen malware which was not blocked by any AV. After a few hours, I realized that by using some trick the calculator can be used to compare the adjacent clusters, by selecting only 2 AVs (each from a different cluster) and using only the "Population Proportion" [B][COLOR=rgb(184, 49, 47)]of one AV[/COLOR][/B]. The trick is that this will work only if the difference in the protection of AVs is not too big. In this post, I will improve this trick by [B][COLOR=rgb(184, 49, 47)]using an average[/COLOR][/B] of the "Population Proportion" generated[B][COLOR=rgb(184, 49, 47)] by both AVs[/COLOR][/B]. Here is an improved calculation for Norton and Microsoft from the two-year statistics: [ATTACH type="full" alt="1675350984032.png"]272619[/ATTACH] For the Confidence Level 90% the sample size is 6632 (lower than in two-year statistics 2021-2022) For the Confidence Level 95% the sample size is 9323 (higher than in two-year statistics 2021-2022). In the two-year statistics (AV-Comparatives + AV-Test + SE Labs) the sample size is only 7548. So for the two-year statistics from OP, the Confidence Level is probably close to 92% (Norton is better than Microsoft). Edit, By using different absolute errors (6.18 for Norton and 9.32 for Microsoft) one can optimize the calculated sample size to 8950 with the Confidence Level 95%) [/QUOTE]

Verification