Reply to thread

Message

<blockquote data-quote="Andy Ful" data-source="post: 1050931" data-attributes="member: 32260">Such videos can be used only when the number of samples is very small or a very specific kind of test is performed.In the first case, the results are not statistically meaningful to compare AVs (sometimes the results can be useful for other things, like showing a possible weakness or demonstrating how the AV works).In the second case, the methodology has nothing to do with the real scenario, so the results can be incorrect.The most important part of the test is how representative is the pule of tested samples, compared to the samples in the wild. Unfortunately, this information is absent in all tests (also made by professional AV testing labs). Most people believe that professional AV testing labs can prepare more representative samples because they share samples with AV vendors and use the testing methodology approved by AMTSO.For me, the advantage of professional tests follows from systematic testing. So, several AVs are tested for a long time month by month, with the same methodology. Such testing can be statistically verified and the comparison results can be statistically meaningful.Comparing AVs without proper statistics is like crystal-ball gazing.</blockquote>

[QUOTE="Andy Ful, post: 1050931, member: 32260"] Such videos can be used only when the [B][COLOR=rgb(0, 168, 133)]number of samples is very small[/COLOR][/B] or a [COLOR=rgb(41, 105, 176)][B]very specific kind of test is performed[/B][/COLOR]. [COLOR=rgb(0, 168, 133)][B]In the first case[/B][/COLOR], the results are not statistically meaningful to compare AVs (sometimes the results can be useful for other things, like showing a possible weakness or demonstrating how the AV works). [B][COLOR=rgb(41, 105, 176)]In the second case[/COLOR][/B], the methodology has nothing to do with the real scenario, so the results can be incorrect. [B][COLOR=rgb(184, 49, 47)]The most important part of the test is how representative is the pule of tested samples, compared to the samples in the wild.[/COLOR][/B] Unfortunately, this information is absent in all tests (also made by professional AV testing labs). Most people believe that professional AV testing labs can prepare more representative samples because they share samples with AV vendors and use the testing methodology approved by AMTSO. For me, the advantage of professional tests follows from systematic testing. So, several AVs are tested for a long time month by month, with the same methodology. Such testing can be statistically verified and the comparison results can be statistically meaningful. Comparing AVs without proper statistics is like crystal-ball gazing. [/QUOTE]

Verification