Reply to thread

Message

<blockquote data-quote="Andy Ful" data-source="post: 1025983" data-attributes="member: 32260">Testing some AVs in the automated scenario is very difficult, so we should give credit to the efforts of AVLab.The testing methodology has been significantly changed and some time is required to solve all possible issues.The AVLab tests are not included on the AMSTO webpage:[URL unfurl=&quot;false&quot;]https://www.amtso.org/tests/[/URL]But the current testing methodology seems OK for most AVs.I could not find the info about which samples were tested. In previous tests, almost all samples were *.exe files which are only about 1/3 of malware in the wild. This can be a problem for Real-World protection tests.There is also an open question related to morphed samples. If morphed samples are allowed, then the statistical variations are bigger, and much more samples are required to make the results statistically significant.Even when morphed samples are not allowed, over 10000 samples are required to see a real difference between Microsoft Defender free and Kaspersky (KIS), with a confidence level of 95%.</blockquote>

[QUOTE="Andy Ful, post: 1025983, member: 32260"] Testing some AVs in the automated scenario is very difficult, so we should give credit to the efforts of AVLab. The testing methodology has been significantly changed and some time is required to solve all possible issues. The AVLab tests are not included on the AMSTO webpage: [URL unfurl="false"]https://www.amtso.org/tests/[/URL] But the current testing methodology seems OK for most AVs. I could not find the info about which samples were tested. In previous tests, almost all samples were *.exe files which are only about 1/3 of malware in the wild. This can be a problem for Real-World protection tests. There is also an open question related to morphed samples. If morphed samples are allowed, then the statistical variations are bigger, and much more samples are required to make the results statistically significant. Even when morphed samples are not allowed, over 10000 samples are required to see a real difference between Microsoft Defender free and Kaspersky (KIS), with a confidence level of 95%. [/QUOTE]

Verification