Forums
New posts
Search forums
News
Security News
Technology News
Giveaways
Giveaways, Promotions and Contests
Discounts & Deals
Reviews
Users Reviews
Video Reviews
Support
Windows Malware Removal Help & Support
Inactive Support Threads
Mac Malware Removal Help & Support
Mobile Malware Removal Help & Support
Blog
Log in
Register
What's new
Search
Search titles only
By:
Search titles only
By:
Reply to thread
Menu
Install the app
Install
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Forums
Security
Security Statistics and Reports
Randomness in the AV Labs testing.
Message
<blockquote data-quote="Andy Ful" data-source="post: 905937" data-attributes="member: 32260"><p>Let's assume that we have the below results for the test:</p><p>2 AVs with 0 undetected malware</p><p>4 AVs with 1 undetected malware</p><p>4 AVs with 2 undetected malware</p><p>3 AVs with 3 undetected malware</p><p>1 AV with 4 undetected malware</p><p></p><p>It can be very close to probabilities for hypothetical AV when:</p><ul> <li data-xf-list-type="ul">k = 60 (number of samples that compromised the hypothetical AV on the large pule of samples 300000);</li> <li data-xf-list-type="ul">m = 300000 (the number of samples in the large pule of samples);</li> <li data-xf-list-type="ul">n = 10250 (number of samples included in the AV Lab test);</li> </ul><p>We can calculate the probabilities:</p><p>p(0) = 0.124 (15.5 * 0.124 ~ 2)</p><p>p(1) = 0.264 (15.5 * 0.264 ~ 4)</p><p>p(2) = 0.275 (15.5 * 0.275 ~ 4)</p><p>p(3) = 0.188 (15.5 * 0.188 ~ 3)</p><p>p(4) = 0.095 (15.5 * 0.095 ~ 1)</p><p></p><p>We can see that these probabilities are approximately proportional to the number of AVs for the concrete amount of undetected malware. The proportionality constant is about 15.5 . We can compare this statistics to the AV-Comparatives Malware test for March 2020:</p><p>3 AVs with 0 undetected malware (F-Secure, G Data, NortonLifeLock)</p><p>4 AVs with 1 undetected malware (ESET, K7, TotalAV, Total Defense)</p><p>3 AVs with 2 undetected malware (Avast, AVG, Bitdefender)</p><p>3 AVs with 3 undetected malware (Avira, Kaspersky, VIPRE)</p><p>1 AV with 4 undetected malware (Panda)</p><p>[URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-march-2020/[/URL]</p><p></p><p>The difference is minimal. For example, if Norton and Total Defense would miss one malware more, then the results for 14 AVs would be very close to random trials for one hypothetical AV.</p><p></p><p>It seems that a similar conclusion was made by AV-Comparatives because it awarded all 14 AVs.</p><ul> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/bitdefender/" target="_blank">Bitdefender</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/eset/" target="_blank">ESET</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/g-data/" target="_blank">G DATA</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/kaspersky-lab/" target="_blank">Kaspersky</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/total-defense/" target="_blank">Total Defense</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/vipre/" target="_blank">VIPRE</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/avast/" target="_blank">Avast*</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/avg/" target="_blank">AVG*</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/avira/" target="_blank">Avira*</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/f-secure/" target="_blank">F-Secure*</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/k7-2/" target="_blank">K7*</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/nortonlifelock/" target="_blank">NortonLifeLock*</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/panda/" target="_blank">Panda*</a></li> <li data-xf-list-type="ul"><a href="https://www.av-comparatives.org/vendors/total-av/" target="_blank">Total AV*</a></li> </ul><p>The 8 products (*) got lower awards due to false alarms.</p><p></p><p>Edit.</p><p>It seems that the same conclusion can be derived from cluster analysis made in the report:</p><p></p><p>[ATTACH=full]247395[/ATTACH]</p><p>The AVs mentioned in my statistical model belong to the cluster one (see at the last column) and were avarded. Other AVs belong to other clusters.</p><p></p><p>Here is what AV-Comparatives say about the importance of clusters:</p><p>"<em>Our tests use much more test cases (samples) per product and month than any similar test performed by other testing labs. Because of the higher statistical significance this achieves, we consider all the products in each results cluster to be equally effective, assuming that they have a false-positives rate below the industry average</em>."</p><p><a href="https://www.av-comparatives.org/real-world-protection-test-methodology/" target="_blank">Real-World Protection Test Methodology - AV-Comparatives (av-comparatives.org)</a></p></blockquote><p></p>
[QUOTE="Andy Ful, post: 905937, member: 32260"] Let's assume that we have the below results for the test: 2 AVs with 0 undetected malware 4 AVs with 1 undetected malware 4 AVs with 2 undetected malware 3 AVs with 3 undetected malware 1 AV with 4 undetected malware It can be very close to probabilities for hypothetical AV when: [LIST] [*]k = 60 (number of samples that compromised the hypothetical AV on the large pule of samples 300000); [*]m = 300000 (the number of samples in the large pule of samples); [*]n = 10250 (number of samples included in the AV Lab test); [/LIST] We can calculate the probabilities: p(0) = 0.124 (15.5 * 0.124 ~ 2) p(1) = 0.264 (15.5 * 0.264 ~ 4) p(2) = 0.275 (15.5 * 0.275 ~ 4) p(3) = 0.188 (15.5 * 0.188 ~ 3) p(4) = 0.095 (15.5 * 0.095 ~ 1) We can see that these probabilities are approximately proportional to the number of AVs for the concrete amount of undetected malware. The proportionality constant is about 15.5 . We can compare this statistics to the AV-Comparatives Malware test for March 2020: 3 AVs with 0 undetected malware (F-Secure, G Data, NortonLifeLock) 4 AVs with 1 undetected malware (ESET, K7, TotalAV, Total Defense) 3 AVs with 2 undetected malware (Avast, AVG, Bitdefender) 3 AVs with 3 undetected malware (Avira, Kaspersky, VIPRE) 1 AV with 4 undetected malware (Panda) [URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-march-2020/[/URL] The difference is minimal. For example, if Norton and Total Defense would miss one malware more, then the results for 14 AVs would be very close to random trials for one hypothetical AV. It seems that a similar conclusion was made by AV-Comparatives because it awarded all 14 AVs. [LIST] [*][URL='https://www.av-comparatives.org/vendors/bitdefender/']Bitdefender[/URL] [*][URL='https://www.av-comparatives.org/vendors/eset/']ESET[/URL] [*][URL='https://www.av-comparatives.org/vendors/g-data/']G DATA[/URL] [*][URL='https://www.av-comparatives.org/vendors/kaspersky-lab/']Kaspersky[/URL] [*][URL='https://www.av-comparatives.org/vendors/total-defense/']Total Defense[/URL] [*][URL='https://www.av-comparatives.org/vendors/vipre/']VIPRE[/URL] [*][URL='https://www.av-comparatives.org/vendors/avast/']Avast*[/URL] [*][URL='https://www.av-comparatives.org/vendors/avg/']AVG*[/URL] [*][URL='https://www.av-comparatives.org/vendors/avira/']Avira*[/URL] [*][URL='https://www.av-comparatives.org/vendors/f-secure/']F-Secure*[/URL] [*][URL='https://www.av-comparatives.org/vendors/k7-2/']K7*[/URL] [*][URL='https://www.av-comparatives.org/vendors/nortonlifelock/']NortonLifeLock*[/URL] [*][URL='https://www.av-comparatives.org/vendors/panda/']Panda*[/URL] [*][URL='https://www.av-comparatives.org/vendors/total-av/']Total AV*[/URL] [/LIST] The 8 products (*) got lower awards due to false alarms. Edit. It seems that the same conclusion can be derived from cluster analysis made in the report: [ATTACH type="full" alt="1602800666137.png"]247395[/ATTACH] The AVs mentioned in my statistical model belong to the cluster one (see at the last column) and were avarded. Other AVs belong to other clusters. Here is what AV-Comparatives say about the importance of clusters: "[I]Our tests use much more test cases (samples) per product and month than any similar test performed by other testing labs. Because of the higher statistical significance this achieves, we consider all the products in each results cluster to be equally effective, assuming that they have a false-positives rate below the industry average[/I]." [URL='https://www.av-comparatives.org/real-world-protection-test-methodology/']Real-World Protection Test Methodology - AV-Comparatives (av-comparatives.org)[/URL] [/QUOTE]
Insert quotes…
Verification
Post reply
Top
