Forums
New posts
Search forums
News
Security News
Technology News
Giveaways
Giveaways, Promotions and Contests
Discounts & Deals
Reviews
Users Reviews
Video Reviews
Support
Windows Malware Removal Help & Support
Inactive Support Threads
Mac Malware Removal Help & Support
Mobile Malware Removal Help & Support
Blog
Log in
Register
What's new
Search
Search titles only
By:
Search titles only
By:
Reply to thread
Menu
Install the app
Install
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Forums
Security
Security Statistics and Reports
Randomness in the AV Labs testing.
Message
<blockquote data-quote="Andy Ful" data-source="post: 905376" data-attributes="member: 32260"><p>Post updated in April 2025.</p><p></p><p>There are about 300.000 new malware threats every day (Windows OS).</p><p>Let's consider the example of the initial pool of 30.000 sufficiently different malware variants in the wild and the particular AV that failed to detect 100 of them.</p><p>Next, we make for the above results a trial to choose 380 samples from these 30.000 and calculate the probabilities for finding in these 380 samples 0, 1, 2, or 3 undetected malware.</p><p>m=30000</p><p>n=380</p><p>k=100</p><p></p><p>As it can be easily calculated, the probability of finding x=0, 1, 2, 3, ... undetected malware is as follows:</p><p><strong>p( x ) = B( m - k , n - x ) * B( k , x ) / B( m , n )</strong></p><p>where B( p , q ) is binomial coefficient.</p><p></p><p>[ATTACH=full]288143[/ATTACH]</p><p></p><p>After some simple calculations, we have:</p><p><strong>p(x) = ( m-k )! * k! * ( m - n )! * n! / [ x! * ( k - x )! * ( n - x )! * ( m - k - n + x )! * m! ]</strong></p><p></p><p>For sufficiently large numbers of samples in the wild ( <strong><span style="color: rgb(0, 168, 133)">m >> k , n</span></strong> ) and a small number of missed samples ( <strong><span style="color: rgb(0, 168, 133)">x << n</span></strong> ), the function <strong>p(x)</strong> depends on the infection rate (<strong> <span style="color: rgb(0, 168, 133)">r = k/m</span></strong> ) and the number of tested samples ( <strong><span style="color: rgb(0, 168, 133)">n</span></strong> ):</p><p></p><p><strong>p( x ) ~ B( n , x ) * r ^ x * (1 - r ) ^ ( n - x )</strong></p><p></p><p>[ATTACH=full]288197[/ATTACH]</p><p></p><p>So, increasing the number of in-the-wild samples does not change significantly the probabilities if the infection rate <strong><span style="color: rgb(0, 168, 133)">k/m</span></strong> does not change and <strong><span style="color: rgb(0, 168, 133)">m</span></strong> is big enough.</p><p>Here are the results of calculations for the number of tested samples <span style="color: rgb(0, 168, 133)"><strong>n=380</strong></span>, infection rate <strong><span style="color: rgb(0, 168, 133)">k/m=1/300</span></strong>, and <strong><span style="color: rgb(0, 168, 133)">x = 0,1,2, 3</span></strong>:</p><p>p(0)=0.28</p><p>p(1)=0.36</p><p>p(2)=0.23</p><p>p(3)=0.1</p><p></p><p>These probabilities show that one particular AV can have a different number of undetected malware (0, 1, 2, 3, ...) when we preselect a smaller pool of samples from the much larger set.</p><p></p><p>We can compare these probabilities with the results of the AV-Comparatives Real-world test (July-August 2020):</p><p>4 AVs with 0 undetected malware</p><p>5 AVs with 1 undetected malware</p><p>3 AVs with 2 undetected malware</p><p>1.5 AVs with 3 undetected malware (I added 0.5 AV for Norton)</p><p>[URL unfurl="true"]https://www.av-comparatives.org/tests/real-world-protection-test-jul-aug-2020-factsheet/[/URL]</p><p></p><p>We can calculate the ratios of the probabilities and numbers of AVs for the particular numbers of undetected malware:</p><p>p(0)/p(1) = 0.79 ~ 4 AVs/5 AVs</p><p>p(0)/p(2) = 1.2 ~ 4 AVs/3 AVs</p><p>p(1)/p(2) = 1.6 ~ 5 AVs/3 Avs</p><p>p(0)/p(3) = 2.9 ~ 4 AVs/1.5 AVs</p><p>p(1)/p(3) = 3.7 ~ 5 AVs/1.5 AVs</p><p>p(2)/p(3) = 2.4 ~ 3 AVs/1.5 AVs</p><p>etc.</p><p></p><p>As we can see, the AV-Comparatives test results for AVs with 0, 1, 2, or 3 undetected malware are very close to the results of the random trials for one particular AV.</p><p></p><p>It means that F-Secure, G-Data, Panda, TrendMicro, Avast, AVG, BitDefender, Avira, Eset, K7, Microsoft, and Norton could have the same number of undetected malware in the wild (July and August). But anyway, they would have different numbers of undetected samples in the July_August test by pure statistics.</p><p></p><p>Conclusion.</p><p>One test with 380 malware samples is not especially reliable for a period of two months.</p><p>Even if the in-the-wild malware detection is the same for any two AVs, they can easily score as 0 undetected malware or 2 undetected malware.</p><p></p><p>Edit 1.</p><p>Post shortened. Added the approximate formula for p(x) and used it to calculate probabilities (instead of the exact formula).</p><p></p><p>Edit 2.</p><p>We do not know how exactly the AV Labs choose the malware samples. But most probably, they chose the test samples from large feeds (over 300,000 suspicious and malicious threats per day) and then remove some morphed samples of the same malware. If so, the approximate formula for p(x) is very accurate.</p><p>The example of the malware feed:</p><p>[URL unfurl="true"]https://www.mrg-effitas.com/services/threat-feeds-malware/[/URL]</p><p></p><p>Edit 3.</p><p>Corrected the typo error in the formula for p(x).</p></blockquote><p></p>
[QUOTE="Andy Ful, post: 905376, member: 32260"] Post updated in April 2025. There are about 300.000 new malware threats every day (Windows OS). Let's consider the example of the initial pool of 30.000 sufficiently different malware variants in the wild and the particular AV that failed to detect 100 of them. Next, we make for the above results a trial to choose 380 samples from these 30.000 and calculate the probabilities for finding in these 380 samples 0, 1, 2, or 3 undetected malware. m=30000 n=380 k=100 As it can be easily calculated, the probability of finding x=0, 1, 2, 3, ... undetected malware is as follows: [B]p( x ) = B( m - k , n - x ) * B( k , x ) / B( m , n )[/B] where B( p , q ) is binomial coefficient. [ATTACH type="full" width="230px" alt="1744978145533.png"]288143[/ATTACH] After some simple calculations, we have: [B]p(x) = ( m-k )! * k! * ( m - n )! * n! / [ x! * ( k - x )! * ( n - x )! * ( m - k - n + x )! * m! ][/B] For sufficiently large numbers of samples in the wild ( [B][COLOR=rgb(0, 168, 133)]m >> k , n[/COLOR][/B] ) and a small number of missed samples ( [B][COLOR=rgb(0, 168, 133)]x << n[/COLOR][/B] ), the function [B]p(x)[/B] depends on the infection rate ([B] [COLOR=rgb(0, 168, 133)]r = k/m[/COLOR][/B] ) and the number of tested samples ( [B][COLOR=rgb(0, 168, 133)]n[/COLOR][/B] ): [B]p( x ) ~ B( n , x ) * r ^ x * (1 - r ) ^ ( n - x )[/B] [ATTACH type="full" width="258px" alt="1745315344863.png"]288197[/ATTACH] So, increasing the number of in-the-wild samples does not change significantly the probabilities if the infection rate [B][COLOR=rgb(0, 168, 133)]k/m[/COLOR][/B] does not change and [B][COLOR=rgb(0, 168, 133)]m[/COLOR][/B] is big enough. Here are the results of calculations for the number of tested samples [COLOR=rgb(0, 168, 133)][B]n=380[/B][/COLOR], infection rate [B][COLOR=rgb(0, 168, 133)]k/m=1/300[/COLOR][/B], and [B][COLOR=rgb(0, 168, 133)]x = 0,1,2, 3[/COLOR][/B]: p(0)=0.28 p(1)=0.36 p(2)=0.23 p(3)=0.1 These probabilities show that one particular AV can have a different number of undetected malware (0, 1, 2, 3, ...) when we preselect a smaller pool of samples from the much larger set. We can compare these probabilities with the results of the AV-Comparatives Real-world test (July-August 2020): 4 AVs with 0 undetected malware 5 AVs with 1 undetected malware 3 AVs with 2 undetected malware 1.5 AVs with 3 undetected malware (I added 0.5 AV for Norton) [URL unfurl="true"]https://www.av-comparatives.org/tests/real-world-protection-test-jul-aug-2020-factsheet/[/URL] We can calculate the ratios of the probabilities and numbers of AVs for the particular numbers of undetected malware: p(0)/p(1) = 0.79 ~ 4 AVs/5 AVs p(0)/p(2) = 1.2 ~ 4 AVs/3 AVs p(1)/p(2) = 1.6 ~ 5 AVs/3 Avs p(0)/p(3) = 2.9 ~ 4 AVs/1.5 AVs p(1)/p(3) = 3.7 ~ 5 AVs/1.5 AVs p(2)/p(3) = 2.4 ~ 3 AVs/1.5 AVs etc. As we can see, the AV-Comparatives test results for AVs with 0, 1, 2, or 3 undetected malware are very close to the results of the random trials for one particular AV. It means that F-Secure, G-Data, Panda, TrendMicro, Avast, AVG, BitDefender, Avira, Eset, K7, Microsoft, and Norton could have the same number of undetected malware in the wild (July and August). But anyway, they would have different numbers of undetected samples in the July_August test by pure statistics. Conclusion. One test with 380 malware samples is not especially reliable for a period of two months. Even if the in-the-wild malware detection is the same for any two AVs, they can easily score as 0 undetected malware or 2 undetected malware. Edit 1. Post shortened. Added the approximate formula for p(x) and used it to calculate probabilities (instead of the exact formula). Edit 2. We do not know how exactly the AV Labs choose the malware samples. But most probably, they chose the test samples from large feeds (over 300,000 suspicious and malicious threats per day) and then remove some morphed samples of the same malware. If so, the approximate formula for p(x) is very accurate. The example of the malware feed: [URL unfurl="true"]https://www.mrg-effitas.com/services/threat-feeds-malware/[/URL] Edit 3. Corrected the typo error in the formula for p(x). [/QUOTE]
Insert quotes…
Verification
Post reply
Top
