Forums
New posts
Search forums
News
Security News
Technology News
Giveaways
Giveaways, Promotions and Contests
Discounts & Deals
Reviews
Users Reviews
Video Reviews
Support
Windows Malware Removal Help & Support
Mac Malware Removal Help & Support
Mobile Malware Removal Help & Support
Blog
Log in
Register
What's new
Search
Search titles only
By:
Search titles only
By:
Reply to thread
Menu
Install the app
Install
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Forums
Security
Security Statistics and Reports
Randomness in the AV Labs testing.
Message
<blockquote data-quote="Andy Ful" data-source="post: 905376" data-attributes="member: 32260"><p>Let's consider the example of the initial pule of 30000 sufficiently different malware variants in the wild and the particular AV which failed to detect 100 of them.</p><p>Next, we make for the above results a trial to choose 380 samples from these 30000 and calculate the probabilities for finding in these 380 samples 0, 1, 2, or 3 undetected malware.</p><p>m=30000</p><p>n=380</p><p>k=100</p><p></p><p>As it can be easily calculated the probability to find x=0, 1, 2, 3, ... undetected malware is as follows:</p><p>p(x) = B(m-k , n-x)* B(k , x) / B(m , n)</p><p>where B(p , q) is binomial coefficient.</p><p></p><p>After some simple calculations we have:</p><p>p(x) = (m-k)! * k! * (m-n)! * n! / [x! * (k-x)! * (n-x)! *(m-k-n+x)! * m!]</p><p></p><p>Here are the results of calculations for x= 0,1,2, and 3:</p><p>p(0)=0.28</p><p>p(1)=0.36</p><p>p(2)=0.23</p><p>p(3)=0.10</p><p></p><p>These probabilities show that one particular AV can have a different number of undetected malware (0, 1, 2, 3, ...) when we preselect a smaller pule of samples from the much larger set.</p><p></p><p>We can compare these probabilities with the results of the AV-Comparatives Real-world test (July-August 2020):</p><p>4 AVs with 0 undetected malware</p><p>5 AVs with 1 undetected malware</p><p>3 AVs with 2 undetected malware</p><p>1.5 AVs with 3 undetected malware (I added 0.5 AV for Norton)</p><p>[URL unfurl="true"]https://www.av-comparatives.org/tests/real-world-protection-test-jul-aug-2020-factsheet/[/URL]</p><p></p><p>We can calculate the ratios of the probabilities and numbers of Avs for the particular numbers of undetected malware:</p><p>p(0)/p(1) = 0.77 ~ 4 AVs/5 AVs</p><p>p(0)/p(2) = 1.22 ~ 4 AVs/3 AVs</p><p>p(1)/p(2) = 1.57 ~ 5 AVs/3 Avs</p><p>p(0)/p(3) = 2.8 ~ 4 AVs/1.5 AVs</p><p>p(1)/p(3) = 3.6 ~ 5 AVs/1.5 AVs</p><p>p(2)/p(3) = 2.3 ~ 3 AVs/1.5 AVs</p><p>etc.</p><p></p><p>As we can see the AV-Comparatives test results for AVs which have 0, 1, 2, or 3 undetected malware are very close to results of the random trials for one particular AV.</p><p></p><p>It means that F-Secure, G-Data, Panda, TrendMicro, Avast, AVG, BitDefender, Avira, Eset, K7, Microsoft, and Norton could have in fact the same real number of undetected malware (100 from 30000). But anyway, they would have different numbers of undetected samples in the July_August test by pure statistics.</p><p></p><p>Is reliable the assumption of 30000 sufficiently different malware variants in the wild for two months? Yes, it is. In the first half of 2019, SonicWall Real-Time Deep Memory Inspection (RTDMI) technology unveiled 74,360 ‘never-before-seen’ malware variants (about 25000 per 2 months).</p><p></p><p>Is reliable the assumption of 100 undetected malware from 30000? Yes, it is.</p><p>This gives on average about 1 undetected malware in 380 samples.</p><p></p><p>Conclusion.</p><p>One test with 380 malware samples is not especially reliable for a period of two months.</p><p>Even if the real malware detection is the same for any two AVs they can easily score as 0 undetected malware and 2 undetected malware.</p><p></p><p>Edit.</p><p>About the impact of the greater number of ‘never-before-seen’ malware variants on calculations:</p><p>[URL unfurl="true"]https://malwaretips.com/threads/randomness-in-the-av-labs-testing.104104/post-905910[/URL]</p></blockquote><p></p>
[QUOTE="Andy Ful, post: 905376, member: 32260"] Let's consider the example of the initial pule of 30000 sufficiently different malware variants in the wild and the particular AV which failed to detect 100 of them. Next, we make for the above results a trial to choose 380 samples from these 30000 and calculate the probabilities for finding in these 380 samples 0, 1, 2, or 3 undetected malware. m=30000 n=380 k=100 As it can be easily calculated the probability to find x=0, 1, 2, 3, ... undetected malware is as follows: p(x) = B(m-k , n-x)* B(k , x) / B(m , n) where B(p , q) is binomial coefficient. After some simple calculations we have: p(x) = (m-k)! * k! * (m-n)! * n! / [x! * (k-x)! * (n-x)! *(m-k-n+x)! * m!] Here are the results of calculations for x= 0,1,2, and 3: p(0)=0.28 p(1)=0.36 p(2)=0.23 p(3)=0.10 These probabilities show that one particular AV can have a different number of undetected malware (0, 1, 2, 3, ...) when we preselect a smaller pule of samples from the much larger set. We can compare these probabilities with the results of the AV-Comparatives Real-world test (July-August 2020): 4 AVs with 0 undetected malware 5 AVs with 1 undetected malware 3 AVs with 2 undetected malware 1.5 AVs with 3 undetected malware (I added 0.5 AV for Norton) [URL unfurl="true"]https://www.av-comparatives.org/tests/real-world-protection-test-jul-aug-2020-factsheet/[/URL] We can calculate the ratios of the probabilities and numbers of Avs for the particular numbers of undetected malware: p(0)/p(1) = 0.77 ~ 4 AVs/5 AVs p(0)/p(2) = 1.22 ~ 4 AVs/3 AVs p(1)/p(2) = 1.57 ~ 5 AVs/3 Avs p(0)/p(3) = 2.8 ~ 4 AVs/1.5 AVs p(1)/p(3) = 3.6 ~ 5 AVs/1.5 AVs p(2)/p(3) = 2.3 ~ 3 AVs/1.5 AVs etc. As we can see the AV-Comparatives test results for AVs which have 0, 1, 2, or 3 undetected malware are very close to results of the random trials for one particular AV. It means that F-Secure, G-Data, Panda, TrendMicro, Avast, AVG, BitDefender, Avira, Eset, K7, Microsoft, and Norton could have in fact the same real number of undetected malware (100 from 30000). But anyway, they would have different numbers of undetected samples in the July_August test by pure statistics. Is reliable the assumption of 30000 sufficiently different malware variants in the wild for two months? Yes, it is. In the first half of 2019, SonicWall Real-Time Deep Memory Inspection (RTDMI) technology unveiled 74,360 ‘never-before-seen’ malware variants (about 25000 per 2 months). Is reliable the assumption of 100 undetected malware from 30000? Yes, it is. This gives on average about 1 undetected malware in 380 samples. Conclusion. One test with 380 malware samples is not especially reliable for a period of two months. Even if the real malware detection is the same for any two AVs they can easily score as 0 undetected malware and 2 undetected malware. Edit. About the impact of the greater number of ‘never-before-seen’ malware variants on calculations: [URL unfurl="true"]https://malwaretips.com/threads/randomness-in-the-av-labs-testing.104104/post-905910[/URL] [/QUOTE]
Insert quotes…
Verification
Post reply
Top