Forums
New posts
Search forums
News
Security News
Technology News
Giveaways
Giveaways, Promotions and Contests
Discounts & Deals
Reviews
Users Reviews
Video Reviews
Support
Windows Malware Removal Help & Support
Inactive Support Threads
Mac Malware Removal Help & Support
Mobile Malware Removal Help & Support
Blog
Log in
Register
What's new
Search
Search titles only
By:
Search titles only
By:
Reply to thread
Menu
Install the app
Install
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Forums
Security
Security Statistics and Reports
Randomness in the AV Labs testing.
Message
<blockquote data-quote="Andy Ful" data-source="post: 909620" data-attributes="member: 32260"><p><strong>AV-Comparatives Malware Protection tests 2019-2020 (four tests) part two.</strong></p><p></p><p>This is a continuation of the post in another thread, where the impact of polymorphic samples was skipped:</p><p>[URL unfurl="true"]https://malwaretips.com/threads/consumer-malware-protection-test-september-2020.104609/post-909426[/URL]</p><p></p><p>In this post, I am going to examine cumulative results for the last 2 years (March 2019, September 2019, March 2020, September 2020), <strong><span style="color: rgb(184, 49, 47)">on the assumption that a strangely high number of missed samples was not caused by several different malware but by one polymorphic malware.</span></strong> Most AVs had such strange results. For example, Kaspersky had 13 missed samples in March 2019 and 9 missed samples in September 2019. What if there were in fact only two polymorphic malware, one in 13 variants and the second in 9 variants? Let's look at the results, where 9+ missed samples were replaced by one polymorphic sample:</p><p></p><p><strong>---------------<span style="color: rgb(41, 105, 176)">-Missed samples</span></strong>----<strong><span style="color: rgb(0, 168, 133)">Clusters</span></strong></p><p><span style="color: rgb(0, 168, 133)"><strong>Avast, AVG.........1+0+2+0...........1,1,1,1</strong></span></p><p><strong><span style="color: rgb(0, 168, 133)">F-Secure ............1+1+0+1...........1,1,1,1</span></strong></p><p><strong><span style="color: rgb(0, 168, 133)">McAfee .............1+1+1+0............1,1,1,1</span></strong></p><p><strong><span style="color: rgb(65, 168, 175)">Norton...............(2)+(2)+0+2.......1,1,1,1</span></strong></p><p><span style="color: rgb(65, 168, 175)"><strong>ESET...................1+1+1+2 ..........1,1,1,1</strong></span></p><p><strong><span style="color: rgb(65, 168, 175)">Kaspersky...........1+1+3+1..... .....1,1,1,1</span></strong></p><p><strong><span style="color: rgb(26, 188, 156)">Panda ................1+1+4+1...........1,1,1,1</span></strong></p><p><strong><span style="color: rgb(26, 188, 156)">Microsoft............2+4+1+0...........1,1,1,1</span></strong></p><p><strong><span style="color: rgb(26, 188, 156)">Bitdefender.........1+5+2+1...........1,1,1,1</span></strong></p><p><strong><span style="color: rgb(97, 189, 109)">K7.......................5+5+1+2...........1,1,1,1</span></strong></p><p><strong><span style="color: rgb(97, 189, 109)">Avira* ................0+4+3+4............1,1,1,2</span></strong></p><p><strong><span style="color: rgb(97, 189, 109)">VIPRE ................4+1+3+4 ...........1,1,1,2</span></strong></p><p><strong><span style="color: rgb(97, 189, 109)">Total Defense.....5+1+1+4............1,1,1,2</span></strong></p><p></p><p>As we can see, the differences between AVs almost vanished. So, in Malware Protection tests, even four different tests from two years are probably not sufficient to see important differences between popular AVs. The final scoring can highly depend on how many polymorphic samples and polymorphic variations were present in the tests. Without knowing it, the AV comparison on the base of such tests is not reliable at all.</p><p>The polymorphic samples could also explain the ridiculous results of four tests in the case of TrendMicro ( 0 missed samples in two tests from the year 2019 and 82+175 = 257 samples in the year 2020 ????).<img class="smilie smilie--emoji" loading="lazy" alt="😵" title="Dizzy face :dizzy_face:" src="https://cdn.jsdelivr.net/joypixels/assets/6.6/png/unicode/64/1f635.png" data-shortname=":dizzy_face:" /></p><p></p><p>The situation is clearer and easier to explain in the case of the Real-World tests, because from the results we know that the polymorphic samples are absent. <img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class="smilie smilie--sprite smilie--sprite109" alt=":)" title="Smile :)" loading="lazy" data-shortname=":)" /></p><p></p><p>[URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-march-2019/[/URL]</p><p>[URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-september-2019/[/URL]</p><p>[URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-march-2020/[/URL]</p><p>[URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-september-2020/[/URL]</p></blockquote><p></p>
[QUOTE="Andy Ful, post: 909620, member: 32260"] [B]AV-Comparatives Malware Protection tests 2019-2020 (four tests) part two.[/B] This is a continuation of the post in another thread, where the impact of polymorphic samples was skipped: [URL unfurl="true"]https://malwaretips.com/threads/consumer-malware-protection-test-september-2020.104609/post-909426[/URL] In this post, I am going to examine cumulative results for the last 2 years (March 2019, September 2019, March 2020, September 2020), [B][COLOR=rgb(184, 49, 47)]on the assumption that a strangely high number of missed samples was not caused by several different malware but by one polymorphic malware.[/COLOR][/B] Most AVs had such strange results. For example, Kaspersky had 13 missed samples in March 2019 and 9 missed samples in September 2019. What if there were in fact only two polymorphic malware, one in 13 variants and the second in 9 variants? Let's look at the results, where 9+ missed samples were replaced by one polymorphic sample: [B]---------------[COLOR=rgb(41, 105, 176)]-Missed samples[/COLOR][/B]----[B][COLOR=rgb(0, 168, 133)]Clusters[/COLOR][/B] [COLOR=rgb(0, 168, 133)][B]Avast, AVG.........1+0+2+0...........1,1,1,1[/B][/COLOR] [B][COLOR=rgb(0, 168, 133)]F-Secure ............1+1+0+1...........1,1,1,1 McAfee .............1+1+1+0............1,1,1,1[/COLOR] [COLOR=rgb(65, 168, 175)]Norton...............(2)+(2)+0+2.......1,1,1,1[/COLOR][/B] [COLOR=rgb(65, 168, 175)][B]ESET...................1+1+1+2 ..........1,1,1,1[/B][/COLOR] [B][COLOR=rgb(65, 168, 175)]Kaspersky...........1+1+3+1..... .....1,1,1,1[/COLOR] [COLOR=rgb(26, 188, 156)]Panda ................1+1+4+1...........1,1,1,1 Microsoft............2+4+1+0...........1,1,1,1 Bitdefender.........1+5+2+1...........1,1,1,1[/COLOR] [COLOR=rgb(97, 189, 109)]K7.......................5+5+1+2...........1,1,1,1 Avira* ................0+4+3+4............1,1,1,2 VIPRE ................4+1+3+4 ...........1,1,1,2 Total Defense.....5+1+1+4............1,1,1,2[/COLOR][/B] As we can see, the differences between AVs almost vanished. So, in Malware Protection tests, even four different tests from two years are probably not sufficient to see important differences between popular AVs. The final scoring can highly depend on how many polymorphic samples and polymorphic variations were present in the tests. Without knowing it, the AV comparison on the base of such tests is not reliable at all. The polymorphic samples could also explain the ridiculous results of four tests in the case of TrendMicro ( 0 missed samples in two tests from the year 2019 and 82+175 = 257 samples in the year 2020 ????).😵 The situation is clearer and easier to explain in the case of the Real-World tests, because from the results we know that the polymorphic samples are absent. :) [URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-march-2019/[/URL] [URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-september-2019/[/URL] [URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-march-2020/[/URL] [URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-september-2020/[/URL] [/QUOTE]
Insert quotes…
Verification
Post reply
Top
