Forums
New posts
Search forums
News
Security News
Technology News
Giveaways
Giveaways, Promotions and Contests
Discounts & Deals
Reviews
Users Reviews
Video Reviews
Support
Windows Malware Removal Help & Support
Inactive Support Threads
Mac Malware Removal Help & Support
Mobile Malware Removal Help & Support
Blog
Log in
Register
What's new
Search
Search titles only
By:
Search titles only
By:
Reply to thread
Menu
Install the app
Install
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Forums
Security
Video Reviews - Security and Privacy
Homemade AV testing (a suggestion).
Message
<blockquote data-quote="Andy Ful" data-source="post: 1117442" data-attributes="member: 32260"><p><strong><span style="font-size: 18px">Homemade AV testing (a suggestion).</span></strong></p><p></p><p>Post updated / redesigned / shortened 14.02.2025</p><p></p><p><span style="color: rgb(0, 168, 133)"><strong>Simplified and standardized version can be found here:</strong></span></p><p>[URL unfurl="true"]https://malwaretips.com/threads/homemade-av-testing-a-suggestion.134815/post-1117709[/URL]</p><p></p><p>In my posts, I often criticized the homemade malware "tests" (YouTube tests), pointing out that the results obtained are not statistically significant and the testing methodology is incorrect. In this post, I am going to suggest a test outline that removes some important issues.</p><p></p><p></p><p><span style="font-size: 18px"><strong>Short description.</strong></span></p><p></p><p>This kind of test is intended for fresh malware samples. It is a kind of competition between a selected AV called <strong><span style="color: rgb(0, 168, 133)">AV4</span></strong> and a collective <span style="color: rgb(41, 105, 176)"><strong>AV123</strong></span> (a collection of three top AVs).</p><p>On each malware sample, the <strong><span style="color: rgb(0, 168, 133)">AV4</span></strong> can win, lose, and draw with a collective <span style="color: rgb(41, 105, 176)"><strong>AV123</strong></span>.</p><p><span style="color: rgb(0, 168, 133)"><strong>AV4</strong></span> loses whenever it is bypassed by the malware sample and <strong><span style="font-size: 18px">all three</span> </strong>top AVs protect against that sample (<span style="color: rgb(41, 105, 176)"><strong>Collective AV123 Pass</strong></span>).</p><p><span style="color: rgb(0, 168, 133)"><strong>AV4</strong></span> wins whenever it protects against the malware sample and <strong><span style="font-size: 18px">at least one</span> </strong>of the top three AVs is bypassed by that sample (<strong><span style="color: rgb(41, 105, 176)">Collective AV123 Failure</span></strong>).</p><p>In other cases, we have a draw between <span style="color: rgb(0, 168, 133)"><strong>AV4</strong></span> and <strong><span style="color: rgb(41, 105, 176)">AV123</span></strong>.</p><p></p><p><strong>Test details.</strong></p><ol> <li data-xf-list-type="ol">Take three top AVs (AV1, AV2, AV3 ---> collective AV123), and next the AV4 which is probably not a top AV (for example Norton + Kaspersky + Bitdefender as collective AV123, and additionally Microsoft Defender as AV4).</li> <li data-xf-list-type="ol">Take 25 fresh malware samples to perform a partial test.</li> <li data-xf-list-type="ol">Test those samples against AV123 and AV4.</li> <li data-xf-list-type="ol">Count the number of the AV4 wins and losses.</li> <li data-xf-list-type="ol"><span style="color: rgb(0, 168, 133)"><strong>End the test if the condition [ wins < losses ] is fulfilled</strong></span>. This will prove with high confidence that AV4 is not a top AV.</li> <li data-xf-list-type="ol"><strong>If not [ wins < losses ], </strong> perform another partial test with a new pule of 25 samples, as in points 2-5. <strong>Do not reset the numbers of wins and losses</strong>. Those numbers should reflect wins and losses of all partial tests.</li> <li data-xf-list-type="ol">If still the condition <strong>[ wins < losses ] does not apply,</strong> continue partial tests but <strong><span style="color: rgb(0, 168, 133)">end</span></strong><span style="color: rgb(0, 168, 133)"><strong> the full test when [ wins < losses ] or 16 tests are done</strong></span>.</li> <li data-xf-list-type="ol">If the condition <strong>[ wins < losses] is not fulfilled after 16 partial tests</strong>, the AV4 is most probably a top AV (or close to top AVs).</li> </ol><p></p><p>Each partial test should be done against AV123, and AV4 in two hours. The VM images must be saved for each of the four AVs.</p><p>To save time, the analysis of possible infections must be done after testing AV1, AV2, AV3, and AV4.</p><p>The partial tests with 25 samples can be done with a few days brake.</p><p></p><p>Example of partial test (25 samples, passed mean that the sample was blocked/detected):</p><p>1-10. All AVs passed <--- 10 draws</p><p>11. AV1 passed, AV2 passed, AV3 passed<strong> (Collective AV123 Pass) ; </strong><span style="color: rgb(184, 49, 47)"><strong>AV4 failed </strong></span><---- losses = 1</p><p>12-15. All AVs passed <--- 4 draws</p><p>16. AV1 passed, <span style="color: rgb(41, 105, 176)"><strong>AV2 failed</strong></span>, AV3 passed <strong>(Collective AV123 Failure) ; </strong> <span style="color: rgb(41, 105, 176)"><strong>AV4 failed</strong></span> <--- draw</p><p>17. All AVs passed <--- 1 draw</p><p>18. <span style="color: rgb(184, 49, 47)"><strong>AV1 </strong></span><strong><span style="color: rgb(184, 49, 47)">failed</span></strong>, <strong><span style="color: rgb(184, 49, 47)">AV2 failed</span></strong>, AV3 passed<strong> (Collective AV123 Failure)</strong> <strong> ; </strong> AV4 passed <---- wins = 1</p><p>19. AV1 passed, <span style="color: rgb(184, 49, 47)"><strong>AV2 failed</strong></span>, AV3 passed <strong>(Collective AV123 Failure) ;</strong> AV4 passed <---- wins = 1 + 1</p><p>20-24. All AVs passed <--- 1 draw</p><p>25. AV1 passed, AV2 passed, AV3 passed <strong>(Collective AV123 Pass) ;</strong> <span style="color: rgb(184, 49, 47)"><strong>AV4 failed</strong></span> <---- losses = 1 + 1</p><p></p><p>End of partial test. The result is inconclusive because it is not true that wins < losses. The test must be continued with another pule of 25 samples.</p><p></p><p>More examples:</p><p>[URL unfurl="false"]https://malwaretips.com/threads/homemade-av-testing-a-suggestion.134815/post-1117642[/URL]</p><p>[URL unfurl="false"]https://malwaretips.com/threads/homemade-av-testing-a-suggestion.134815/post-1117528[/URL]</p><p>[URL unfurl="false"]https://malwaretips.com/threads/homemade-av-testing-a-suggestion.134815/post-1117531[/URL]</p><p>[URL unfurl="false"]https://malwaretips.com/threads/homemade-av-testing-a-suggestion.134815/post-1117613[/URL]</p></blockquote><p></p>
[QUOTE="Andy Ful, post: 1117442, member: 32260"] [B][SIZE=5]Homemade AV testing (a suggestion).[/SIZE][/B] Post updated / redesigned / shortened 14.02.2025 [COLOR=rgb(0, 168, 133)][B]Simplified and standardized version can be found here:[/B][/COLOR] [URL unfurl="true"]https://malwaretips.com/threads/homemade-av-testing-a-suggestion.134815/post-1117709[/URL] In my posts, I often criticized the homemade malware "tests" (YouTube tests), pointing out that the results obtained are not statistically significant and the testing methodology is incorrect. In this post, I am going to suggest a test outline that removes some important issues. [SIZE=5][B]Short description.[/B][/SIZE] This kind of test is intended for fresh malware samples. It is a kind of competition between a selected AV called [B][COLOR=rgb(0, 168, 133)]AV4[/COLOR][/B] and a collective [COLOR=rgb(41, 105, 176)][B]AV123[/B][/COLOR] (a collection of three top AVs). On each malware sample, the [B][COLOR=rgb(0, 168, 133)]AV4[/COLOR][/B] can win, lose, and draw with a collective [COLOR=rgb(41, 105, 176)][B]AV123[/B][/COLOR]. [COLOR=rgb(0, 168, 133)][B]AV4[/B][/COLOR] loses whenever it is bypassed by the malware sample and [B][SIZE=5]all three[/SIZE] [/B]top AVs protect against that sample ([COLOR=rgb(41, 105, 176)][B]Collective AV123 Pass[/B][/COLOR]). [COLOR=rgb(0, 168, 133)][B]AV4[/B][/COLOR] wins whenever it protects against the malware sample and [B][SIZE=5]at least one[/SIZE] [/B]of the top three AVs is bypassed by that sample ([B][COLOR=rgb(41, 105, 176)]Collective AV123 Failure[/COLOR][/B]). In other cases, we have a draw between [COLOR=rgb(0, 168, 133)][B]AV4[/B][/COLOR] and [B][COLOR=rgb(41, 105, 176)]AV123[/COLOR][/B]. [B]Test details.[/B] [LIST=1] [*]Take three top AVs (AV1, AV2, AV3 ---> collective AV123), and next the AV4 which is probably not a top AV (for example Norton + Kaspersky + Bitdefender as collective AV123, and additionally Microsoft Defender as AV4). [*]Take 25 fresh malware samples to perform a partial test. [*]Test those samples against AV123 and AV4. [*]Count the number of the AV4 wins and losses. [*][COLOR=rgb(0, 168, 133)][B]End the test if the condition [ wins < losses ] is fulfilled[/B][/COLOR]. This will prove with high confidence that AV4 is not a top AV. [*][B]If not [ wins < losses ], [/B] perform another partial test with a new pule of 25 samples, as in points 2-5. [B]Do not reset the numbers of wins and losses[/B]. Those numbers should reflect wins and losses of all partial tests. [*]If still the condition [B][ wins < losses ] does not apply,[/B] continue partial tests but [B][COLOR=rgb(0, 168, 133)]end[/COLOR][/B][COLOR=rgb(0, 168, 133)][B] the full test when [ wins < losses ] or 16 tests are done[/B][/COLOR]. [*]If the condition [B][ wins < losses] is not fulfilled after 16 partial tests[/B], the AV4 is most probably a top AV (or close to top AVs). [/LIST] Each partial test should be done against AV123, and AV4 in two hours. The VM images must be saved for each of the four AVs. To save time, the analysis of possible infections must be done after testing AV1, AV2, AV3, and AV4. The partial tests with 25 samples can be done with a few days brake. Example of partial test (25 samples, passed mean that the sample was blocked/detected): 1-10. All AVs passed <--- 10 draws 11. AV1 passed, AV2 passed, AV3 passed[B] (Collective AV123 Pass) ; [/B][COLOR=rgb(184, 49, 47)][B]AV4 failed [/B][/COLOR]<---- losses = 1 12-15. All AVs passed <--- 4 draws 16. AV1 passed, [COLOR=rgb(41, 105, 176)][B]AV2 failed[/B][/COLOR], AV3 passed [B](Collective AV123 Failure) ; [/B] [COLOR=rgb(41, 105, 176)][B]AV4 failed[/B][/COLOR] <--- draw 17. All AVs passed <--- 1 draw 18. [COLOR=rgb(184, 49, 47)][B]AV1 [/B][/COLOR][B][COLOR=rgb(184, 49, 47)]failed[/COLOR][/B], [B][COLOR=rgb(184, 49, 47)]AV2 failed[/COLOR][/B], AV3 passed[B] (Collective AV123 Failure)[/B] [B] ; [/B] AV4 passed <---- wins = 1 19. AV1 passed, [COLOR=rgb(184, 49, 47)][B]AV2 failed[/B][/COLOR], AV3 passed [B](Collective AV123 Failure) ;[/B] AV4 passed <---- wins = 1 + 1 20-24. All AVs passed <--- 1 draw 25. AV1 passed, AV2 passed, AV3 passed [B](Collective AV123 Pass) ;[/B] [COLOR=rgb(184, 49, 47)][B]AV4 failed[/B][/COLOR] <---- losses = 1 + 1 End of partial test. The result is inconclusive because it is not true that wins < losses. The test must be continued with another pule of 25 samples. More examples: [URL unfurl="false"]https://malwaretips.com/threads/homemade-av-testing-a-suggestion.134815/post-1117642[/URL] [URL unfurl="false"]https://malwaretips.com/threads/homemade-av-testing-a-suggestion.134815/post-1117528[/URL] [URL unfurl="false"]https://malwaretips.com/threads/homemade-av-testing-a-suggestion.134815/post-1117531[/URL] [URL unfurl="false"]https://malwaretips.com/threads/homemade-av-testing-a-suggestion.134815/post-1117613[/URL] [/QUOTE]
Insert quotes…
Verification
Post reply
Top
