Forums
New posts
Search forums
News
Security News
Technology News
Giveaways
Giveaways, Promotions and Contests
Discounts & Deals
Reviews
Users Reviews
Video Reviews
Support
Windows Malware Removal Help & Support
Inactive Support Threads
Mac Malware Removal Help & Support
Mobile Malware Removal Help & Support
Blog
Log in
Register
What's new
Search
Search titles only
By:
Search titles only
By:
Reply to thread
Menu
Install the app
Install
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Forums
Security
Video Reviews - Security and Privacy
Homemade AV testing (a suggestion).
Message
<blockquote data-quote="Andy Ful" data-source="post: 1117709" data-attributes="member: 32260"><p><strong><span style="font-size: 18px">One against the triple.</span></strong></p><p></p><p>This is a simplified and standardized version of the testing procedure from the OP.</p><p></p><p>Many YouTube videos present the battle between two AVs (on the same pule of malware samples). The common issue is that the presented differences between the tested AVs are often statistically insignificant.</p><p>The situation can be improved when replacing one of the tested AVs with a triple AV123 of the top AVs (AV1, AV2, AV3). So we now have the battle between two parties: AV123 and AV4. In the examples mentioned in the previous posts, I proposed Bitdefender, Kaspersky, and Norton as AV1, AV2, and AV3. The tested AV4 can be for example Microsoft Defender.</p><p></p><p>To preserve the idea of the battle between two parties we must define when the AV123 fails on the concrete sample, and when the test result can be statistically significant:</p><ol> <li data-xf-list-type="ol"><strong><span style="color: rgb(0, 168, 133)">AV123 failure on the concrete sample happens when </span><span style="color: rgb(184, 49, 47)">at least one</span><span style="color: rgb(0, 168, 133)"> of AV1, AV2, or AV3 fails on that concrete sample.</span></strong></li> <li data-xf-list-type="ol"><strong><span style="color: rgb(0, 168, 133)">If </span><span style="color: rgb(41, 105, 176)">AV123 failures <span style="font-size: 18px"><</span> AV4 failures</span><span style="color: rgb(0, 168, 133)">, and <strong>the test is done on about </strong></span><span style="color: rgb(61, 142, 185)"><strong>400 fresh samples</strong></span><span style="color: rgb(0, 168, 133)"><strong> - </strong>the AV4 presents (statistically significant) lower protection.</span></strong></li> </ol><p><strong>Some thoughts about testing.</strong></p><ol> <li data-xf-list-type="ol">Testing few-day-old samples is pretty much useless.</li> <li data-xf-list-type="ol">The more 0-day samples in the pule of samples, the better the test reflects protection in the wild.</li> <li data-xf-list-type="ol">It is hard to find and test many 0-day samples in one day, so the test can be divided in time into several "partial tests" with a smaller number of samples.</li> <li data-xf-list-type="ol">AVs should be tested at approximately the same time (one partial test should be completed in 2 hours)</li> <li data-xf-list-type="ol">The testing procedure requires checking/confirming which concrete sample bypassed the protection of AV123 or AV4. This is usually possible when running each sample against a concrete AV (except those detected by manual scan) on the clean VM image.</li> <li data-xf-list-type="ol">If the concrete sample bypasses one of AV1, AV2, or AV3, then testing that sample against two other top AVs is not necessary (AV123 already failed).</li> <li data-xf-list-type="ol">The condition AV123 failures <span style="font-size: 18px"><</span> AV4 failures, gives similar results as statistical methods used by AV-Comparatives and AV-Test.</li> <li data-xf-list-type="ol">Uploading the sample to VirusTotal, cloud sandboxes, etc. is possible, but only after the sample has been tested. The uploaded samples are often shared with AV vendors.</li> </ol></blockquote><p></p>
[QUOTE="Andy Ful, post: 1117709, member: 32260"] [B][SIZE=5]One against the triple.[/SIZE][/B] This is a simplified and standardized version of the testing procedure from the OP. Many YouTube videos present the battle between two AVs (on the same pule of malware samples). The common issue is that the presented differences between the tested AVs are often statistically insignificant. The situation can be improved when replacing one of the tested AVs with a triple AV123 of the top AVs (AV1, AV2, AV3). So we now have the battle between two parties: AV123 and AV4. In the examples mentioned in the previous posts, I proposed Bitdefender, Kaspersky, and Norton as AV1, AV2, and AV3. The tested AV4 can be for example Microsoft Defender. To preserve the idea of the battle between two parties we must define when the AV123 fails on the concrete sample, and when the test result can be statistically significant: [LIST=1] [*][B][COLOR=rgb(0, 168, 133)]AV123 failure on the concrete sample happens when [/COLOR][COLOR=rgb(184, 49, 47)]at least one[/COLOR][COLOR=rgb(0, 168, 133)] of AV1, AV2, or AV3 fails on that concrete sample.[/COLOR][/B] [*][B][COLOR=rgb(0, 168, 133)]If [/COLOR][COLOR=rgb(41, 105, 176)]AV123 failures [SIZE=5]<[/SIZE] AV4 failures[/COLOR][COLOR=rgb(0, 168, 133)], and [B]the test is done on about [/B][/COLOR][COLOR=rgb(61, 142, 185)][B]400 fresh samples[/B][/COLOR][COLOR=rgb(0, 168, 133)][B] - [/B]the AV4 presents (statistically significant) lower protection.[/COLOR][/B] [/LIST] [B]Some thoughts about testing.[/B] [LIST=1] [*]Testing few-day-old samples is pretty much useless. [*]The more 0-day samples in the pule of samples, the better the test reflects protection in the wild. [*]It is hard to find and test many 0-day samples in one day, so the test can be divided in time into several "partial tests" with a smaller number of samples. [*]AVs should be tested at approximately the same time (one partial test should be completed in 2 hours) [*]The testing procedure requires checking/confirming which concrete sample bypassed the protection of AV123 or AV4. This is usually possible when running each sample against a concrete AV (except those detected by manual scan) on the clean VM image. [*]If the concrete sample bypasses one of AV1, AV2, or AV3, then testing that sample against two other top AVs is not necessary (AV123 already failed). [*]The condition AV123 failures [SIZE=5]<[/SIZE] AV4 failures, gives similar results as statistical methods used by AV-Comparatives and AV-Test. [*]Uploading the sample to VirusTotal, cloud sandboxes, etc. is possible, but only after the sample has been tested. The uploaded samples are often shared with AV vendors. [/LIST] [/QUOTE]
Insert quotes…
Verification
Post reply
Top
