Forums
New posts
Search forums
News
Security News
Technology News
Giveaways
Giveaways, Promotions and Contests
Discounts & Deals
Reviews
Users Reviews
Video Reviews
Support
Windows Malware Removal Help & Support
Inactive Support Threads
Mac Malware Removal Help & Support
Mobile Malware Removal Help & Support
Blog
Log in
Register
What's new
Search
Search titles only
By:
Search titles only
By:
Reply to thread
Menu
Install the app
Install
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Forums
Security
Video Reviews - Security and Privacy
Homemade AV testing (a suggestion).
Message
<blockquote data-quote="Andy Ful" data-source="post: 1117613" data-attributes="member: 32260"><p>Let's check this testing method against some extreme cases.</p><p>In the OP we had the win/lose/draw conditions:</p><ol> <li data-xf-list-type="ol"><strong>AV4</strong> loses whenever it is bypassed by the malware sample and <strong>all three </strong>top AVs protect against that sample (<strong>Collective AV123 Pass</strong>).</li> <li data-xf-list-type="ol"><strong>AV4</strong> wins whenever it protects against the malware sample and <strong>at least one </strong>of the top three AVs is bypassed by that sample (<strong>Collective AV123 Failure</strong>).</li> <li data-xf-list-type="ol">In other cases, we have a draw between <strong>AV4</strong> and <strong>AV123</strong>.</li> </ol><p><strong>End the test if the condition [ wins < losses ] is fulfilled</strong>. This will prove with high confidence that AV4 is not a top AV.</p><p>If the condition <strong>[ wins < losses] is not fulfilled after 16 partial tests</strong>, the AV4 is most probably a top AV (or close to top AVs).</p><p></p><p>****************************</p><p>****************************</p><p><strong><span style="color: rgb(0, 168, 133)">Example 1</span></strong> (All AVs are identical top AVs).</p><p>AV1, AV2, AV3, AV4 = AV</p><p>If all AVs are identical, the condition from point 1 can never be fulfilled. So, there are no losses, and the ending condition <strong>[ wins < losses ] </strong>can never be fulfilled too. After doing 16 partial tests, we must conclude that AV4 is most probably the top AV.</p><p>****************************</p><p>****************************</p><p><strong><span style="color: rgb(0, 168, 133)">Example 2</span></strong> (AV4 = AV1)</p><p>As in Example 1, the condition from point 1 can never be fulfilled. After doing 16 partial tests, we must conclude that AV4 is most probably the top AV.</p><p>****************************</p><p>****************************</p><p><span style="color: rgb(0, 168, 133)"><strong>Example 3</strong></span> (AV4 fails on the samples that can bypass at least one of AV1, AV2, or AV3).</p><p>This example includes also Examples 1 and 2, so we additionally assume that all tested AVs have different protection. In such a case the set S4 of samples that bypassed AV4 is a sum of similar sets for AV1, AV2, and AV3. We have S4 = S1 + S2 + S3, and S4 is usually significantly larger than any of S1, S2, and S3.</p><p>From condition in point 2, it follows that AV4 cannot win at all. If there are no wins, the ending condition <strong>[ wins < losses ]</strong> can be easily fulfilled so AV4 is not a top AV.</p><p>***************************</p><p>***************************</p><p><strong><span style="color: rgb(0, 168, 133)">Example 4</span></strong> (AV4 fails on different samples than any of AV1, AV2, and AV3).</p><p>This means that the set S4 (samples that bypassed AV4) does not have common samples with any of S1, S2, and S3.</p><p>From condition 1 it follows that any sample from S(4) must generate a loss ----> losses = n(S4) = number of elements in S4</p><p>From condition 2 it follows that any sample from S1, S2, or S3 must generate a win.</p><p></p><p>The condition <strong>[ wins < losses ]</strong> looks now: n(S1 + S2 + S3) < n(S4)</p><p>If S4 is a top (or close) AV the above condition can hardly hold, because usually n(S4) ~ n(S) < n(S1 + S2 + S3) where S is any of S1, S2, and S3.</p><p>Furthermore, for non-top AVs, we usually have: n(S4) > n(S1 + S2 + S3), and then the ending condition is fulfilled.</p><p>**************************</p></blockquote><p></p>
[QUOTE="Andy Ful, post: 1117613, member: 32260"] Let's check this testing method against some extreme cases. In the OP we had the win/lose/draw conditions: [LIST=1] [*][B]AV4[/B] loses whenever it is bypassed by the malware sample and [B]all three [/B]top AVs protect against that sample ([B]Collective AV123 Pass[/B]). [*][B]AV4[/B] wins whenever it protects against the malware sample and [B]at least one [/B]of the top three AVs is bypassed by that sample ([B]Collective AV123 Failure[/B]). [*]In other cases, we have a draw between [B]AV4[/B] and [B]AV123[/B]. [/LIST] [B]End the test if the condition [ wins < losses ] is fulfilled[/B]. This will prove with high confidence that AV4 is not a top AV. If the condition [B][ wins < losses] is not fulfilled after 16 partial tests[/B], the AV4 is most probably a top AV (or close to top AVs). **************************** **************************** [B][COLOR=rgb(0, 168, 133)]Example 1[/COLOR][/B] (All AVs are identical top AVs). AV1, AV2, AV3, AV4 = AV If all AVs are identical, the condition from point 1 can never be fulfilled. So, there are no losses, and the ending condition [B][ wins < losses ] [/B]can never be fulfilled too. After doing 16 partial tests, we must conclude that AV4 is most probably the top AV. **************************** **************************** [B][COLOR=rgb(0, 168, 133)]Example 2[/COLOR][/B] (AV4 = AV1) As in Example 1, the condition from point 1 can never be fulfilled. After doing 16 partial tests, we must conclude that AV4 is most probably the top AV. **************************** **************************** [COLOR=rgb(0, 168, 133)][B]Example 3[/B][/COLOR] (AV4 fails on the samples that can bypass at least one of AV1, AV2, or AV3). This example includes also Examples 1 and 2, so we additionally assume that all tested AVs have different protection. In such a case the set S4 of samples that bypassed AV4 is a sum of similar sets for AV1, AV2, and AV3. We have S4 = S1 + S2 + S3, and S4 is usually significantly larger than any of S1, S2, and S3. From condition in point 2, it follows that AV4 cannot win at all. If there are no wins, the ending condition [B][ wins < losses ][/B] can be easily fulfilled so AV4 is not a top AV. *************************** *************************** [B][COLOR=rgb(0, 168, 133)]Example 4[/COLOR][/B] (AV4 fails on different samples than any of AV1, AV2, and AV3). This means that the set S4 (samples that bypassed AV4) does not have common samples with any of S1, S2, and S3. From condition 1 it follows that any sample from S(4) must generate a loss ----> losses = n(S4) = number of elements in S4 From condition 2 it follows that any sample from S1, S2, or S3 must generate a win. The condition [B][ wins < losses ][/B] looks now: n(S1 + S2 + S3) < n(S4) If S4 is a top (or close) AV the above condition can hardly hold, because usually n(S4) ~ n(S) < n(S1 + S2 + S3) where S is any of S1, S2, and S3. Furthermore, for non-top AVs, we usually have: n(S4) > n(S1 + S2 + S3), and then the ending condition is fulfilled. ************************** [/QUOTE]
Insert quotes…
Verification
Post reply
Top
