Reply to thread

Message: <blockquote data-quote="Andy Ful" data-source="post: 977821" data-attributes="member: 32260">How many unknown samples are included in AVLab tests?The question should be asked because most AVs usually miss 0 samples in AVLab tests.It seems that a similar methodology to AVLab tests can be seen in AV-Comparatives Malware Protection tests. The samples are executed from the disk and AVs are on default settings.Let's compare the results of Avast, Avira, Defender, and Malwarebytes in AV-Comparatives tests.AV-Comparatives Malware Protection 2021 few-weeks-old prevalent samples (about 20 000 total samples)Avast+Avira ............. 4+4 missed samplesDefender ....................18 missed samplesMalwarebytes ........... 22 missed samplesWe can also gather the results of Avast and Avira for a similar number of samples in AVLab tests:AVLab  (about 18 000 samples in 13 tests, January 2020 - January 2022)Avast+Avira ............... 0+1Unfortunately, only Avast and Avira are commonly tested both by AVLab and AV-Comparatives.The number of missed samples strongly suggests that in the AVLab tests the unknown samples are very rare for Avira and Avast (even rarer than for few-weeks-old prevalent samples). The results of Avast and Avira are typical in AVLab tests for many tested AVs.From the results of AV-Comparatives tests, it follows that the difference in protection between Defender and Malwarebytes is very small: about 4 samples per 20 000.If we decrease the number of unknown samples, then we can expect that this difference in protection should be similarly small. When we have 0 unknown samples then both AVs will miss 0 samples.Now, let's look at the current AVLab results (missed samples):Malwarebytes  ......... 1 per 1834 samples, or proportionally ~ 11 per 20 000 samplesDefender .............  397 per 1834 samples, or proportionally ~ 4338 per 20 000 samplesThe difference in protection: (4338-11) per 20 000 samplesThe number of missed samples by Defender is horribly high compared to Malwarebytes, which is in clear contradiction to the results of AV-Comparatives.Now the difference in protection is over 4000 samples and for AV-Comparatives it was only 4 samples.Hmmm!!!We can call this result truly anomalous!!! It is obvious that in the AVLab test many samples are unknown to Defender, even if they are known to Avast and Avira.Summary.In the AVLab tests, we have three groups of AVs (at least). <ul> <li data-xf-list-type="ul">Group 1 (most of the tested AVs) - almost all samples are known to these AVs,</li> <li data-xf-list-type="ul">Group 2 (Defender, Trend Micro) - <s>many</s> some "dead" samples are unknown to these AVs,</li> <li data-xf-list-type="ul">Group 3 (Comodo, Emsisoft,...) - these AVs use file reputation, HIPS, etc., so they do not care much about "dead" samples.</li> </ul>In such a situation is very possible, that the missed samples are unknown to Defender(or Trend Micro), because they never hit the customers and these vendors are simply slow with adding such ("dead") samples to the cloud signature database or behavior-based detections in the cloud.I added Trend Micro to Group 2 because it got a similar result as Defender when tested by AVLab in May 2020.[URL unfurl="false"]https://avlab.pl/en/results-may-2020/[/URL]One could suspect that AVLab tests are flawed, but kinda similar behavior is also visible for Trend Micro in the AV-Comparatives tests:Consumer TestsAV-Comparatives Malware Protection 2021........................... March ..... SepAvast ................... 1 ............ 3Avira  ................... 2 ............ 2Bitdefender ......... 0 ............ 2Eset .................... 10 .......... 18Kaspersky ........... 4 ........... 10Malwarebytes ..... 6 ........... 16McAfee ............... 0 ............. 0Microsoft ......... 15 ............. 3Norton ................ 0 ............. 0Panda ................. 2 ............. 7TrendMicro .... 103 .......... 121Vipre .................. 0 .............. 2Samples .... 10013 ...... 10029Over 200 missed samples per 20 000 total samples is an anomalous result, but still many times lower than for Defender in AVLab test (over 4000 missed samples per 20 000 total samples). Furthermore, Defender missed only about 20 samples per 20 000 total samples in AV-Comparatives tests.<img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class="smilie smilie--sprite smilie--sprite132" alt=":unsure:" title="Unsure    :unsure:" loading="lazy" data-shortname=":unsure:" />One of the MT members (@McMcbrad) asked the Trend Micro staff about this behavior. The answer was that those missed samples were not important for the security of TrendMicro customers. That is why I called such samples "dead".TrendMicro probably does not care to add quickly the signatures of relatively old "dead" samples, and Defender can add most of such signatures in one or two weeks. Avast, Avira, and probably many AVs can manage such samples in the cloud very quickly (via fast signatures or behavior-based detections). So even when the customers were infected in the wild, the Malware Protection test cannot reflect this in the test results.Why there can be many "dead" samples?From the Microsoft reports it follows that most malware can hit only once (over 90% infections). After one shot it becomes 'dead' and it is not used in the attacks anymore. The AVs are very quick to add fast signatures or behavior-based detections in the cloud. In most cases, the fast signature can be created in several minutes after the successful infection. Most often, the attackers use a new polymorphic version of malware which can give far more chances to infect another computer. If the AV is slow with managing such "dead" samples, many of them can be counted as misses in the Malware Protection tests.Why the "dead" samples are not so important in the Real-World tests?Such tests use 0-day or 1-day malware, so the AV vendors have no time to manage the "dead" samples.The problem of "dead" samples is most important for a few days malware and in the less degree for a few weeks malware. Such malware samples are used in Malware Protection tests.Can the "dead" samples be reused?I am not sure. If so then probably in cracks and pirated software. But, the chances of infections via "dead" samples are very low.Does it mean that Defender on default settings is OK in businesses?No. Defender in these settings is not good at fighting lateral movement. This can be partially seen in the AVLab. The "dead" but unknown samples can mimic the fresh unknown samples used in the highly targeted attacks. These malware samples can be executed at Level 3 as payloads without the MOTW, bypassing Defender's BAFS and SmartScreen. Other AVs can have more chances, because they have got some additional features like Network Protection, file reputation lookup, HIPS, etc.The AVLab seems to mimic the business scenario only for Defender when there are many "dead samples" and the BAFS feature is disabled. It is possible that a similar situation can be also for Comodo and Emsisoft (Group 3), but these AVs can block the unknown "dead" samples by the reputation file lookup.The perfect results for several other AVs can probably reflect only the fact that in one or two days, they can complete the detections in the cloud by borrowing "dead" samples from other AVs or hunting the payloads. So, in the test with few-days-old samples, almost all malware can be detected/blocked by the cloud backend (fast signatures + behavior-based detections).Post edited.Added Group 3 to underline the distinctive features based on file reputation, HIPS, etc.Edit 1.I examined other AVLab tests since the year 2020. Defender and Trend Micro missed several samples in some tests and Defender blocked all samples in some others. So, the "dead" samples can be probably a reason for missing several samples by Defender and Trend Micro in these tests, but not for a few hundred Defender misses in the current test.<img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class="smilie smilie--sprite smilie--sprite111" alt=":(" title="Frown    :(" loading="lazy" data-shortname=":(" />Edit 2.My reasoning about unknown samples was based on the assumption that in the AVLab tests the samples were executed without MOTW. This followed directly from the Defender results (BAFS did not work in the current test). Anyway from the testing methodology, it follows that files must have MOTW, so the BAFS issue was caused by another reason. In this case, we must skip the Avast from considerations because for EXE files with MOTW it uses CyberCapture (detonation in the sandbox). This feature makes a big difference when comparing AVLab with AV-Comparatives. Fortunately, this does not change the main reasoning - still the 4 samples missed in AV-Comparatives tests by Avira are slightly greater than 1 sample missed by this AV in AVLab tests. So, the number of unknown samples in AVLab tests is comparable (probably smaller) than in AV-Comparatives tests.See also:[URL unfurl="false"]https://malwaretips.com/threads/modern-protection-without-signatures-%E2%80%93-comparison-test-on-real-threats-advanced-in-the-wild-malware-test.112630/post-977993[/URL]</blockquote>

Verification

Top