AV-Comparatives Business Security Test March-April 2021 – Factsheet

[EndangeredPootis]

As an example of the problem, we can take the perfect results of the test from the PC Security Channel:



The tested product was Defender on MAX settings (all ASR rules enabled). The test was performed on 1764 EXE samples and all samples were blocked. Many people think that such tests can reflect real AV protection. But in fact, the differences between popular AVs in the detection of EXE files are very small and this test skipped the most important security layers (web protection, email protection) and totally skipped scripting malware. So, after watching this video one still does not have an idea about real Defender protection in the wild.

That is why I do not take into account the YouTube tests when discussing the AV testing. Such tests are only a demonstration of the AV capabilities.

And neither does AV testing sites, as they are the same, just throw thousands of sample at an product and call it an "test", so why dont we stop believing in their tests? they are no different afterall.

 

[Andy Ful]

And neither does AV testing sites, as they are the same, just throw thousands of sample at an product and call it an "test", so why dont we stop believing in their tests? they are no different afterall.

Yes - that is usually true for any single test (also made by the AV testing Lab). But there are some important differences as compared to AV testing Labs:

1. YouTube testers do not repeat tests periodically for several years.
2. YouTube videos do not test several AVs on the same pule of samples at the same time.
3. YouTube tests are not made by professionals. Many YouTubers do not even understand the results of tests made by themselves.
4. The testing methodology of YouTube tests is usually unknown and the number of samples is too small, so one cannot make any reliable conclusions.

So, any conclusion based on YouTube tests is questionable. On the contrary, the analysis of the tests made by AV-Comparatives, AV-Test, SE Labs, etc. is possible and some limited conclusions based on the statistics can be made.

It does not mean that YouTube tests are useless. There are many useful YouTube tests if one treats them as a demonstration of capabilities or demonstration of some features (or lack of some features).

 

[Andy Ful]

The tests from the PC Security Channel are exceptional among most YouTube tests. The author knows a lot about malware and many tests can be treated as a convincing demonstration of some AV features. Anyway, it is impossible to compare the protection in the wild of different AVs on the basis of these tests.

 

[Andy Ful]

My attitude for MH tests is kinda special. They include some valuable information about AV security that cannot be derived from other tests. The MH testers often try non-default settings combined with additional security layers. These tests include much more diversity compared to other tests. Furthermore, the testers usually understand well the protection features of tested AVs. Although the test results are not easy to interpret, I believe that the experienced testers can evaluate the AV protection without doing any statistics.