AV-Comparatives Malware Protection tests 2019-2020 (four tests) part two.
This is a continuation of the post in another thread, where the impact of polymorphic samples was skipped:
[URL unfurl="true"]https://malwaretips.com/threads/consumer-malware-protection-test-september-2020.104609/post-909426[/URL]
In this post, I am going to examine cumulative results for the last 2 years (March 2019, September 2019, March 2020, September 2020), on the assumption that a strangely high number of missed samples was not caused by several different malware but by one polymorphic malware. Most AVs had such strange results. For example, Kaspersky had 13 missed samples in March 2019 and 9 missed samples in September 2019. What if there were in fact only two polymorphic malware, one in 13 variants and the second in 9 variants? Let's look at the results, where 9+ missed samples were replaced by one polymorphic sample:
----------------Missed samples----Clusters
Avast, AVG.........1+0+2+0...........1,1,1,1
F-Secure ............1+1+0+1...........1,1,1,1
McAfee .............1+1+1+0............1,1,1,1
Norton...............(2)+(2)+0+2.......1,1,1,1
ESET...................1+1+1+2 ..........1,1,1,1
Kaspersky...........1+1+3+1..... .....1,1,1,1
Panda ................1+1+4+1...........1,1,1,1
Microsoft............2+4+1+0...........1,1,1,1
Bitdefender.........1+5+2+1...........1,1,1,1
K7.......................5+5+1+2...........1,1,1,1
Avira* ................0+4+3+4............1,1,1,2
VIPRE ................4+1+3+4 ...........1,1,1,2
Total Defense.....5+1+1+4............1,1,1,2
As we can see, the differences between AVs almost vanished. So, in Malware Protection tests, even four different tests from two years are probably not sufficient to see important differences between popular AVs. The final scoring can highly depend on how many polymorphic samples and polymorphic variations were present in the tests. Without knowing it, the AV comparison on the base of such tests is not reliable at all.
The polymorphic samples could also explain the ridiculous results of four tests in the case of TrendMicro ( 0 missed samples in two tests from the year 2019 and 82+175 = 257 samples in the year 2020 ????).
The situation is clearer and easier to explain in the case of the Real-World tests, because from the results we know that the polymorphic samples are absent. 
[URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-march-2019/[/URL]
[URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-september-2019/[/URL]
[URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-march-2020/[/URL]
[URL unfurl="true"]https://www.av-comparatives.org/tests/malware-protection-test-september-2020/[/URL]
