Q&A How to perform a trustworthy test of anti-malware solution?

McMcbrad

Level 20
Oct 16, 2020
967
Hi everyone,

We see many anti-malware tests on specialised on YouTube, as well as on specialised platforms. Whilst AV-Test.org, AV-comparatives.org, VirusBulletin, ICSALabs and MRG Effitas have a set standard, methodology, team of malware hunters and a regulator (AMTSO) to which some comply, tests on YouTube are a different story.
Whether you are looking to trust a test or conduct it yourself, this post will outline everything wrong I've seen in tests and provide recommendations on how to DIY properly.

Everything wrong with tests on YouTube

There are many users providing this sort of testing on YouTube. They usually get a large set of malware and malicious links, testing web-blocking, behavioural blocking and standard antivirus.
Should you trust those tests 100%? Not really.
Take a look at this Malwarebytes test uploaded by a user named "Vincent Tests".
The test has been uploaded on the 6th of November 2020 (6 days prior to this post). Take a closer look at the video and you'll notice that the modification date of these files is in 2017... 3 years ago. So this user is actually testing Malwarebytes against old malware. I can't help, but assume one of the following things - the user is either a Malwarebytes die-hard and tries to promote the solution, or is just not competent enough to discover actual prevalent malware.
He most probably downloaded this set from GitHub or similar, without even noticing when it was uploaded.

Now lets take a look at another test - this time by "The PC Security Channel". Leo is one of the more popular testers around and he even has his own testing script called Malex - it automates anti-malware testing.
We can see Leo attempting to test ESET against 0-day ransomware. He does not obtain proper 0-days, but rather disables real-time protection and relies on HIPS.
What Leo seems to either not know, or purposely ignore is that ESET is a product, highly-reliant on signatures, static and dynamic SCANNING. Product makes minimal usage of other technologies to detect malware and its most powerful features have been turned off.
This makes me doubt both his malware hunting abilities and validity of all other tests of his.

If I keep browsing tests on YouTube I can find many other examples like these.

Malware Hunting and testing tips

First and foremost, discover a reliable malware source. Unless you have one, do not jump into testing. If you type "malware sharing" or anything similar on google you might come up with several pages that host malicious files or links as well as phishing pages. For security reasons I would not share any of those.
Do not assume however, that employees behind anti-malware solutions can't check these sources regularly. Just because your solution has blocked many, or all of them doesn't mean it's effective.
Some sources actually require subscription, whilst others may require you to be a company or provide you with a demo mode.

Second, make sure you know what you are downloading. Do not assume that everything on your malware source is dangerous. Just because something has been flagged doesn't mean it's actually malicious at the moment. It might be a false positive, might be corrupt or might be relying on connections to servers already dead. Automated malware analysis tools are usually designed to be very sensitive. Hybrid Analyses is one tool that ticks many boxes - it's linked to VirusTotal and it integrate MITRE ATT&CK matrix.
Make sure you've analysed the behaviour and it is really malicious before actually putting the sample to use.
If your sample in question relies on vulnerabilities in its attack chain, make sure they haven't been patched. Users are always supposed to keep their software updated. Make sure that malware is relevant to your test - I've seen people testing solutions against Linux malware on Windows. Does this solution even scan for those?

Third, if you are looking to calculate percentage of effectiveness, make sure you've deduplicated your testing set. Your malware source might have many instances of one sample. Look carefully at the file hash, as well as indicators such as file behaviour, size and icons amongst others to make sure you are getting realistic results. You should be even more careful if you are looking to share these results online - you might actually mislead users and put them at risk.

Fourth - make sure you are testing a product with all components turned on, aggressiveness no lower than default and up-to-date. We don't really know how a security solution has been engineered - components might have been designed to share information with each other. By turning some of them off (like turning standard antivirus and relying on behavioural blocking), you might be decreasing the effectiveness of others. The security solution might not report that to you. To protect the software from being tested by attackers and bypassed, companies keep their malware detection process as secret as possible.

Fifth - have patience in the malware hunting process and never jump to conclusions on the basis of just one test. Getting quality malware samples is not easy - especially if you are looking for 0-days. You might be lucky to ever find one, without a subscription in common threat hunting portals. It's always best to test a solution again and again for a period of time before concluding how good it actually is.
Do not assume that a test from few years or even few months ago is still valid - threat landscape, as well as the technologies vendors use change every day.
I've seen even AV companies use tests from 2017 or 2011 to convince you they are the most effective ones.
This is from the Bitdefender website:
1605210449525.png

January 2011 - 2020 - August 2020 overall score. Great, but things change quickly.

Sixth - decide on the type of test. Are you trying to do a real-world protection test - in that case focus on already known malware executables & MS Office documents and links, or are you testing against sophisticated, evasive malware - in that case focus on scripts and other fileless-based attacks.

And last, but not least - always take tests already conducted by others with a grain of salt. We don't know what's actually in that set and where it has been obtained from.
The MalwareTips's testing hub is a very reliable place where you can see different solutions tested.

Apart from looking at raw malware detection, also consider other features - for example, the solution in question might not be the best against ransomware, but it might offer you Backup or file lock capabilities. It might not be the best against phishing, but it might offer you identity scanning. It might be a bit more effective than others, but with a huge performance impact, or - it might simply not meet your needs.
Always send all undetected samples to AV companies, communicate to them and try to discover whitepapers on how their technologies work so, you know what exactly you are testing.

I encourage other malware hunters, testers and knowledgeable users to engage in this discussion with other tips on how to conduct or find a trustworthy anti-malware tests.
 
Last edited:

McMcbrad

Level 20
Oct 16, 2020
967
You nailed it, basically there are so many ways to do (or screw) a test that people should just use these tests as "entertainment purposes" only. Only an inexperienced user would base their protection, or advice, solely on the results of these tests alone.
Not only you can screw a test up or manipulate it, but the malware hunting process is time-consuming and I am sure other malware hunters on here can back me up. When I see them testing against a large set of malware, it gets clear to me that they just went on a common platform and downloaded everything they saw, without even checking what's there.
 

EndangeredPootis

Level 8
Verified
Sep 8, 2019
394
I dont trust test sites due to the fact youre basically taking their word for it, the fact they are so inconcistent like VirusBulletin and AV test having most of their products supposedly have a 100% detection ratio, they are also willing to test PUP's, infact, one scored 100% detection ratio for 0 zero day malware despite only having the Avira engine and nothing else, infact, one PUP had a 70% detection ratio one month, yet the very next month it had a 90% detection ratio, those differences are not normal
 

McMcbrad

Level 20
Oct 16, 2020
967
I dont trust test sites due to the fact youre basically taking their word for it, the fact they are so inconcistent like VirusBulletin and AV test having most of their products supposedly have a 100% detection ratio, they are also willing to test PUP's, infact, one scored 100% detection ratio for 0 zero day malware despite only having the Avira engine and nothing else, infact, one PUP had a 70% detection ratio one month, yet the very next month it had a 90% detection ratio, those differences are not normal
Are you talking about the Protected(.)net software?
 

upnorth

Moderator
Verified
Staff member
Malware Hunter
Jul 27, 2015
4,112
Personal I always grasp the key factor trust with/from security products/vendors/companies or developers, much better if a test comes included with an easy to understand disclaimer. Youtube security test videos and their creators ( not all ) can still be pretty hard to fully trust for many other reasons.

Just as mentioned by others, it exist many different views on how to perform tests of security products, both hardware and software. Mostly talked about here on MT is obvious software, but I highly recommend try watch a bit hardware information/reviews as well. Broaden the view so to speak. That will hopefully also help viewers be able to better interpret results. If they can't, dosen't matter how well any test of any product is performed.

Don't over complicate a test. Try to think a bit out of the box and simplify it instead. Here on MT for example there are many that have a pretty good understanding and knowledge about the language/lingo that is used, but there are many members and guests that don't. We should try to help also them.
the malware hunting process is time-consuming
Correct! Malware hunting is a process that eats time and energy. But it's also very rewarding with new lessons almost every time.
 

Cortex

Level 25
Verified
Aug 4, 2016
1,409
Leo does give convoluted explanations as to disabling parts of an AV as with ESET - It's often mentioned in the comments & he does answer but I'm not convinced, I find his reviews are interesting entertainment if not overly accurate - He was or still is an Emsisoft employee not sure in what capacity.
 

McMcbrad

Level 20
Oct 16, 2020
967
Leo does give convoluted explanations as to disabling parts of an AV as with ESET - It's often mentioned in the comments & he does answer but I'm not convinced, I find he reviews are interesting if not overly accurate - He was or still is an Emsisoft employee not sure in what capacity.
He is turning off the first layer of defence with the assumption that brand new malware will bypass it anyway and will come to the next layers. That's exactly his issue, that he assumes and doesn't know for sure. This is the reason why in the above post, at least 5 times I have said "do not assume". In the case of ESET this is totally wrong, as the product has been designed to be heavily relying on its scanning engine. This is not "bad" approach, because on some official tests, reliable or not, we can observe that products with effective "oldschool" scanning engines still make top-scores, whilst products like Cylance, Heimdal Thor and others that rely on the so-called next-gen approaches are "mediocre" at their best. Leo should first learn, then respect and take into account the way product has been engineered, before he proceeds to his assumptions.
 
Last edited:

Cortex

Level 25
Verified
Aug 4, 2016
1,409
He is turning off the first layer of defence with the assumption that brand new malware will bypass it anyway and will come to the next layers. That's exactly his issue, that he assumes and doesn't know for sure. This is the reason why in the above post, at least 5 times I have said "do not assume". In the case of ESET this is totally wrong, as the product has been designed to be heavily relying on its scanning engine. This is not wrong, because on some official tests, reliable or not, we can observe that products with effective "oldschool" scanning engines still make top-scores, whilst products like Cylance, Heimdal Thor and others that rely on the so-called next-gen approaches are "mediocre" at their best. Leo should first learn, then respect and take into account the way product has been engineered, before he proceeds to his assumptions.
I agree totally, I was a person who mentioned this over a year ago (Leo) but he decided not to change his methodology - I do find it interesting to view YouTube test often for a good look at GUI etc - One issue with these tests it's a one off, based on that day test the AV either gets great or avoid, long term is needed & not a single session that sadly many base their purchase on or feel they've been conned in buying what is actually a fully legitimate program - I've never done malware testing but used ESET & Kaspersky for example on my & others PC's to know they both pretty good or I would get phone calls :D:D
 

McMcbrad

Level 20
Oct 16, 2020
967
Oh Norton’s 0-day protection is a joke and I don’t know how these scores are being achieved. Are they confusing 0-day with 100-day? This product laid an extremely solid foundation a decade ago and just stopped moving forward. It now looks like a dead project.
 

Andy Ful

Level 67
Verified
Trusted
Content Creator
Dec 23, 2014
5,629
AV-Test 0-day testing is based on web threats. So, the result will be much better as compared to Malware Hub. But, the result = one missed malware per almost five years (about 7000 0-day URLs and malicious email attachments) is not especially convincing. So, even such a professional AV testing lab can have some issues.
 
Last edited:

McMcbrad

Level 20
Oct 16, 2020
967
AV-Test 0-day testing is based on web threats. So, the result will be much better as compared to Malware Hub. But, the result = one missed malware per almost five years (about 7000 0-day URLs and malicious email attachments) is not especially convincing. So, even such a professional AV testing lab can have some issues.
The problem with AV-test.org and AV-Comparatives.org is that they are major marketing points. How much you sell might depend on that, so I don't doubt that over time companies might either have received an insider information about how everything gets obtained and/or they may have used missed samples to train machine learning. We can go even further in our assumptions, but we don't really know for sure. All I can say is when the noise of the money sounds, truth can be muted.
It's also not impossible that both testing organisations have some "depth" when searching for links, based on probability that a user might fall on that specific place of the web.
Over the years many vendors have left and re-joined these tests and many have claimed that these tests are unrealistic.

However, Norton specially, whether it's on links or samples, hasn't been too good in the last few years. The product's market share and prices have gone down and this leads to a certain underdevelopment, as percentage of the revenue gets reinvested in R&D. As a business they went through many changes of CEO and management, and none of their directors did any good, apart from implementing an even more effective cost-cutting. We can see that it doesn't really keep current with technology nowadays.
 
Last edited:

Adrian Ścibor

From AVLab.pl
Verified
Apr 9, 2018
42
Hello @McMcbrad
Thank you for post. In my opinion the worst side YouTube testing is delivering malware samples to the machine.
It is bad when testers doing that by copied over Drag and Drop to the guest system. Does not matter one sample or hundreds and packed into one ZIP file.
I have seen many time how they doing that.
Some antivirus doesn't scan files over unknown protocols. Also, in the real world, the system is not infected over vmwaretools.exe or virtualbox.exe.
Moreover the copied files in this way do not have their source in file associations. The antivirus does not know if the file was downloaded from the Internet or copied from a USB flash drive. It's very important in my opinion.
The files should be delivered via a flash drive, browser (even from the local network), SFTP or other protocols, not over Drag and Drop.
 

Andy Ful

Level 67
Verified
Trusted
Content Creator
Dec 23, 2014
5,629
I think that the test should include some files packed in ZIP files (and some other types of archives), anyway. This is a common way of delivering malware via email attachments. Such delivery method avoids Windows SmartScreen Application Reputation in many cases (unpacked malware has not got MOTW) and the AV can have a problem to recognize if the file was delivered via flasdrive or Internet. Of course, using only malware packed in archives would not be real-world test.

Edit.
From some reports it follows that AV testing labs are interested in Machine Learning to improve the testing platform. The enormous number of new samples and diversity of attacks require changing the testing methodology.
 
Last edited:

Andy Ful

Level 67
Verified
Trusted
Content Creator
Dec 23, 2014
5,629
In my opinion the big problem of Youtube testing is that most of such tests cannot say anything interesting about the AV protection as compared to another AVs. Simply, the number of samples is too small and usually the AVs are tested on different samples. It is techically impossible for one person to gather several thousands of fresh samples a day, which probably would be necessary to see the real difference between popular AVs (there are over 300000 new malware a day) tested on different samples. There are also several delivery methods which are ignored in these tests.
The situation is better when the tests are performed for a long time by several (trained) people with several AVs on the same super-fresh samples (like on Malware Hub). But, even such testing has its own cons.
 

McMcbrad

Level 20
Oct 16, 2020
967
I have no idea why this software is even tested, considering it's only based on Avira and nothing else. Testing Avira will give a clear overview of the software's effectiveness. The way company operates makes their products a hard no for me and in an ocean of quality AVs, many of them for free, I don't see why anybody might decide to pay for that.
 
  • Like
Reactions: Protomartyr

EndangeredPootis

Level 8
Verified
Sep 8, 2019
394
I have no idea why this software is even tested, considering it's only based on Avira and nothing else. Testing Avira will give a clear overview of the software's effectiveness. The way company operates makes their products a hard no for me and in an ocean of quality AVs, many of them for free, I don't see why anybody might decide to pay for that.
Aslong as you got $$$ they will test anything, possibly if you pay even more they may screw with the results to make it look better, who knows afterall
 
Top