McMcbrad
Level 20
- Oct 16, 2020
- 967
Hi everyone,
We see many anti-malware tests on specialised on YouTube, as well as on specialised platforms. Whilst AV-Test.org, AV-comparatives.org, VirusBulletin, ICSALabs and MRG Effitas have a set standard, methodology, team of malware hunters and a regulator (AMTSO) to which some comply, tests on YouTube are a different story.
Whether you are looking to trust a test or conduct it yourself, this post will outline everything wrong I've seen in tests and provide recommendations on how to DIY properly.
Everything wrong with tests on YouTube
There are many users providing this sort of testing on YouTube. They usually get a large set of malware and malicious links, testing web-blocking, behavioural blocking and standard antivirus.
Should you trust those tests 100%? Not really.
Take a look at this Malwarebytes test uploaded by a user named "Vincent Tests".
The test has been uploaded on the 6th of November 2020 (6 days prior to this post). Take a closer look at the video and you'll notice that the modification date of these files is in 2017... 3 years ago. So this user is actually testing Malwarebytes against old malware. I can't help, but assume one of the following things - the user is either a Malwarebytes die-hard and tries to promote the solution, or is just not competent enough to discover actual prevalent malware.
He most probably downloaded this set from GitHub or similar, without even noticing when it was uploaded.
Now lets take a look at another test - this time by "The PC Security Channel". Leo is one of the more popular testers around and he even has his own testing script called Malex - it automates anti-malware testing.
We can see Leo attempting to test ESET against 0-day ransomware. He does not obtain proper 0-days, but rather disables real-time protection and relies on HIPS.
What Leo seems to either not know, or purposely ignore is that ESET is a product, highly-reliant on signatures, static and dynamic SCANNING. Product makes minimal usage of other technologies to detect malware and its most powerful features have been turned off.
This makes me doubt both his malware hunting abilities and validity of all other tests of his.
If I keep browsing tests on YouTube I can find many other examples like these.
Malware Hunting and testing tips
First and foremost, discover a reliable malware source. Unless you have one, do not jump into testing. If you type "malware sharing" or anything similar on google you might come up with several pages that host malicious files or links as well as phishing pages. For security reasons I would not share any of those.
Do not assume however, that employees behind anti-malware solutions can't check these sources regularly. Just because your solution has blocked many, or all of them doesn't mean it's effective.
Some sources actually require subscription, whilst others may require you to be a company or provide you with a demo mode.
Second, make sure you know what you are downloading. Do not assume that everything on your malware source is dangerous. Just because something has been flagged doesn't mean it's actually malicious at the moment. It might be a false positive, might be corrupt or might be relying on connections to servers already dead. Automated malware analysis tools are usually designed to be very sensitive. Hybrid Analyses is one tool that ticks many boxes - it's linked to VirusTotal and it integrate MITRE ATT&CK matrix.
Make sure you've analysed the behaviour and it is really malicious before actually putting the sample to use.
If your sample in question relies on vulnerabilities in its attack chain, make sure they haven't been patched. Users are always supposed to keep their software updated. Make sure that malware is relevant to your test - I've seen people testing solutions against Linux malware on Windows. Does this solution even scan for those?
Third, if you are looking to calculate percentage of effectiveness, make sure you've deduplicated your testing set. Your malware source might have many instances of one sample. Look carefully at the file hash, as well as indicators such as file behaviour, size and icons amongst others to make sure you are getting realistic results. You should be even more careful if you are looking to share these results online - you might actually mislead users and put them at risk.
Fourth - make sure you are testing a product with all components turned on, aggressiveness no lower than default and up-to-date. We don't really know how a security solution has been engineered - components might have been designed to share information with each other. By turning some of them off (like turning standard antivirus and relying on behavioural blocking), you might be decreasing the effectiveness of others. The security solution might not report that to you. To protect the software from being tested by attackers and bypassed, companies keep their malware detection process as secret as possible.
Fifth - have patience in the malware hunting process and never jump to conclusions on the basis of just one test. Getting quality malware samples is not easy - especially if you are looking for 0-days. You might be lucky to ever find one, without a subscription in common threat hunting portals. It's always best to test a solution again and again for a period of time before concluding how good it actually is.
Do not assume that a test from few years or even few months ago is still valid - threat landscape, as well as the technologies vendors use change every day.
I've seen even AV companies use tests from 2017 or 2011 to convince you they are the most effective ones.
This is from the Bitdefender website:
January 2011 - 2020 - August 2020 overall score. Great, but things change quickly.
Sixth - decide on the type of test. Are you trying to do a real-world protection test - in that case focus on already known malware executables & MS Office documents and links, or are you testing against sophisticated, evasive malware - in that case focus on scripts and other fileless-based attacks.
And last, but not least - always take tests already conducted by others with a grain of salt. We don't know what's actually in that set and where it has been obtained from.
The MalwareTips's testing hub is a very reliable place where you can see different solutions tested.
Apart from looking at raw malware detection, also consider other features - for example, the solution in question might not be the best against ransomware, but it might offer you Backup or file lock capabilities. It might not be the best against phishing, but it might offer you identity scanning. It might be a bit more effective than others, but with a huge performance impact, or - it might simply not meet your needs.
Always send all undetected samples to AV companies, communicate to them and try to discover whitepapers on how their technologies work so, you know what exactly you are testing.
I encourage other malware hunters, testers and knowledgeable users to engage in this discussion with other tips on how to conduct or find a trustworthy anti-malware tests.
We see many anti-malware tests on specialised on YouTube, as well as on specialised platforms. Whilst AV-Test.org, AV-comparatives.org, VirusBulletin, ICSALabs and MRG Effitas have a set standard, methodology, team of malware hunters and a regulator (AMTSO) to which some comply, tests on YouTube are a different story.
Whether you are looking to trust a test or conduct it yourself, this post will outline everything wrong I've seen in tests and provide recommendations on how to DIY properly.
Everything wrong with tests on YouTube
There are many users providing this sort of testing on YouTube. They usually get a large set of malware and malicious links, testing web-blocking, behavioural blocking and standard antivirus.
Should you trust those tests 100%? Not really.
Take a look at this Malwarebytes test uploaded by a user named "Vincent Tests".
He most probably downloaded this set from GitHub or similar, without even noticing when it was uploaded.
Now lets take a look at another test - this time by "The PC Security Channel". Leo is one of the more popular testers around and he even has his own testing script called Malex - it automates anti-malware testing.
What Leo seems to either not know, or purposely ignore is that ESET is a product, highly-reliant on signatures, static and dynamic SCANNING. Product makes minimal usage of other technologies to detect malware and its most powerful features have been turned off.
This makes me doubt both his malware hunting abilities and validity of all other tests of his.
If I keep browsing tests on YouTube I can find many other examples like these.
Malware Hunting and testing tips
First and foremost, discover a reliable malware source. Unless you have one, do not jump into testing. If you type "malware sharing" or anything similar on google you might come up with several pages that host malicious files or links as well as phishing pages. For security reasons I would not share any of those.
Do not assume however, that employees behind anti-malware solutions can't check these sources regularly. Just because your solution has blocked many, or all of them doesn't mean it's effective.
Some sources actually require subscription, whilst others may require you to be a company or provide you with a demo mode.
Second, make sure you know what you are downloading. Do not assume that everything on your malware source is dangerous. Just because something has been flagged doesn't mean it's actually malicious at the moment. It might be a false positive, might be corrupt or might be relying on connections to servers already dead. Automated malware analysis tools are usually designed to be very sensitive. Hybrid Analyses is one tool that ticks many boxes - it's linked to VirusTotal and it integrate MITRE ATT&CK matrix.
Make sure you've analysed the behaviour and it is really malicious before actually putting the sample to use.
If your sample in question relies on vulnerabilities in its attack chain, make sure they haven't been patched. Users are always supposed to keep their software updated. Make sure that malware is relevant to your test - I've seen people testing solutions against Linux malware on Windows. Does this solution even scan for those?
Third, if you are looking to calculate percentage of effectiveness, make sure you've deduplicated your testing set. Your malware source might have many instances of one sample. Look carefully at the file hash, as well as indicators such as file behaviour, size and icons amongst others to make sure you are getting realistic results. You should be even more careful if you are looking to share these results online - you might actually mislead users and put them at risk.
Fourth - make sure you are testing a product with all components turned on, aggressiveness no lower than default and up-to-date. We don't really know how a security solution has been engineered - components might have been designed to share information with each other. By turning some of them off (like turning standard antivirus and relying on behavioural blocking), you might be decreasing the effectiveness of others. The security solution might not report that to you. To protect the software from being tested by attackers and bypassed, companies keep their malware detection process as secret as possible.
Fifth - have patience in the malware hunting process and never jump to conclusions on the basis of just one test. Getting quality malware samples is not easy - especially if you are looking for 0-days. You might be lucky to ever find one, without a subscription in common threat hunting portals. It's always best to test a solution again and again for a period of time before concluding how good it actually is.
Do not assume that a test from few years or even few months ago is still valid - threat landscape, as well as the technologies vendors use change every day.
I've seen even AV companies use tests from 2017 or 2011 to convince you they are the most effective ones.
This is from the Bitdefender website:
January 2011 - 2020 - August 2020 overall score. Great, but things change quickly.
Sixth - decide on the type of test. Are you trying to do a real-world protection test - in that case focus on already known malware executables & MS Office documents and links, or are you testing against sophisticated, evasive malware - in that case focus on scripts and other fileless-based attacks.
And last, but not least - always take tests already conducted by others with a grain of salt. We don't know what's actually in that set and where it has been obtained from.
The MalwareTips's testing hub is a very reliable place where you can see different solutions tested.
Apart from looking at raw malware detection, also consider other features - for example, the solution in question might not be the best against ransomware, but it might offer you Backup or file lock capabilities. It might not be the best against phishing, but it might offer you identity scanning. It might be a bit more effective than others, but with a huge performance impact, or - it might simply not meet your needs.
Always send all undetected samples to AV companies, communicate to them and try to discover whitepapers on how their technologies work so, you know what exactly you are testing.
I encourage other malware hunters, testers and knowledgeable users to engage in this discussion with other tips on how to conduct or find a trustworthy anti-malware tests.
Last edited:
