AV-Comparatives Business Security Test August-September 2021 – Factsheet

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Gandalf_The_Grey

Level 76
Thread author
Verified
Honorary Member
Top Poster
Content Creator
Well-known
Apr 24, 2016
6,505
Introduction
This is a short fact sheet for our Business Main-Test Series, containing the results of the Business Malware Protection Test (September) and Business Real-World Protection Test (August-September). The full report, including the Performance Test and product reviews, will be released in December. To be certified in December 2021 as an “Approved Business Product” by AV-Comparatives, the tested products must score at least 90% in the Malware Protection Test with zero false alarms on common business software, and at least 90% in the overall Real-World Protection Test (i.e. over the course of four months), with less than one hundred false alarms on any clean software/websites (and with zero false alarms on common business software). Tested products must also avoid major performance issues (impact score must be below 40) and have fixed all reported bugs in order to gain certification.

Please note that the results of the Business Main-Test Series cannot be compared with the results of the Consumer Main-Test Series, as the tests are done at different times, with different sets, different settings, etc.

Starting from 2022, products will be required to have an FP rate on non-business files below the Remarkably High threshold.
Below we have listed relevant deviations from default settings (i.e. setting changes applied by the vendors):

Acronis:
“Backup”, “Vulnerability assessment”, “Patch management” and “Data protection map” disabled.

Bitdefender: “Fileless Attack Protection”, “Sandbox Analyzer” (for Applications and Documents) and “Scan SSL” enabled. “Encryption” and “Patch Management” add-ons registered and enabled. “HyperDetect” and “Device Sensor” disabled. “Update ring” changed to “Fast ring”. “Web Traffic Scan” enabled for HTTP Web traffic and Incoming POP3 emails.

Cisco: “On Execute File and Process Scan” set to Active; “Exploit Prevention: Script Control” and “TETRA Deep Scan File” enabled; “Event Tracing for Windows” enabled.

CrowdStrike: everything enabled and set to maximum, i.e. “Extra Aggressive”. “Sensor Visibility” for “Firmware” disabled. Uploading of “Unknown Detection-Related Executables” and “Unknown Executables” disabled.

Cybereason: “Anti-Malware” enabled; “Signatures mode” set to “Disinfect”; “Behavioral document protection” enabled; “Artificial intelligence” and “Anti-Exploit” set to “Aggressive”; “Exploit protection”, “PowerShell and .NET”, “Anti-Ransomware” and “App Control” enabled and set to “Prevent”; all “Collection features” enabled; “Scan archives on access” enabled.

Elastic: MalwareScore (“windows.advanced.malware.threshold”) set to “aggressive”.

ESET: All “Real-Time & Machine Learning Protection” settings set to “Aggressive”.

FireEye: “Real-Time Indicator Detection” disabled, “Exploit Guard” and “Malware Protection” enabled.

Fortinet: “Sandbox analysis” (FortiSandbox) and FortiEDR enabled. “Submit files from USB Sources” disabled; “Exclude Files from Trusted Sources” for “Sandbox Detection” enabled; in “Execution Prevention”, “Suspicious Script Execution” was disabled and “Unconfirmed File Detected” was enabled; eXtended Detection (XDR) was disabled.

G Data: “BEAST Behavior Monitoring” set to “Halt program and move to quarantine”. “G DATA WebProtection” add-on for Google Chrome installed and activated.

Malwarebytes: “Expert System Algorithms”, “Block penetration testing attacks”, “Disable IE VB Scripting”, “Java Malicious Inbound/outbound Shell Protection”, “Earlier RTP blocking”, “Enhanced sandbox protection” and “Thorough scan” enabled; “RET ROP Gadget detection” and “Malicious LoadLibrary Protection” enabled for all applications; “Protection for MessageBox Payload” enabled for MS Office; “Malwarebytes Browser Guard” Chrome extension enabled.

Microsoft: Google Chrome extension “Windows Defender Browser Protection” installed and enabled.

Sophos: “Threat Case creation” and “Web Control” disabled.

VIPRE: “DNS Traffic Filtering” and “Malicious URL Blocking for HTTPS Traffic” enabled. “Firewall” and “IDS” enabled and set to “Block With Notify”.

VMware: policy set to “Advanced”.

Avast, K7, Kaspersky, Panda: default settings.
Test Results
Real-World Protection Test (August-September)
Schermafbeelding 2021-10-15 201551.jpg
Malware Protection Test (September)
Schermafbeelding 2021-10-15 201709.jpg

Schermafbeelding 2021-10-15 201849.jpg
Schermafbeelding 2021-10-15 201910.jpg
 

Anthony Qian

Level 9
Verified
Well-known
Apr 17, 2021
448
For perspective, this means 18 malwares missed in 1016 samples for Business Malware Protection Test, Kaspersky for example with 99 % missed 10 samples; I dont think that there is any reason for concern.
Microsoft missed only 1 sample; Avast and Bitdefender missed only 2 samples! I think relative performance is more important in a test.
 

Nightwalker

Level 24
Verified
Honorary Member
Top Poster
Content Creator
Well-known
May 26, 2014
1,339
Microsoft missed only 1 sample; Avast and Bitdefender missed only 2 samples! I think relative performance is more important in a test.

It just means that in this test with those specific samples those antivirus solutions detected more, statistically it doesnt mean that ESET is a bad product incapable of doing its job, because 98,2 % is still quite good.

There is no reason to concern at all, except that if you care that the antivirus you are using is ranked first in lab tests.
 
  • Like
Reactions: Nevi and roger_m

printing

Level 1
Nov 14, 2020
41
How can i convince my boss that the Microsoft Defender is good enough over a paid vendor?
Is quoting the report good enough?
I doubt any companies would take the risk and be blame for not choosing a paid vendor if a pc is compromised.
 

Nightwalker

Level 24
Verified
Honorary Member
Top Poster
Content Creator
Well-known
May 26, 2014
1,339
How can i convince my boss that the Microsoft Defender is good enough over a paid vendor?
Is quoting the report good enough?
I doubt any companies would take the risk and be blame for not choosing a paid vendor if a pc is compromised.
I dont think that it is enough, you can use Gartner Magic Quadrant for a more insightful and complementary report.

 

Anthony Qian

Level 9
Verified
Well-known
Apr 17, 2021
448
It just means that in this test with those specific samples those antivirus solutions detected more, statistically it doesnt mean that ESET is a bad product incapable of doing its job, because 98,2 % is still quite good.

There is no reason to concern at all, except that if you care that the antivirus you are using is ranked first in lab tests.
Based on other products' performance, I don't think this test is hard. ESET, given its high price, is supposed to do better.
 

Anthony Qian

Level 9
Verified
Well-known
Apr 17, 2021
448
How can i convince my boss that the Microsoft Defender is good enough over a paid vendor?
Is quoting the report good enough?
I doubt any companies would take the risk and be blame for not choosing a paid vendor if a pc is compromised.
If you don't trust AV-C, then look at AV-Test result. Microsoft Defender is just a nice solution that keeps getting better. With such a large user base, I am confident that its performance will continue to improve.
 
  • Like
Reactions: printing

Anthony Qian

Level 9
Verified
Well-known
Apr 17, 2021
448
You should look at the results of multiple tests, rather than just one test.
Actually I did.
  1. Since the beginning of this year, ESET's home product has been declining in the AV-C Malware Protection Test. (ESET )
  2. Since the beginning of the year, ESET has performed poorly in AV-Test tests in terms of Protection.(Test antivirus software ESET)
  3. According to SE Labs Enterprise Endpoint Protection (2021 Q2), ESET came in last place in terms of protection. (Enterprise Endpoint Protection (2021 Q2) - SE Labs - Reports)
 
  • Like
Reactions: Nevi and roger_m

The_King

Level 12
Verified
Top Poster
Well-known
Aug 2, 2020
542
How can i convince my boss that the Microsoft Defender is good enough over a paid vendor?
Is quoting the report good enough?
I doubt any companies would take the risk and be blame for not choosing a paid vendor if a pc is compromised.
If you are running a small business without a network then you may get away with just an AV solution.

If the business is running on a network architecture then you should have an Endpoint AV in place
to manage security on users PC. All it would take is one user to open a compromised email attachment and the whole network
could be compromised with Ransomware or malware.
 

printing

Level 1
Nov 14, 2020
41
I dont think that it is enough, you can use Gartner Magic Quadrant for a more insightful and complementary report.

wow!! thanks !!!
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top