AVLab.pl May 2021 - Advanced In The Wild Malware Test

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Adrian Ścibor

From AVLab.pl
Verified
Apr 9, 2018
55
Hello MalwareTips Members!

The May edition of our “Advanced In the Wild Malware Test” reveals a different approach to applying security to Windows 10 by developers that design protection software. In our study that is complaint with MITRE tactics and techniques, we have analyzed 11 solutions that protect endpoints. The test lasting uninterruptedly for the whole month, 24 hours a day, was possible to carry out thanks to a programmed system that performs tedious calculations and actions in the Windows system, automating the entire test procedure (aggregating and analyzing logs, giving a final verdict). The design and operation of this system are described in this article and in the methodology.

Useful links:
 

Attachments

  • avlab-may-2021.png
    avlab-may-2021.png
    111.7 KB · Views: 114

Lenny_Fox

Level 22
Verified
Oct 1, 2019
1,125
Question to forum members:

When bitdefender traffic light provides the same protection as level 1of Bitdefender, based on these test a free combo of Bitdefender Trafficlight (level 1), Smartscreen (Defender level2) and Avast free (level 3) would be a nice combo? Hard_Configurator even has an Avast profile, so what is your opinion about this?

I am asking, because currently I run Bitdefender trafficlight, Smartscreen and Kaspersky free with Simple Windows Hardening on my girlfriend's laptop (without issues). I have disabled Kaspersky HTTPS scanning.

So what would you advice based on these tests: Bitdefender Traffic Light - SmartScreen - Avast free (without webprotection) or Bitdefender Traffic Light - Smartscreen - Kaspersky free (without HTTPS scannning)? Both with either H_C or SWH in the mix.

Thanks Lenny
 

Gandalf_The_Grey

Level 50
Verified
Trusted
Content Creator
Apr 24, 2016
3,903
The best combination as seen from these tests would be Bitdefender TrafficLight (hopefully the same as Bitdefender's L1) combined with Microsoft Defender (L1 and L2).

It seems that the L2 protection of Microsoft Defender is better than the others who let get things through at L1).
LEVEL 1:
The browser level, i.e. a virus has been stopped before or right after it has been downloaded.
LEVEL 2:
The system level, i.e. a virus has been downloaded, but it hasn’t been allowed to run.
LEVEL 3:
The analysis level, i.e. a virus has been run and blocked by a tested product.
FAIL:
The failure, i.e. a virus hasn’t been blocked and it has infected a system.
My advice would be:
Bitdefender TrafficLight - SmartScreen - Microsoft Defender (ConfigureDefender High settings) - Simple Windows Hardening (Default settings).

EDIT: If you want to choose between Avast and Kaspersky, based on this test, I would go for Kaspersky because of its higher L1 score and some L2 protection.
 
Last edited:
F

ForgottenSeer 85179

The best combination as seen from these tests would be Bitdefender TrafficLight (hopefully the same as Bitdefender's L1) combined with Microsoft Defender (L1 and L2).

It seems that the L2 protection of Microsoft Defender is better than the others who let get things through at L1).

My advice would be:
Bitdefender TrafficLight - SmartScreen - Microsoft Defender (ConfigureDefender High settings) - Simple Windows Hardening (Default settings).
Why not use NextDNS as L1?
It use Google Safebrowsing and many other security protections which should be more then enough for normal user.
In combination with Edge SmartScreen this should block most malware.
 

roger_m

Level 34
Verified
Content Creator
Dec 4, 2014
2,320
Even Microsoft Defender gets 100%. That means paying for third party AV for security is a waste of money.
As I've discussed with you before, it's not. Many people don't like MD, or simply prefer alternatives. MD is very good, so are big name third party antiviruses. Just use whatever works best for you. In my case, that means I use third party antiviruses.
 

roger_m

Level 34
Verified
Content Creator
Dec 4, 2014
2,320
It's not a waste of money for you because you have some specific need of a feature for a peculiar use case not related to security.

Yet for the vast majority of people. paying for 3rd party security software is a waste of money.
There are many people who find MD to be too heavy for their liking. For those users it makes sense to not use it, and it could be argued that it would be silly for them to use it, if it's making their computers run slower. For those that are happy with its performance, while there may be no need to use an alternative, there is nothing wrong with doing so. In any case, it makes sense for people to use whatever security software they are happy with. If they are happy using third party software, and it provides comparable or better protection than WD, then it's certainly not a waste.
 

SeriousHoax

Level 37
Verified
Mar 16, 2019
2,658
Bitdefender product vs TrafficLight L1 will not be the same always. BD has SSL scanning and can detect and block scripts and hosts connected as third party which TrafficLight can't. Bitdefender's web protection is 100% system-wide so it's not bound to browsers only. Here, Bitdefender L1 > TrafficLight L1.

In some cases, for example; if "A(.)com/abc.exe" is downloaded by the browser where BD has no data about the host "A(.)com" but has signature/cloud data for the file "abc.exe" then TrafficLight will miss that file while Bitdefender will block it before it has finished downloading. So in scenarios like this, Bitdefender L1 > TrafficLight L1.

TrafficLight has one slight advantage and that is, it loads data from the cloud so it's always up to date with known malicious and phishing sites while Bitdefender relies on hourly signatures so there could be some delays. In such cases, TrafficLight will block more malicious sites than Bitdefender. This case has been tested and proven by some members here on the forum including me.
 

Adrian Ścibor

From AVLab.pl
Verified
Apr 9, 2018
55
Even Microsoft Defender gets 100%. That means paying for third party AV for security is a waste of money.

We are working on a huge comparative the highest packages knows better as Total Protection, Premium, Complete or similar like that. Moreover we will consider Microsoft Defender + Windows 10 features with EDGE (like Windows Sandbox, Password Manager in EDGE, Smart Screen etc.). Please, be patient :) Report should be available online till next month.

I think the protection is not the only one feature product when you have to consider security product buying.
 

Local Host

Level 24
Verified
Sep 26, 2017
1,322
Unfortunately for most people, anything other than Microsoft Defender causes them grief. When they start chasing other features, then they experience problems. It is the reason that a significant portion of MalwareTips members change AV more frequently than they change their underwear. Except for specific use cases to solve specific problems (like for file hoarders that complain MD slows down their system), there is little advantage of these third party AV over native Windows security and Microsoft Defender.

For protection, I would choose Polish made SpyShelter Firewall. It outperforms everything else. It is so light on system. It is so fast. What it does protects a system like no other. It passes where all other AV fail. 😁
This is a clear case of user problem, not Software problem, considering you claim there's issues in using third-party AVs.

Not only that but you live in the illusion that only Microsoft is supplying free offers, Kaspersky Free is more than enough for the average user and is superior to MD by a mile.
There are many people who find MD to be too heavy for their liking. For those users it makes sense to not use it, and it could be argued that it would be silly for them to use it, if it's making their computers run slower. For those that are happy with its performance, while there may be no need to use an alternative, there is nothing wrong with doing so. In any case, it makes sense for people to use whatever security software they are happy with. If they are happy using third party software, and it provides comparable or better protection than WD, then it's certainly not a waste.

True, using MD increases build time in VS by 200% (which is not optimal for a work station), then we have lack of configurability in the UI (forcing users to rely on third-parties), last but not least the bugs and false positives, MD is the only AV that bothers me about my private Software (since it's unknown to them, their buggy whitelist doesn't help either), while Kaspersky doesn't bat an eye.
 

Digmor Crusher

Level 11
Verified
Jan 27, 2018
525
The best combination as seen from these tests would be Bitdefender TrafficLight (hopefully the same as Bitdefender's L1) combined with Microsoft Defender (L1 and L2).

It seems that the L2 protection of Microsoft Defender is better than the others who let get things through at L1).

My advice would be:
Bitdefender TrafficLight - SmartScreen - Microsoft Defender (ConfigureDefender High settings) - Simple Windows Hardening (Default settings).

EDIT: If you want to choose between Avast and Kaspersky, based on this test, I would go for Kaspersky because of its higher L1 score and some L2 protection.
Trafficlight in Edge, Chrome or both?
 

MacDefender

Level 14
Verified
Oct 13, 2019
685
I think one thing that people are missing is that competition is good for the whole industry. I am not a professional malware writer, but I've found it takes me less than an hour to write a proof of concept that bypasses Microsoft Defender or another AV. But where it gets challenging is trying to write something that defeats all of them at once. As soon as you start doing that, most AVs' heuristics start picking up on both the code and runtime behavior that you're using to do that.

A world with just Microsoft Defender IMO would not be as secure as a world where attackers have to worry about defeating at least the top 3-5 AVs to stand a chance at spreading.

MD is good, but just because one product is good doesn't mean that everything else is superfluous.
 

Andy Ful

Level 72
Verified
Trusted
Content Creator
Dec 23, 2014
6,117
There is no need to dispute the Defender protection (and other AVs too).
According to the Consumer tests, the group of top AVs includes Norton, F-Secure, Kaspersky, and Trend-Micro (other paid AVs did not participate in all tests), but we should probably add Bitdefender to this group.
Norton uses very aggressive default settings (many false positives) based on the reputation in the cloud so it has got consistently best scorings.
Close after the top AVs are free AVs (Avast, Avira, Microsoft Defender) and several good paid AVs (including McAfee).

The cumulative scorings in AV-Test, AV-Comparatives, and SE Labs from the year 2019 until now (two and half year period):

Real-World 2019-2021: AV-Test, AV-Comparatives, SE Labs

-------------------Missed samples
Norton 360....................8 + 0 + 3 + 0 = 11
Trend Micro IS............13 + 6 + 0 + 1 = 20
F-Secure.....................19 + 1+*1 + 1 = 22
Kaspersky IS..............19 + 0 + 4 + 1 = 24

Avast..........................37 + 0 + 0 + 0 = 37
Microsoft....................37 + 0+*9 + 0 = 46
Avira (Free, Pro)..........46 + 2 + 8 + 3 = 59

McAfee (IS,TP)............76 + 1 + 3 + 0 = 80


Malware Protection 2019-2021: AV-Test, AV-Comparatives

-------------------Missed samples
Norton 360...............4 + 0 + 0.....= 4
F-Secure Safe........26 + 0 +*6....= 32

McAfee (IS,TP)......37 + 0 + 0.....= 37
Kaspersky IS..........28 + 8 + 4.....= 40
Microsoft...............25 + 0 +15....= 40
Avira (Free, Pro).....37 + 5 + 2.....= 44

Avast Free..............59 + 0 + 1.... = 60
TrendMicro IS........257+ 0 + 103 = 360

The AVs with the same color should not be differentiated (due to statistical errors).
The first column (large font) includes the cumulative results from 2019-2020.
The next two columns include the scorings from AV-Test and AV-Comparatives in the year 2021 (so far).
The fourth column in the Real-World table includes the results of SE Labs from the year 2021 (so far).
The last column (large font) includes the cumulative results from 2019-2021 (so far).

F-Secure did not participate in 3 AV-Comparatives tests in the year 2021 (Real-World FEB-MAR + APR-MAY, Malware Protection MAR 2021) and Microsoft Defender in one test (Real-World APR-MAY), so their results for the year 2021 were averaged on the basis of previous tests.
 
Last edited:

MacDefender

Level 14
Verified
Oct 13, 2019
685
Someone made a reply and I answered. Or is MalwareTips a place where free exchange and debate are not permitted? The tone I am getting from you is to "just shut up." That is censorship.
Censorship is different from repeatedly making the same point or dismissing other viewpoints without adding something of value to the conversation.

You first said very definitively:
Even Microsoft Defender gets 100%. That means paying for third party AV for security is a waste of money.
which many of us pointed out reasons why that's not the case, trying to be helpful. Making strong statements like that with a single or narrow set of tests to back up the claim is what will invite a lot of disagreement here. Many of us here, myself included, both agree that MD is a great product while also seeing value in the myriad of third party choices, many of which are stronger for security reasons and not just aesthetic or peculiar non-security reasons.
 

roger_m

Level 34
Verified
Content Creator
Dec 4, 2014
2,320
There is no illusion. Windows users do not need to install anything. Microsoft Defender is already there. And that is exactly what 70% of all Windows users do. They do nothing and just use stock Windows. And they live a lifetime on their Windows device and never have an infection. That fact adds up to "Thrid party AV is a waste of money for most users."

If Kaspersky free was so great, then everybody would be using it. Instead, most of those that do seek free AV opt for Avast.

Microsoft Defender gets the same scores at labs as Kaspersky products do. Kaspersky offers no advantage over Microsoft Defender according to the numbers.
The reason that so many people use Microsoft Defender is because it's free and is included with Windows. It's not that there aren't better alternatives. There are, with Kaspersky being a great example. But, since MD is already included with Windows, and works well, a lot of people aren't going to bother switching to a third party antivirus. However, if Windows did not include an antivirus, and you had to manually download and install MD, just like any third party antivirus, then it would have a much smaller market share. For plenty of users, maybe their system would run much better if they were using a lighter alternative, but since they've only ever used MD, they don't know any better. That's not to say that MD is particularly heavy, but plenty of people find some third party antiviruses to be significantly lighter.

If you look at test scores from testing organisations, they indicate that there typically any of the big name antiviruses will provide similar levels of protection. However, if you look at the tests here in the Malware Hub, where often more recent malware is tested, you will much more of difference between antiviruses. To provide one example, Panda often does very well when tested by testing organisations, but as you can see from the tests here, often fails terribly at detecting zero-day malware. So the idea that that all major antiviruses perform roughly the same and therefore that there is no (or very little) difference between MD and alternatives, is not true.

Finally, the idea that MD users never get infected, is a strange one. Some users of any antivirus will get infected. If you keep your system updated and are not click happy, then typically it will be very hard to get infected, no matter what antivirus you use. However, if you are click happy, then sooner or later you will get infected no matter what antivirus you use. Microsoft Defender provides adequate protection, which is great. But it's not reason enough to simply avoid third party antiviruses, which will in some cases provide a much better user experience.
 
Top