Disclaimer

This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
We encourage you to compare these results with others and take informed decisions on what security products to use.
Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Adrian Ścibor

From AVLab.pl
Verified
In October 2019, the CheckLab.pl organization prepared a summary of popular free applications to protect personal computers. So what is the best free antivirus? It is difficult to answer this question because a variety of free solutions makes the decision is not so obvious, additionally only big developers can offer relatively good protection. It is important to remember that free antivirus do not protect against all types of attacks. Free security software does not have the variety of functionality as commercial equivalents, so free antivirus should be at least complemented by a safe browser for online banking as well as a set of browser extensions to improve privacy and data protection.
In two paragraphs [...]:
  • Comparison of free and paid antivirus - what are the differences?
  • What is missing in free products?
[...] we tried to explain the differences between free and paid security products.

checklab_free_av_october_2019_chart.png

You can read full report at: Comparison of free security product to protect personal computers

Chart and table: Recent results

Awards: Awards

And about CheckLab: About us
 

Adrian Ścibor

From AVLab.pl
Verified
CCAV wasn't discontinued when the tests were performed, and it's up to AV developers to request to participate in the tests.
Exactly. This turned out already during the test.

In addition, Comodo technology remains in others Comodo products. Please note that Comodo has blocked 100% malware by running it in the sandbox:

See at Recent results and Level 3: The analysis level, i.e. a virus has been run and blocked by a tested product.

Which means they didn't have signatures. Proactive technology worked perfectly. But this also means that our testing method shows differences in products. It really works: 100% is not equal to 100% if there is no proactive protection.
 

Andy Ful

Level 62
Verified
Trusted
Content Creator
Hi, Adrian Ścibor
Thanks for the interesting test.
I think, that it would be more consistent to test the default Windows 10 built-in security setup (WD + Edge) available to average users without installing 3rd party applications. If someone wants to change the default Windows security, then he/she should be smart enough to install appropriate web browser extension, for example WDBP.
The results for WD without web browser protection can be easily predicted without any test and they cannot be compared to results of tests made by other AV Labs, which test WD + Edge (or Chrome + WDBP extension) in the real-world scenario.

Of course, the conclusion that follows from this test is true. Windows Defender with a web browser without URL filtering feature (anti-phishing and anti-malware) is worse in the real-world scenario than most free AVs with web protection.:)(y)
 
Last edited:

SeriousHoax

Level 29
Verified
Malware Tester
I think, that it would be more consistent to test the default Windows 10 built-in security setup (WD + Edge) available to average users without installing 3rd party applications.
Tbh, I think their method of testing Windows Defender is more appropriate because not everybody uses Microsoft Edge or installs WDBP extension which is not even available for Firefox. It's Microsoft's fault to not integrate smartscreen into their AV yet. Free AVs like Avast, Kaspersky Free, and some other doesn't need an extension and if required they ask the user to install them. So, I think this is a fair test and result for Windows Defender.
 

Andy Ful

Level 62
Verified
Trusted
Content Creator
Tbh, I think their method of testing Windows Defender is more appropriate because not everybody uses Microsoft Edge or installs WDBP extension which is not even available for Firefox. It's Microsoft's fault to not integrate smartscreen into their AV yet. Free AVs like Avast, Kaspersky Free, and some other doesn't need an extension and if required they ask the user to install them. So, I think this is a fair test and result for Windows Defender.
I think that we can gently disagree on this subject. More consistent for me may not be more appropriate for you. :)(y)
 

Andy Ful

Level 62
Verified
Trusted
Content Creator
The one thing that I would add in this test is WD + Chrome + WDBP or WD + Edge Chromium. In this way the users could see why WD without Edge (SmartScreen) can be a spoiled setup, and how it can be repaired.:)
It would be also interesting to see the results of WD + ConfigureDefender (HIGH or MAX Protection level). But, this will not happen because AV labs usually test the default configurations.
 
Last edited:

Adrian Ścibor

From AVLab.pl
Verified
slightly off topic....but doesn't smart screen run when chrome is launched as well?
It not. The SmartScreen technology works with IE or EDGE only. You have to install a browser extension know as Windows Defender Browser Protection, if you need protection on the 3rd party browser level. But this extenstion is not the same like SmartScreen. Moreover it is very strange why Microsoft's extension is not available for Firefox...

Also please consider that we are testing technologies for protection. The extension Windows Defender Browser Protection is not part of the official package from Microsoft. It is not ready for out-of-the-box. It means as default settings.
 

Mjolnir

Level 1
Verified
It was just an observation of mine when running Comodo Firewall - Whenever I launch Chrome - Smartscreen launches at the same time. It does not happen with Firefox - But with Edge and Chrome the behavior is identical. I wonder if Comodo firewall causes some kind of anomaly.
 

Andy Ful

Level 62
Verified
Trusted
Content Creator
It was just an observation of mine when running Comodo Firewall - Whenever I launch Chrome - Smartscreen launches at the same time. It does not happen with Firefox - But with Edge and Chrome the behavior is identical. I wonder if Comodo firewall causes some kind of anomaly.
It is probably not SmartScreen, but WD Block at First Sight feature. (y)
 
Last edited:

Andy Ful

Level 62
Verified
Trusted
Content Creator
I noticed one peculiar thing about testing methodology. In the case of WD, the samples were not blocked/quarantined after download. It means that WD Block at First Sight feature was not triggered. If so, then the most important WD feature (in the real-world scenario) was dysfunctional (it works by default with web browsers).
I think that the Manager which was used in the test did run the samples without "Mark Of The Web" (MOTW). That is not good, because some other AVs also can use MOTW to trigger additional protection in the real-world scenario. For example, the Avast CyberCapture feature does not work for EXE files without MOTW.

There is another thing that can differ this test from the tests made by other AV testing Labs. The samples were not demanding for most of the free AVs (100% protection????). Such results strongly suggest that the AV web/browser protection blocked the fresh samples and the others were simply blocked by signatures in the cloud. If so, then more advanced AV features were tested only for AVs without (or weak) web/browser protection.
 
Last edited:

notabot

Level 15
It not. The SmartScreen technology works with IE or EDGE only. You have to install a browser extension know as Windows Defender Browser Protection, if you need protection on the 3rd party browser level. But this extenstion is not the same like SmartScreen. Moreover it is very strange why Microsoft's extension is not available for Firefox...

Also please consider that we are testing technologies for protection. The extension Windows Defender Browser Protection is not part of the official package from Microsoft. It is not ready for out-of-the-box. It means as default settings.
I don't understand, Chrome also adds the Mark of the Web which triggers smartscreen.
 

Andy Ful

Level 62
Verified
Trusted
Content Creator
I don't understand, Chrome also adds the Mark of the Web which triggers smartscreen.
The post was about SmartScreen anti-phishing and anti-malware protection in Edge and IE web browsers. This protection works before any malicious file can be downloaded, and also works on any Windows version with Internet Explorer 9+ (anti-phishing module works also on IE 7+).
Additionally, SmartScreen is integrated into Windows 8+ operating system via SmartScreen Application Reputation. So, any file with MOTW (downloaded to hard disk) is checked by SmartScreen when the user wants to run this file from Explorer, or any popular web browser.

SmartScreen and Block at First Sight are the most important Windows/WD features to prevent malware in the real-world scenario. AV testing Labs usually disable/ignore SmartScreen alerts after executing the files, but allow either "SmartScreen in Edge/IE" or WD "Block at First Sight" feature. It seems that in this test both features are bypassed by the testing methodology.
 
Last edited:

Adrian Ścibor

From AVLab.pl
Verified
It seems that changing the browser is not something important. In tests that check the effectiveness of a particular endpoint agent, it is important to examine whether the program blocks the threat by various methods. The protocol for delivering malware to the system is important, not the browser.

On the other hand, Chrome / Chromium-based are the most-chosen browsers for users. Security vendors usually already have browser extensions for Chrome.

Earlier, testing with EDGE was not friendly because some vendors did not have the extension for non-popular browser.
Edge / Chromium changes this and we'll check what we can do to automate it. For example, we didn't have a problem with Chrome. But there were problems with Firefox. That is why the change of browser in our test methodology is not so obvious.

It's nothing wrong that the product X does not block the threat in the browser. It should do this at one of three levels in our methodology. But the sooner the better. Although we do not award any points for early blocking already in the browser. If the product does not block malware in the browser, then we have a chance to check whether proactive protection is effective or just a myth.

And as for Windows Defender ... The browser must be the same for all products tested. Our point of view, a very small company, we would have to prepare new modules to automate only Windows Defender machines. This is not something we would like to do.
 

Andy Ful

Level 62
Verified
Trusted
Content Creator
Hello Adrian Ścibor,

It seems that changing the browser is not something important.
Except when it is important. WD is not like any AV because it is a part of built-in Windows security. For example, Microsoft cannot suggest/force users to install WDBP if they install Chrome. Other AVs can do it.

In tests that check the effectiveness of a particular endpoint agent, it is important to examine whether the program blocks the threat by various methods. The protocol for delivering malware to the system is important, not the browser.
Not in the real-world scenario, which is most important for the home users who are the main customers of free AVs. The CheckLab testing methodology is interesting but can silently invalidate the protection of some free AVs (WD "Block at First Sight" or Avast CyberCapture).

Edge / Chromium changes this and we'll check what we can do to automate it. For example, we didn't have a problem with Chrome. But there were problems with Firefox. That is why the change of browser in our test methodology is not so obvious.
The testing methodology is questionable for Avast and WD (even with Edge / Chromium), because some important AV features are bypassed in the real-world scenario (CyberCapture and Block at First Sight).

It's nothing wrong that the product X does not block the threat in the browser. It should do this at one of three levels in our methodology. But the sooner the better. Although we do not award any points for early blocking already in the browser. If the product does not block malware in the browser, then we have a chance to check whether proactive protection is effective or just a myth.
It is obviously not the right point of view for any AV which was made to work with Windows built-in browsers (Edge or IE) protected by SmartScreen.

And as for Windows Defender ... The browser must be the same for all products tested. Our point of view, a very small company, we would have to prepare new modules to automate only Windows Defender machines. This is not something we would like to do.
So, It would be better to not test WD (and probably Avast) in such a test.
If you will drop the web/browser protection part in your test, then it will be similar to the AV-Comparatives "Malware Protection Test". Such a test is focused on the malware delivered from network drives, USB or cover scenarios where the malware is already on the disk.
In this type of test, the AV features (including Avast and WD) are not invalidated.

The CheckLab testing methodology is interesting. But, there is no need to test AVs which does not fit the testing methodology.:)(y)

Edit.
It is good that someone in my country decided to face in a professional way the extremely complex and demanding AV testing problem.(y)(y)(y)
 
Last edited:

notabot

Level 15
The CheckLab testing methodology is interesting. But, there is no need to test AVs which does not fit the testing methodology.:)(y)

Edit.
It is good that someone in my country decided to face in a professional way the extremely complex and demanding AV testing problem.(y)(y)(y)
We need more tests and we need tests on more stuff. Many folks question the validity of here tests and instead go and do their own testing, which all this says imo is we need more testing labs, with more methodologies and it's good this lab's staff is actually opening a dialogue.

I'd like to see tests expand beyond AVs as well, eg to UTM devices, to IoT devices ( do they use strong ciphers? is all info encrypted? how good is their key management? are the cloud endpoints they connect to secure? do they rely on upnp? etc etc ).

They could also compare non AV mechanisms, eg how well did a hardened chrome with hardened exploit guard settings perform against known exploits during the last year?

For AV's I'd also like to see a per module evaluation and comparison.
 
Top