[ CheckLab.pl/en ] Differences in software for PC protection in CheckLab tests (January 2020)

Disclaimer
  1. This test shows how an antivirus behaves with certain threats, in a specific environment and under certain conditions.
    We encourage you to compare these results with others and take informed decisions on what security products to use.
    Before buying an antivirus you should consider factors such as price, ease of use, compatibility, and support. Installing a free trial version allows an antivirus to be tested in everyday use before purchase.

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
173
Hello Dear Readers.

In January 2020 the CheckLab.pl organization prepared a list of popular solutions to protect home computers and work stations of micro, medium, and large enterprises. Among the tested solutions there are 8 specialized top-class products for end customers, and also 2 solutions for companies — Comodo Advanced Endpoint Protection and Webroot Business Endpoint Protection.

In the January comparison, the CheckLab employees would like to draw the attention to:

(1) Differences in blocking threats that arise from the protection mechanisms implemented in a tested security solution.

(2) The effectiveness of signatureless protection is shown as the Level 3 in charts, and in the table.


Recent results you can find at: Recent results

Full report: Differences in software for PC protection in CheckLab.pl tests (January 2020)

checklab-january-2020-results.png


sha256-malware-checklab-en.png


You probably wonder why we publish sample details? Published checksums of malicious software have a beneficial influence on transparency of the tests, and build trust to the testing organization.

This year at our website CheckLab.pl/en we want to add visualize the sample behavior in the system - dynamic correlation of events based on Sysmon tool. We want to be transparent much as possible.

We have several other development plans also, but we will describe it in a separate article.
 

mlnevese

Level 26
Verified
Top Poster
Well-known
May 3, 2015
1,531
Soon we'll have the "which one is better" discussions, the wave of people changing security solutions because of this lab result and the people who will say the test is bad because their favorite didn't perform well/was not included.
 
Last edited:

Burrito

Level 24
Verified
Top Poster
Well-known
May 16, 2018
1,363
Hello Dear Readers.

In January 2020 the CheckLab.pl organization prepared a list of popular solutions to protect home computers and work stations of micro, medium, and large enterprises. Among the tested solutions there are 8 specialized top-class products for end customers, and also 2 solutions for companies — Comodo Advanced Endpoint Protection and Webroot Business Endpoint Protection.

We have several other development plans also, but we will describe it in a separate article.

Thanks for the test results Adrian.

I understand a reality of the malware test business. You show a lot of red (detection failure) --- and you potentially lose a paying AV client.

Tests with results where all vendors detect all/most malware with different capabilities are great for retaining possible paying customers.

But not so valuable for us consumers to compare products.

I think what you see here in Malware Hub is closer to reality in terms of capability detection failure rates in the real world.

The 'Everybody gets a Trophy' test results --- they maybe don't provide as much value to actually evaluate products.

But the break-out of different detection modes for different products is still instructive.

Thanks.
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
173
Looking at the graphs, i don't see anyone of the tested solution failing, just some of them intervening earlier than others in the attack chain.

G DATA did not detect 3 samples. Even so reached 99% and get the BEST +++ certificate. We need to improve the charts to somehow include up to 3 decimal places. The red line on the chart is not visible. We need to work on it to improve charts in some way.

Thanks for the test results Adrian.

I think what you see here in Malware Hub is closer to reality in terms of capability detection failure rates in the real world.

Thanks.

We can cooperate with Malware Hub if possible but I haven't talked to anyone about it yet.
 

Tiamati

Level 12
Verified
Top Poster
Well-known
Nov 8, 2016
574
We can cooperate with Malware Hub if possible but I haven't talked to anyone about it yet.

It would be awesome!

G DATA did not detect 3 samples. Even so reached 99% and get the BEST +++ certificate. We need to improve the charts to somehow include up to 3 decimal places. The red line on the chart is not visible. We need to work on it to improve charts in some way.

@Adrian Ścibor

I guess you already seem a lot of discuss around here about the lack of confidence the independence tests have in security forums. It's just tooooo hard to believe that everyone is SO good... Webroot for example is always getting horrible results in mt hub, but it seems to get perfect results in "independent tests". It would be good if you and other authors from independent tests took part on those discussions and show your side of the history. ;)
 

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
173
I guess you already seem a lot of discuss around here about the lack of confidence the independence tests have in security forums. It's just tooooo hard to believe that everyone is SO good... Webroot for example is always getting horrible results in mt hub, but it seems to get perfect results in "independent tests". It would be good if you and other authors from independent tests took part on those discussions and show your side of the history

The Webroot product is specific. It is official news that Webroot does not participate in AV-C testing because their methodology cannot examine Webroot's solution - their cloud technology with RollBack. We can do it in CheckLab tests. Some time ago a Webroot employee met with an AV-C to discuss this case. The AV-C methodology cannot show Webroot protection in my opinion.

Independent MalwareTips Hub Testers should look at the following files when analyzing the result. It is often the case that Webroot needs up to 20-30 minutes to detect and rollback the infection.

C:\ProgramData\WRData\WRLog.txt
C:\ProgramData\WRData\WRLog.log !!!!!!!
C:\ProgramData\WRData\ace1.db
C:\ProgramData\WRData\dbk.db
 
Last edited:

Evjl's Rain

Level 47
Verified
Honorary Member
Top Poster
Content Creator
Malware Hunter
Apr 18, 2016
3,684
The Webroot product is specific. It is official news that Webroot does not participate in AV-C testing because their methodology cannot examine Webroot's solution - their cloud technology with RollBack. We can do it in CheckLab tests. Some time ago a Webroot employee met with an AV-C to discuss this case. The AV-C methodology cannot show Webroot protection in my opinion.

Independent MalwareTips Hub Testers should look at the following files when analyzing the result. It is often the case that Webroot needs up to 20-30 minutes to detect and rollback the infection.

C:\ProgramData\WRData\WRLog.txt
C:\ProgramData\WRData\WRLog.log !!!!!!!
C:\ProgramData\WRData\ace1.db
C:\ProgramData\WRData\dbk.db
I would like to know if webroot can prevent data stealers like spywares or keyloggers? Does it prevent those from connecting to the internet? If they are actively running in the infected system + they send our data back to the creators, how can webroot revert that (too late)?
therefore, I think rollback is a great idea only if the primary protection is strong enough (not the case with webroot). because there is an interval when users are infected, especially with ransomwares -> users lose their files for some time before they are rolled back
occasionally, malwares actions cannot be rolled back or it takes ages
 
Last edited:

Adrian Ścibor

From AVLab.pl
Thread author
Verified
Well-known
Apr 9, 2018
173
I would like to know if webroot can prevent data stealers like spywares or keyloggers? Does it prevent those from connecting to the internet? If they are actively running in the infected system + they send our data back to the creators, how can webroot revert that (too late)?
therefore, I think rollback is a great idea only if the primary protection is strong enough (not the case with webroot). because there is an interval when users are infected, especially with ransomwares -> users lose their files for some time before they are rolled back
occasionally, malwares actions cannot be rolled back or it takes ages

Very interesting idea. We can talk with Webroot and prepare paper that shows how Wobroot react on keyloggers and stealers data.
 
Last edited:

imuade

Level 12
Verified
Top Poster
Well-known
Jul 29, 2018
566
I would like to know if webroot can prevent data stealers like spywares or keyloggers? Does it prevent those from connecting to the internet? If they are actively running in the infected system + they send our data back to the creators, how can webroot revert that (too late)?
therefore, I think rollback is a great idea only if the primary protection is strong enough (not the case with webroot). because there is an interval when users are infected, especially with ransomwares -> users lose their files for some time before they are rolled back
occasionally, malwares actions cannot be rolled back or it takes ages
Very interesting idea. We can talk with Webroot and prepare paper that shows how Wobroot react on keyloggers and stealers data.
I think you need to change two WRSA settings to check this:
 

SeriousHoax

Level 47
Well-known
Mar 16, 2019
3,630
I think one of the important thing to take out of this test is an AV's ability to protect at browser level. Bitdefender gives us the perfect example here. It blocked almost everything at browser level which is brilliant. Anyone who has used their TrafficLight extension already have an idea about how good they are in this department. This is why Bitdefender always performs excellent in other test labs like AV-Test, AV-Comparatives (Product of the year) even though in the hub it didn't always performed well. Same goes to Avira. Nothing is wrong with our malware hub tests but one should remember that it showcases a products signatures and behavioral protection capabilities but the first line of defense is ignored.
So, every tests are different and all of them are valid in their own way and useful when done right. CheckLab-pl's tests are more transparent than other labs so we highly appreciate it.
 
F

ForgottenSeer 72227

This is why Bitdefender always performs excellent in other test labs like AV-Test, AV-Comparatives (Product of the year) even though in the hub it didn't always performed well. Same goes to Avira. Nothing is wrong with our malware hub tests but one should remember that it showcases a products signatures and behavioral protection capabilities but the first line of defense is ignored.
So, every tests are different and all of them are valid in their own way and useful when done right. CheckLab-pl's tests are more transparent than other labs so we highly appreciate it.

This is why it's important to understand what the test is trying to accomplish and try to look at the bigger picture. Every test has their pros and cons, comparing this test to the HUB is a great example. Each test focuses on different things, but this test highlights what other components can potentially add to the overall protection, something the HUB doesn't capture. The HUB however focuses on zero day malware, something that we cannot verify with other tests because they didn't provide us with the info to verify the samples, with the exception of this test. I really wish that the other professional tests would follow this example of how to provide us with the checksums, but they won't unfortunately. Their explanations are just weak excuses because this test shows that it actually can be done.
 

woodrowbone

Level 10
Verified
Dec 24, 2011
480
Very interesting idea. We can talk with Webroot and prepare paper that shows how Wobroot react on keyloggers and stealers data.

If I remember correctly, Webroot monitors all unknown files that are executed, in "monitored" state the file should be locked down by the firewall, unable to connect out to command servers etc.
The interesting thing would be for you to test if this holds up in a real world scenario Adrian (y)

/W
 
  • Like
Reactions: ichito

ichito

Level 11
Verified
Top Poster
Content Creator
Well-known
Dec 12, 2013
541
BTW, someone actually checked how old are their samples?
It's easy to find on their methodology page
4. Selecting samples for tests
4.1. Every 24 hours, the system downloads malware collected in the past 24 hours from all honeypots.
/QUOTE]
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top