[CheckLab.pl] - Test of free antivirus

[oldschool]

Hello Adrian Ścibor,


Except when it is important. WD is not like any AV because it is a part of built-in Windows security. For example, Microsoft cannot suggest/force users to install WDBP if they install Chrome. Other AVs can do it.


Not in the real-world scenario, which is most important for the home users who are the main customers of free AVs. The CheckLab testing methodology is interesting but can silently invalidate the protection of some free AVs (WD "Block at First Sight" or Avast CyberCapture).


The testing methodology is questionable for Avast and WD (even with Edge / Chromium), because some important AV features are bypassed in the real-world scenario (CyberCapture and Block at First Sight).


It is obviously not the right point of view for any AV which was made to work with Windows built-in browsers (Edge or IE) protected by SmartScreen.


So, It would be better to not test WD (and probably Avast) in such a test.
If you will drop the web/browser protection part in your test, then it will be similar to the AV-Comparatives "Malware Protection Test". Such a test is focused on the malware delivered from network drives, USB or cover scenarios where the malware is already on the disk.
In this type of test, the AV features (including Avast and WD) are not invalidated.

The CheckLab testing methodology is interesting. But, there is no need to test AVs which does not fit the testing methodology.

Edit.
It is good that someone in my country decided to face in a professional way the extremely complex and demanding AV testing problem.

Thanks for your detailed reply. I always learn so much from your posts.

 

[Adrian Ścibor]

Hello Adrian Ścibor,

Except when it is important. WD is not like any AV because it is a part of built-in Windows security. For example, Microsoft cannot suggest/force users to install WDBP if they install Chrome. Other AVs can do it.

We cannot be responsible for what Microsoft do or do not do. If they don't provide protection for other browsers, they starts with a heavy cargo on his back.

Not in the real-world scenario, which is most important for the home users who are the main customers of free AVs. The CheckLab testing methodology is interesting but can silently invalidate the protection of some free AVs (WD "Block at First Sight" or Avast CyberCapture).

The Avast engineer responsible for contacting with us has a different opinion. Avast company is satisfied with our testing methods. Even thanks to them we have made some corrections on the backend's script. I cannot agree with you by that.

So, It would be better to not test WD (and probably Avast) in such a test.
If you will drop the web/browser protection part in your test, then it will be similar to the AV-Comparatives "Malware Protection Test". Such a test is focused on the malware delivered from network drives, USB or cover scenarios where the malware is already on the disk.
In this type of test, the AV features (including Avast and WD) are not invalidated.

There is a some difference in detecting the same malware transmitted through different protocols. It may be strange, but it really is. Especially if malware is delivered to the system via drag-and-drop, i.e. via a process such as vmtools.exe. by users on many YouTube videos.

We will discuss internal about how we can improve testing methods. I will refer to SmartScreen too. Let's see the situation on back. Do you think it's good if SmartScreen will block everything? Then how to check the protection of an anti-malware product that will not have the opportunity to react on malware? SmartScreen is also known for False Positive. We do not prepare such of tests. So if we want to examine protection, e.g. Kaspersky, the SmartScreen technology must be turned off to check at what level Kaspersky can block malware. Otherwise SmartScreen will block probalby 100%? malware on Level 2 according to our methodology. But the users will not know the real effectiveness of Kaspersky. Therefore wat is your opinion on this case?

 

[Andy Ful]

The Avast engineer responsible for contacting with us has a different opinion. Avast company is satisfied with our testing methods. Even thanks to them we have made some corrections on the backend's script. I cannot agree with you by that.

Why should he complain if Avast scored 100%. Simply, CyberCapture was not needed.
Here is the fragment from Avast FAQ:
"What conditions lead to a file being locked and uploaded by CyberCapture?

Currently, CyberCapture triggers when you run or download suspicious files from the Internet that CyberCapture has not previously encountered. We plan to expand this condition in the future to cover more sources."

CyberCapture - FAQs | Official Avast Support

What is CyberCapture? CyberCapture is a feature in Avast Antivirus that detects and analyzes rare, suspicious files. If you attempt to run a suspicious file, CyberCapture locks the file from your PC and sends it to the Avast Threat Labs where it is analyzed in a safe, virtual environment. You are...

support.avast.com

I tested this feature recently and confirmed that it works only if the file has MOTW with the information about the Internet zone, similarly to WD "Block at First Sight".

We will discuss internal about how we can improve testing methods. I will refer to SmartScreen too. ... Therefore wat is your opinion on this case?

I think that SmartScreen integrated with Explorer can be disabled in the test (most AV Labs do it). But, disabling SmartScreen in the web browser (IE, Edge, Edge Chromium) or using another web browser without WDBP, would be controversial to me in the real-world test.
In my opinion, it is natural to use the web browser extension if such an extension was intentionally prepared by the vendor for users who do not like IE or Edge. Microsoft would like much to force installing WDBP in Chrome-based web browsers, if this would not be forbidden by the Anti-Monopoly Law.

The second problem can be the "Block at First Sight" feature which works only when the files are downloaded from the Internet, or originate from the Internet zone (files with MOTW). Normally such files are checked via the cloud backend (heuristics, machine learning, and automated analysis). If the file is recognized as malicious, then it is automatically quarantined. This is the strongest detection feature of WD and it obviously does not work with CheckLab's methodology, because the malicious files could be executed in the test.

Finally, the most appropriate would be contacting with Microsoft to see what is their opinion. Maybe they will accept the methodology as it is (like in the case of Avast).

 

[Adrian Ścibor]

The second problem can be the "Block at First Sight" feature which works only when the files are downloaded from the Internet, or originate from the Internet zone (files with MOTW). Normally such files are checked via the cloud backend (heuristics, machine learning, and automated analysis). If the file is recognized as malicious, then it is automatically quarantined. This is the strongest detection feature of WD and it obviously does not work with CheckLab's methodology, because the malicious files could be executed in the test.

Finally, the most appropriate would be contacting with Microsoft to see what is their opinion. Maybe they will accept the methodology as it is (like in the case of Avast).

Therefore according to what you say - users shouldn't separate SmartScreen in their browser from Windows Defender. So how do you think you can only test the effectiveness of Windows Defender antivirus?

Suppose the user does not know what the message from SmartScreen. They downloaded some application. Starts it and ignore the warning. If ppp is harmful the WD should respond, right? So why do you think we should not run the samples? What if the application is trusted, such as CCleaner, which contained malicious code and SmartScreen is not enough? Is testing a WD by running samples wrong?

Finally how do you think we should test the WD?

 

[Evjl's Rain]

Therefore according to what you say - users shouldn't separate SmartScreen in their browser from Windows Defender. So how do you think you can only test the effectiveness of Windows Defender antivirus?

Suppose the user does not know what the message from SmartScreen. They downloaded some application. Starts it and ignore the warning. If ppp is harmful the WD should respond, right? So why do you think we should not run the samples? What if the application is trusted, such as CCleaner, which contained malicious code and SmartScreen is not enough? Is testing a WD by running samples wrong?

Finally how do you think we should test the WD?

I like the way you test WD because that's how I saw WD being infected many times in my country. Any test is valid for WD in specific conditions (malwares from USB flashdrives, network drives, password-protected zip files,...)
I think you misunderstood Andy somehow. Let me summarize it:
1/ A file must be downloaded from a browser (chrome, for example). Ignore IE/Edge's SmartScreen (which is the browser filter)
2/ Windows Smartscreen can be included or ignored, depends on your decision but it's better to add a description that SmartScreen is on or off
3/ The file after being downloaded should have "Mark-of-the-web" (when you right-click to the file and select "Properties", there should be "Unblock" button - but don't click it)
4/ When the file has Mark-of-the-web, Block-at-first-sign (BAFS) will be initiated. If the file doesn't have it, WD won't use BAFS

Andy means BAFS is the most powerful feature of WD so when you ignore it, you reduce WD strength a lot against new malwares

 

[SeriousHoax]

I like the way you test WD because that's how I saw WD being infected many times in my country. Any test is valid for WD in specific conditions (malwares from USB flashdrives, network drives, password-protected zip files,...)
I think you misunderstood Andy somehow. Let me summarize it:
1/ A file must be downloaded from a browser (chrome, for example). Ignore IE/Edge's SmartScreen (which is the browser filter)
2/ Windows Smartscreen can be included or ignored, depends on your decision but it's better to add a description that SmartScreen is on or off
3/ The file after being downloaded should have "Mark-of-the-web" (when you right-click to the file and select "Properties", there should be "Unblock" button - but don't click it)
4/ When the file has Mark-of-the-web, Block-at-first-sign (BAFS) will be initiated. If the file doesn't have it, WD won't use BAFS

Andy means BAFS is the most powerful feature of WD so when you ignore it, you reduce WD strength a lot against new malwares

Thanks for the clear summarization. I was a bit confused with Andy's response here as well. Anyway, continue the discussion guys. It's helpful.

 

[Andy Ful]

Therefore according to what you say - users shouldn't separate SmartScreen in their browser from Windows Defender. So how do you think you can only test the effectiveness of Windows Defender antivirus?

SmartScreen integrated with native Edge allows downloading files if they are not recognized as malicious. But, it can block the websites with a bad reputation (phishing or malware websites). So, the unrecognized malicious file can be downloaded (with MOTW) if the URL is not blocked. Next, the WD "Block at First Sight" is triggered to check the file against the cloud backend. In the real-world scenario, WD + Edge would block most samples on Level 1 (P1). In your test, all malicious samples were blocked by WD on Level 3 (P3).
At present, "Block at First Sight" can check portable executable files, scripts, and macros (also in archives).
If the file with MOTW is not recognized as malicious, then it can be executed and you will see the alert from the SmartScreen integrated with Explorer - the file reputation has been checked. If the user ignores this alert, then the file is finally executed. From this moment the SmartScreen and "Block at First Sight" are usually not triggered if the malware downloads payloads, because they are downloaded without MOTW. In fact, this was probably tested in your test.

Finally how do you think we should test the WD?

I can see three possibilities:

1. Do not change the methodology and do not test WD.
2. Test WD with WDBP + keep MOTW of downloaded files.
3. Test WD with WDBP (keep MOTW) and without WDBP (keep MOTW).

I would prefer the point 3, because it is easy to apply with your current methodology, it is most informative to the users, and it takes into account that WD is not the ordinary AV (it is preinstalled on Windows).
The requirement of keeping MOTW of downloaded files should be probably consulted with Microsoft. Before Evjl's Rain tests on Malware Hub, I thought that the MOTW is not necessary when the file is executed and checked by the cloud backend (cloud-delivered protection). The tests confirmed that there is a difference due to "Block at First Sight" which can detect files on access, and was stronger than cloud-delivered protection in Evjl's Rain tests. The tests were done some time ago, so this might be changed by Microsoft.

 

[Andy Ful]

Hi

I like the way you test WD because that's how I saw WD being infected many times in my country. Any test is valid for WD in specific conditions (malwares from USB flashdrives, network drives, password-protected zip files,...)
...

You know well how WD works - the importance of BAFS can be followed from your extensive tests on Malware Hub (thank you).

1. Your observation about infections could be easily confirmed by performing the Malware Protection test which is just right for this scenario -------> malwares from USB flashdrives, network drives, password-protected zip files,....
2. By the way, the infections you saw might be caused also by poor (interrupted) Internet connection or simply by running executables offline. As we know from some other tests, Windows Defender has poor offline signatures.

I would not also recommend WD in such cases.

 

[Evjl's Rain]

Hi

You know well how WD works - the importance of BASF can be followed from your extensive tests on Malware Hub (thank you).

1. Your observation about infections could be easily confirmed by performing the Malware Protection test which is just right for this scenario -------> malwares from USB flashdrives, network drives, password-protected zip files,....
2. By the way, the infections you saw might be caused also by poor (interrupted) Internet connection or simply by running executables offline. As we know from some other tests, Windows Defender has poor offline signatures.

I would not also recommend WD in such cases.

since WD is not consistent in all conditions, I would not use it in any of my PCs
those vectors are very common in my country and the internet connection is not always stable

I demand an AV to work well in all conditions and in the most difficult situations like when offline

I also noticed any small change with Windows Updates can affect heavily the functionality of WD. My WD while testing in the hub refused to update for many occasions because I did some tweaks to Windows Updates and privacy
then, WD only has its cloud signatures. The offline one was gone because it couldn't update
it was hard to fix

 

[LDogg]

Comodo Cloud Antivirus is a discontinued product - so a waste of time. Test should include Norton and F-secure.

Doesn't seem to have discountinued via this link as there's still an option to download it for free: Cloud Antivirus | Comodo Free Proactive Protection Software

 

[Andy Ful]

since WD is not consistent in all conditions, I would not use it in any of my PCs
those vectors are very common in my country ...

Your post can start the discussion of why one threw out the particular AV from his/her computer (price, resources, slowdowns, ads, selling data, incompatibilities, broken Windows Updates, broken system after updates, etc.). Let's do not do it.

 

[Andy Ful]

Honestly, I did not found the AV Lab which could test the real protection of AVs in one type of test. So far, we have real-world tests focused on the web-based threats and malware protection tests focused on the threats from USB, network drives, CD/DVD images, etc.
All tests except one I can recall, do not take into account how quickly the AV can protect all customers after the malware infected the first customer (first infection).
Another big problem: how representative can be the pule of the malware samples, which is like a needle in a haystack of all new malware. What it means the score 95% when the error (related to how representative is the sample) is not known. Is it 95% +- 0.1% or 95% +- 5% ?
There are some other factors like malware prevalence, etc.

So, testing the AVs can be very complicated and time-consuming. The testing methodology is not a science and results can have unknown errors. Are the people who do it, artists or scientists? I do not know but I really admire their efforts.

 

[Mjolnir]

Doesn't seem to have discountinued via this link as there's still an option to download it for free: Cloud Antivirus | Comodo Free Proactive Protection Software

Comodo Cloud Antivirus (CCAV) to be discontinued

Hi, Just been announced on the forums. Comodo Cloud (CCAV) will be discontinued to focus on the main product Comodo Antivirus. https://forums.comodo.com/news-announcements-feedback-ccav/notice-of-comodo-cloud-antivirus-discontinuation-t125043.0.html;topicseen

malwaretips.com

 

[Andy Ful]

There is also an unknown impact on detection results related to the number of AV customers. The AV1 with more customers can protect better than AV2 with fewer customers, even when both AVs have the same detection in AV tests. This cannot be easily calculated, because it can depend on malware prevalence, the number of new malware, and some other factors like the speed of protecting customers after the first infection (cloud response).
The advantage of AV1 over AV2 takes place when the same unknown malware tries to attack several customers.
For example, if the AV2 customers will be attacked 100 times by the same malware, then the AV1 customers will be attacked proportionally 100 *n1/n2 times, where n1 and n2 are numbers of customers and n1 > n2. But, if both AVs can protect just after the first attack, then there will be only 1 infected in AV1 group and 1 infected in AV2 group. Because the AV2 group is smaller, then the AV2 customer has greater chances to be hit by this malware.

 

[Adrian Ścibor]

Our findings:

First:
Machine with Comodo Internet Security. Antivirus engine has been disabled to check the settings you have recommended with SmartScreen and MOTW. Anyway... Windows system detects Comodo which is a basic anti-virus. Enabled SmartScreen for EDGE and Windows Defender SmartScreen for Apps.



Second:
Machine with Windows Defender. Default settings. Only UAC was disabled so that it does not interfere with the approval of automatic actions.

Third:
Machine with Bitdefender. Protection was disabled to confirm our findings.

Fourth:
Machine with G DATA. As above protection was disabled and SmartScreen for apps and EDGE was set as WARN.

What was done:

1. Machine with Comodo. Windows recognize Comodo as primary antivirus.
a. downloading malware from EDGE.
Result: The file has been downloaded. MOTW is visible in the properties of the downloaded file:



b. downloading malware from Chrome.
Result: The file has been downloaded at all. MOTW does not exist. It is not visible in the properties as above.



2. Machine with Windows Defender.
a. Downloading malware from Chrome.
Result: MOTW exists. The file has been blocked by Windows Defender. Malware cannot be moved or started.
b. Download from EDGE. MOTW also exists. The file has been blocked before executed.

3. Machine with Bitdefender. Protection was disabled. Windows recognize Bitdefender as primary AV.
a. Downloading malware from Chrome.
Result: MOTW doesn't exist. File is downloaded without problem (as in Comodo machine).



b. Downloading from EDGE.
Result: File was downloaded. But the MOTW exist.

4. Machine with G DATA.
a. Downloading malware from Chrome.
Result: File is downloaded and no MOTW.
b. Downloading malware from EDGE.
Result: MOTW exist.


General conclusions:

MOTW does not exist in Chrome if any antivirus, other than Windows Defender is installed in Windows. I suppose...
MOTW works only with EDGE if 3rd party antivirus is installed on Windows.
MOTW works with EDGE and Chrome if Windows Defender is primary antivirus.


Whats next?

What we can do as AVLab.pl/CheckLab.pk organization? We can easly adapt Windows Defender with default settings to our methodology with Chrome browser. Rest of antiviruses could be tested without changes.


Dear readers, thanks for this lesson. Do you have any suggestion or own experience that you can share with us?

 

[Evjl's Rain]

@Adrian Ścibor that's a bit weird to me. In my machine with Windows 8.1 and Chromium browser (not google chrome), MOTW is always present (for .exe files) regardless of what AV I use. I definitely never use Windows Defender
I saw that your malware sample's extension is not .exe but .exe ([a number starting from 1]) which may interfere with MOTW integration, I'm not sure
however, the first sample doesn't have (number) but still doesn't have MOTW
not sure what Microsoft does to Windows 10 to cause this problem

 

[Adrian Ścibor]

@Adrian Ścibor that's a bit weird to me. In my machine with Windows 8.1 and Chromium browser (not google chrome), MOTW is always present (for .exe files) regardless of what AV I use. I definitely never use Windows Defender
I saw that your malware sample's extension is not .exe but .exe ([a number starting from 1]) which may interfere with MOTW integration, I'm not sure
however, the first sample doesn't have (number) but still doesn't have MOTW
not sure what Microsoft does to Windows 10 to cause this problem

Could you check on Windows 10 in virtual machine?

 

[Evjl's Rain]

Could you check on Windows 10 in virtual machine?

here you are, this is my test on Windows 10 x64 v1903 in VMware
Browser: Google Chrome (latest)
AV: No AV - WD was disabled using Defender Control v1.5 (sordum.org)

I downloaded OOshutup from majorgeeks. MOTW was present
I also tried to download a .msi file (0patch) from their website, MOTW was present

 

[SeriousHoax]

here you are, this is my test on Windows 10 x64 v1903 in VMware
Browser: Google Chrome (latest)
AV: No AV - WD was disabled using Defender Control v1.5 (sordum.org)

I downloaded OOshutup from majorgeeks. MOTW was present
I also tried to download a .msi file (0patch) from their website, MOTW was present

View attachment 230500

@Adrian Ścibor Same result with Firefox. MOTW is present when downloaded from Firefox and I have ESET Internet Security active as my AV. MOTW won't be present if downloaded via a download manager like IDM.

SmartScreen is very aggressive though. Suggested me to not run it because of the publisher being unknown. This file was released 19 days ago.

 

[Adrian Ścibor]

@Adrian Ścibor Same result with Firefox. MOTW is present when downloaded from Firefox and I have ESET Internet Security active as my AV. MOTW won't be present if downloaded via a download manager like IDM.
View attachment 230507
SmartScreen is very aggressive though. Suggested me to not run it because of the publisher being unknown. This file was released 19 days ago.
View attachment 230514

May you check the same by Chrome?