Hello
Adrian Ścibor,
It seems that changing the browser is not something important.
Except when it is important. WD is not like any AV because it is a part of built-in Windows security. For example, Microsoft cannot suggest/force users to install WDBP if they install Chrome. Other AVs can do it.
In tests that check the effectiveness of a particular endpoint agent, it is important to examine whether the program blocks the threat by various methods. The protocol for delivering malware to the system is important, not the browser.
Not in the real-world scenario, which is most important for the home users who are the main customers of free AVs. The CheckLab testing methodology is interesting but can silently invalidate the protection of some free AVs (WD "Block at First Sight" or Avast CyberCapture).
Edge / Chromium changes this and we'll check what we can do to automate it. For example, we didn't have a problem with Chrome. But there were problems with Firefox. That is why the change of browser in our test methodology is not so obvious.
The testing methodology is questionable for Avast and WD (even with Edge / Chromium), because some important AV features are bypassed in the real-world scenario (CyberCapture and Block at First Sight).
It's nothing wrong that the product X does not block the threat in the browser. It should do this at one of three levels in our methodology. But the sooner the better. Although we do not award any points for early blocking already in the browser. If the product does not block malware in the browser, then we have a chance to check whether proactive protection is effective or just a myth.
It is obviously not the right point of view for any AV which was made to work with Windows built-in browsers (Edge or IE) protected by SmartScreen.
And as for Windows Defender ... The browser must be the same for all products tested. Our point of view, a very small company, we would have to prepare new modules to automate only Windows Defender machines. This is not something we would like to do.
So, It would be better to not test WD (and probably Avast) in such a test.
If you will drop the web/browser protection part in your test, then it will be similar to the AV-Comparatives "Malware Protection Test". Such a test is focused on the malware delivered from network drives, USB or cover scenarios where the malware is already on the disk.
In this type of test, the AV features (including Avast and WD) are not invalidated.
The CheckLab testing methodology is interesting. But, there is no need to test AVs which does not fit the testing methodology.
Edit.
It is good that someone in my country decided to face in a professional way the extremely complex and demanding AV testing problem.