AVLab.pl Advanced In-The-Wild Malware Test - September 2025

[Parkinsond]

And another:

This website is not detected by Bitdefender, according to VirustTotal database; however, Bitdefender scored 100%, equal to Webroot!

VirusTotal

VirusTotal

www.virustotal.com

Assessment Of Effectiveness Of Protection Against In-the-wild Threats In The Windows Environment (September 2025) » AVLab Cybersecurity Foundation

Since the last edition of the Advanced In-The-Wild Malware Test in July 2025, several significant changes worth noting have taken place in the methodology and

avlab.pl

 

[Adrian Ścibor]

This website is not detected by Bitdefender, according to VirustTotal database; however, Bitdefender scored 100%, equal to Webroot!

VirusTotal does not use the same engines 1:1 as in the desktop versions. I mentioned this once at an IT conference in Poland:

CP - CheckPoint engine on VirusTotal VS. CheckPoint installed on workstations VS 4 different samples:

Antivirus software on your computer is not the same as the engine in VirusTotal​

The antivirus engines used in VirusTotal operate from the command line usually and API. As a result, they may not have access to the modules that are included in real security packages. This demonstrates a practical approach to testing. For example, malware that would be blocked by a firewall module in a real-world scenario will not be blocked by the antivirus engine on VirusTotal.

As stated in the official VirusTotal document, the antivirus engines used are binary versions that run from the command line. They will not behave exactly the same as the versions we install on our computers: AV product on VirusTotal detects a file and its equivalent commercial version does not

The engines used on VirusTotal do not have a firewall usually, sandbox, HIPS, DLP, script virus blocking, or other modules. It depend on specific configuration demonstrated by vendor, also the condfigration is not reveal for VT users).

The difference between antivirus software installed on computers and the engines on VirusTotal is the first important reason why you should not rely on VirusTotal scan results compared to results from our test.

Edit:

Besides, the 100% result in the image you show refers to combined protection (PRE+POST), so it is not relevant now whether Webroot blocked something as a known URL and Bitdefender did not.

Additionally, I showed screenshots for Webroot from November 2025, and we are commenting here on the results from the September 2025 of Advanced In-The-Wild Malware Test.

 

[Parkinsond]

VirusTotal does not use the same engines 1:1 as in the desktop versions

But it used for 19 vendors which detected the website!

 

[Adrian Ścibor]

But it used for 19 vendors which detected the website!

I don't understand the question.

No detection at the PRE_Launch stage is not equal to FAIL in our test, because there is another stage, POST_Launch, and the sample is run in Windows 11, then is the final verdict for the sample determined, the Remediation Time is calculated, the logs are copied, and so on.

 

[Parkinsond]

No detection at the PRE_Launch stage is not equal to FAIL in our test

100% score in your test is the sum of pre and post launch.

 

[LinuxFan58]

I think there is a discrepancy in what tests proof and people experience.

Professional test laboraties proof that security software provide adequent protection even against the latest threats.

I am convinced that even the "ordinary average PC user" is well protected by the products tested by those testlab.

Only there are bugs in programs which can be exploited and some companies knowingly delay patches or keep using old versions while every now and then software developers patch bugs to late. This combined with human stupidity falling for social engineering and sphere phising is the reason we read about the millions of damage caused by ransomware attacks.

I think this discrepancy (what we read and the often 100% protection test results) leads to popular belief in security forums that the tests are skewed or the professional testlab use old samples.

The other effect of this discrepancy is that some people feel the need to protect themselves against those threats like some people prep for the next world war.

Because security is big business, it security consultancy firms and the security vendors keep on feeding that emotion by publishing the latest intrusion discoveries and the catastrophic effect they might have.

 

[Parkinsond]

I think discrepancy leads to popular belief in security forums that the tests are skewed or the professional testlab use old samples

Exactly, users expect to see test results with K, B, Avast scoring higher than QuickHeal and Webroot.
When they score equally, there is two possibilitie; the first, the test is flawed; the second, such products are overestimated by fans and marketing brokers (each forum has some on), and ultimately all security products are equal.

 

[LinuxFan58]

And there is also the fun factor (e.g members on a Chromebook running all sorts of audits and intrusion detection software in a Linux VM)

 

[Parkinsond]

And there is also the fun factor (e.g members on a Chromebook running all sorts of audits and intrusion detection software in a Linux VM)

Cannot comment regarding ChromeOS or Linus distros, as I never used before.

 

[ForgottenSeer 123960]

And there is also the fun factor (e.g members on a Chromebook running all sorts of audits and intrusion detection software in a Linux VM)

There is learning then there is ignorant, which category do you fall into.

It's already been discussed a thousand times in this forum that these tests are baseline test. They simply can not account for all variables in real world scenarios, including users systems, software and habits.

There are those that obviously have mental health issues pertaining to the overkill security setups and obsession. The latter part is a category you should be familiar with, as you constantly follow me and dig as if you want to kiss me or something. Not sure what your problem is. Are you simple boy, do you think I'm cute?

Seriously though back on topic. There other parts of this not discussed enough in these type forums are habits. This is because it's a show here, it's ego's, its flaunting. You can take a test group of ten people, teach them good habits, hand the systemd with just windows default security and an add blocker on their browser and turn them loose for a year, and chances are very great none of them would end up infected. They might become a victim to a breach and have data stolen, but this breach will not be in their personal system and will not matter how much they use their habits as it will happen in a server with some service they use and trust and they will have no control over this. The best you can do is teach them to be very selective on what they use and divulge.

This is reality

 

[Adrian Ścibor]

As we described in the article about increasing the Excellence Certificate threshold: We Are Changing The Certification Thresholds In The Advanced-In-The-Wild Malware Test » AVLab Cybersecurity Foundation.

In reality, 100% effectiveness in a limited sample set does not guarantee 100% effectiveness across the entire threat population.

The result is always subject to randomness, as the product may simply be “lucky” with the selected sample set. All it takes is for the next 10 random samples to be different and the result could fall below 100%.

And so on...

 

[ForgottenSeer 123960]

Cannot comment regarding ChromeOS or Linus distros, as I never used before.

Open source software is a different beast then closed source windows, especially when you consider most hackers live and breath Linux. They will know how to bust through Linux easier then they ever will Windows. Linux is what runs the web, hell it is the web. All your severs are Linux for example. Simple minds can not grasp this because they barely can grasp browser extensions for blocking ads. They spend too much time on forums following other users and degrade them not realizing the valuable time they waste doing so. Some of us sirnd this time learning everything we can. Not because we are paranoid, or have the need to run tons of security, but because we know when and where to apply what's is needed to secure without having to go over board with it.

For example. I run Chromebook with Linux environment, audits and Intrusion Detection software, hence why the previous idiot mentioned that. What he doesn't mention is that I run Wireshark and Nmap and test the intrusion detection system at default settings "Suricata" because this helps me establish attack vectors that are actually applicable and then I create firewall rules for iptables based on those. It was learning how to to defend the front door of a Linux machine while simultaneously becoming very familiar with a very powerful IDS/IPS, which one can actually create custom rules in as well to prevent such attacks like packet fragmentation with timing attacks for example or rate limit incoming pings to prevent floods ECT.

Now mind you this is very different then setting up a system for malware defense and it worrying about getting infected. This about learning how to harden a network and devices from automated attacks ECT.

 

[Khushal]

Exactly, users expect to see test results with K, B, Avast scoring higher than QuickHeal and Webroot.
When they score equally, there is two possibilitie; the first, the test is flawed; the second, such products are overestimated by fans and marketing brokers (each forum has some on), and ultimately all security products are equal.

I can confirm second is not true.

 

[Parkinsond]

I can confirm second is not true.

So we are left here with a solitary possibility.

 

[Trident]

The discussion here is now many different topics all at once and it’s difficult to follow up.

If Bitdefender failed to detect this link or website, it means Bitdefender simply got no report of it. Could be that none of the Bitdefender users actually ever fell onto that website.
Bitdefender can still detect the executable once it is downloaded.

Similarly, there will be many cases where Webroot won’t identify the link but the downloaded payload will be blocked (especially if user increased the heuristics sensitivity).

As discussed before, Webroot never had problems with malware in executable form, in fact, it has consistently demonstrated that typical methods that could render other solution ineffective, like padding and so on, don’t work with Webroot.
When it comes to more advanced malware and attacks which also affect home users, Webroot is not effective.

The 100% protection on this test means that all malware at some stage of the attack used executables.

It’s important to note that companies like Sophos for example (their Sophos Labs division) process 500K malicious files on average daily.
So any test uses “one drop from an ocean”.
This is where discrepancies come from.
There is a clear disclaimer on the AV-Comparatives website for the same.

 

[Parkinsond]

The discussion here is now many different topics all at once and it’s difficult to follow up.

If Bitdefender failed to detect this link or website, it means Bitdefender simply got no report of it. Could be that none of the Bitdefender users actually ever fell onto that website.
Bitdefender can still detect the executable once it is downloaded.

Similarly, there will be many cases where Webroot won’t identify the link but the downloaded payload will be blocked (especially if user increased the heuristics sensitivity).

As discussed before, Webroot never had problems with malware in executable form, in fact, it has consistently demonstrated that typical methods that could render other solution ineffective like padding and so on, don’t work with Webroot.
When it comes to more advanced malware and attacks which also affect home users, Webroot is not effective.

The 100% protection on this test means that all malware at some stage of the attack used executables.

It’s important to note that companies like Sophos for example (their Sophos Labs division) process 500K malicious files on average daily.
So any test uses “one drop from an ocean”.
This is where discrepancies come from.
There is a clear disclaimer on the AV-Comparatives website for the same.

Which test results would you consider more?
A test with 9 out 10 security products score the full mark or a test with two products scoring full mark, one product missed sample, two products missed two samples, and so on?
Which one reflects the real-world experience more?

 

[Divine_Barakah]

Which test results would you consider more?
A test with 9 out 10 security products score the full mark or a test with two products scoring full mark, one product missed sample, two products missed two samples, and so on?
Which one reflects the real-world experience more?

Is it not the same with AV-Test? I don't see a huge difference in protection levels in AV-Test for example.

 

[ForgottenSeer 114717]

I feel @bazang chiming in here

 

[Trident]

Which test results would you consider more?
A test with 9 out 10 security products score the full mark or a test with two products scoring full mark, one product missed sample, two products missed two samples, and so on?
Which one reflects the real-world experience more?

The real world experience is very subjective and dependant on many factors.
We saw here users disabling Kaspersky and executing files from the sort of KittyWallpaperMeow.exe which effectively led to 0% protection and a high number of threads which at one point evolved to humour (because at that point there was just no other outcome).

Similarly, we see people running Webroot with no issue whatsoever.

So the experience with the solutions varies per person and usage but generally a test where all solutions demonstrate insane efficacy means the malware was either known, or just more from what was already known.

With unknown malware results will be different.

 

[Parkinsond]

The real world experience is very subjective and dependant on many factors.
We saw here users disabling Kaspersky and executing files from the sort of KittyWallpaperMeow.exe which effectively led to 0% protection and a high number of threads which at one point evolved to humour (because at that point there was just no other outcome).

Similarly, we see people running Webroot with no issue whatsoever.

So the experience with the solutions varies per person and usage but generally a test where all solutions demonstrate insane efficacy means the malware was either known, or just more from what was already known.

With unknown malware results will be different.

Cann't agree more