SE Labs Test of Enterprise AV

simmerskool

Level 31
Thread author
Verified
Top Poster
Well-known
Apr 16, 2017
2,073
Last edited by a moderator:

Jonny Quest

Level 17
Verified
Top Poster
Well-known
Mar 2, 2023
799
What was edited, your link? I did a search using your Thread title. Shows Endpoint protection Q2 April-June, correct?

compare.jpg
 

simmerskool

Level 31
Thread author
Verified
Top Poster
Well-known
Apr 16, 2017
2,073
What was edited, your link? I did a search using your Thread title. Shows Endpoint protection Q2 April-June, correct?

View attachment 277442
What was edited, your link? I did a search using your Thread title. Shows Endpoint protection Q2 April-June, correct?

View attachment 277442
Not sure, I guess I had 2 links, one a summary and one a pdf of the full report, and originally, I included the URL with pdf link. is that against the rules? :unsure: if so, sorry.
Also I posted in Security News, but apparently that was the wrong forum. Oh well.
 

Jonny Quest

Level 17
Verified
Top Poster
Well-known
Mar 2, 2023
799
Not sure, I guess I had 2 links, one a summary and one a pdf of the full report, and originally, I included the URL with pdf link. is that against the rules? :unsure: if so, sorry.
Also I posted in Security News, but apparently that was the wrong forum. Oh well.
No problem, my friend. I just know how passionate you are regarding Enterprise/Endpoint security, that I didn't know why you hadn't included a link :)
But, it's all good now :)
 

ScandinavianFish

Level 7
Verified
Dec 12, 2021
317
Interesting. The crowdstrike results conflict with @shadora testing results.

I would trust our member testing more than a company which might have ulterior motives.
This is the exact reason I don't trust any of these labs, as the results showcased by members like Shadowra aswell as my own testing that I sometimes do in my free timealmost always contradict these so called "independent' labs. A example being Kaspersky that consistently score 100% in terms of malware detection from testing conducted here, which I have also personally seen when testing it myself (I don't use it and i'm no fanboy, but it is a top notch product which we all know from testing.), and yet a couple of these labs show results as low as 97%. Yes, no product is perfect and can actually provide 100% protection, but inconsistencies this large make me very suspicious as to how they actually test products, or even if they intentionally scew the results.

Another example is TotalAV, one of those grayware products that use the Avira engine. Sometimes it gets the lowest score out of all tested products, and sometimes it get's a perfect result, outscoring even top products like Kaspersky.
 
Last edited:

Silverwing

New Member
Jul 26, 2023
6
This is the exact reason I don't trust any of these labs, as the results showcased by members like Shadowra aswell as my own testing that I sometimes do in my free timealmost always contradict these so called "independent' labs. A example being Kaspersky that consistently score 100% in terms of malware detection from testing conducted here, which I have also personally seen when testing it myself (I don't use it and i'm no fanboy, but it is a top notch product which we all know from testing.), and yet a couple of these labs show results as low as 97%. Yes, no product is perfect and can actually provide 100% protection, but inconsistencies this large make me very suspicious as to how they actually test products, or even if they intentionally scew the results.

Another example is TotalAV, one of those grayware products that use the Avira engine. Sometimes it gets the lowest score out of all tested products, and sometimes it get's a perfect result, outscoring even top products like Kaspersky.
These tests arent accurate at all. Bitdefender did horibble yet its the 2nd best thing tested here besides Kaspersky's EDR which is extremely good.
 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,182
It is worth mentioning that any test made on the MT forum cannot reliably suggest whether the SE Labs test is accurate or not.
In the same way, the SE Labs test cannot disprove the tests made by AV-Comparatives, AV-Test, etc.
Also, the tests made by SE Labs, AV-Comparatives, and AV-Test cannot disprove the results of tests made on the MT forum.

All these tests can show only a part of the truth. They are not in contradiction to each other, because when one wants to reliably compare the results, the statistical error is bigger than the differences of the results. The main reasons for that are very different testing methodologies, a small number of samples, and small differences in the results of tested AVs.

I cannot see any reason to prefer any particular methodology used in different tests. :) (y)
 
Last edited:

ScandinavianFish

Level 7
Verified
Dec 12, 2021
317
It is worth mentioning that any test made on the MT forum cannot reliably suggest whether the SE Labs test is accurate or not.
In the same way, the SE Labs test cannot disprove the tests made by AV-Comparatives, AV-Test, etc.
Also, the tests made by SE Labs, AV-Comparatives, and AV-Test cannot disprove the results of tests made on the MT forum.

All these tests show a part of truth. They are not in contradiction to each other, because when one wants to reliably compare the results, the statistical error is bigger than the differences of the results. The main reasons for that are very different testing methodologies, a small number of samples, and small differences in the results of tested AVs.
The main difference is that members on this forum can post video's confirming the results, while these labs just post a bunch of numbers without context or anything confirming the results. They are simply not transparent enough to warrant any kind of trust.

Another problem is that they have to use so many samples that once they begin "testing, many of those samples will be many days old, which could explain why even sub-par products get such high detection rates, while members here can get samples while they are still fresh (albeit not as many), which is a more realistic scenario. Afterall, cybercriminals rarely use the same malware for more than a week, as at that point too many products will be detecting it (unless it's inflated to several hundred megabytes.). We can also get many more different types of malware thanks to sources like Malware Bazaar, while many (if not most) of the samples used by independent testing labs are likely duplicates or just variants of known malware.
 
Last edited:

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,182
The main difference is that members on this forum can post video's confirming the results, ...
Such videos can be used only when the number of samples is very small or a very specific kind of test is performed.
In the first case, the results are not statistically meaningful to compare AVs (sometimes the results can be useful for other things, like showing a possible weakness or demonstrating how the AV works).
In the second case, the methodology has nothing to do with the real scenario, so the results can be incorrect.

The most important part of the test is how representative is the pule of tested samples, compared to the samples in the wild. Unfortunately, this information is absent in all tests (also made by professional AV testing labs). Most people believe that professional AV testing labs can prepare more representative samples because they share samples with AV vendors and use the testing methodology approved by AMTSO.

For me, the advantage of professional tests follows from systematic testing. So, several AVs are tested for a long time month by month, with the same methodology. Such testing can be statistically verified and the comparison results can be statistically meaningful.

Comparing AVs without proper statistics is like crystal-ball gazing.
 
Last edited:

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,182
Some people who test AVs for a long time can intuitively know that a few AVs can provide better protection. But, their conclusions are usually very similar to statistical considerations based on professional tests.:)
 

Andy Ful

From Hard_Configurator Tools
Verified
Honorary Member
Top Poster
Developer
Well-known
Dec 23, 2014
8,182
The AVs do not protect people against all possible malware and all possible attacks. Their main purpose is to prevent the malware epidemic and decrease the profit gained via cyber attacks. So, breaking the AV (on default settings) by making a POC or modifying a known malware is not a great challenge. It is also not a problem to create several malware samples and gain some profit. The AV vendors do not bother to prevent such activities, just like the police do not bother to catch all petty criminals.
 
Last edited:

Sandbox Breaker

Level 9
Well-known
Jan 6, 2022
436
The AVs do not protect people against all possible malware and all possible attacks. Their main purpose is to prevent the malware epidemic and decrease the profit gained via cyber attacks. So, breaking the AV (on default settings) by making a POC or modifying a known malware is not a great challenge. It is also not a problem to create several malware samples and gain some profit. The AV vendors do not bother to prevent such activities, just like the police do not bother to catch all petty criminals.
Said like a true master!
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top