SE Labs Test of Enterprise AV

[simmerskool]

Welcome to the second edition of the Enterprise Advanced Security test that compares different endpoint security products directly.
...but do not see Checkpoint Harmony or DeepInstinct

Endpoint Detection Compared: in-depth EDR security report

Endpoint Detection Compared. SE Labs tested a variety of Endpoint Detection and Response products against a range of hacking attacks.

selabs.uk

PDFSource

 

[Jonny Quest]

What was edited, your link? I did a search using your Thread title. Shows Endpoint protection Q2 April-June, correct?

 

[Andy Ful]

The link (no download):

Endpoint Detection Compared: in-depth EDR security report

Endpoint Detection Compared. SE Labs tested a variety of Endpoint Detection and Response products against a range of hacking attacks.

selabs.uk

 

[Ink]

What was edited, your link?

No, only accidental formatting was removed.

I found this URL using web search: 2023 - SE Labs

 

[simmerskool]

What was edited, your link? I did a search using your Thread title. Shows Endpoint protection Q2 April-June, correct?

View attachment 277442

What was edited, your link? I did a search using your Thread title. Shows Endpoint protection Q2 April-June, correct?

View attachment 277442

Not sure, I guess I had 2 links, one a summary and one a pdf of the full report, and originally, I included the URL with pdf link. is that against the rules? if so, sorry.
Also I posted in Security News, but apparently that was the wrong forum. Oh well.

 

[Jonny Quest]

Not sure, I guess I had 2 links, one a summary and one a pdf of the full report, and originally, I included the URL with pdf link. is that against the rules? if so, sorry.
Also I posted in Security News, but apparently that was the wrong forum. Oh well.

No problem, my friend. I just know how passionate you are regarding Enterprise/Endpoint security, that I didn't know why you hadn't included a link
But, it's all good now

 

[cartaphilus]

Interesting. The crowdstrike results conflict with @shadora testing results.

I would trust our member testing more than a company which might have ulterior motives.

 

[harlan4096]

No, only accidental formatting was removed.

I found this URL using web search: 2023 - SE Labs

I also moved the thread, since was posted in wrong section, probably that movement removed original link, so I added the current 2...

 

[ScandinavianFish]

Interesting. The crowdstrike results conflict with @shadora testing results.

I would trust our member testing more than a company which might have ulterior motives.

This is the exact reason I don't trust any of these labs, as the results showcased by members like Shadowra aswell as my own testing that I sometimes do in my free timealmost always contradict these so called "independent' labs. A example being Kaspersky that consistently score 100% in terms of malware detection from testing conducted here, which I have also personally seen when testing it myself (I don't use it and i'm no fanboy, but it is a top notch product which we all know from testing.), and yet a couple of these labs show results as low as 97%. Yes, no product is perfect and can actually provide 100% protection, but inconsistencies this large make me very suspicious as to how they actually test products, or even if they intentionally scew the results.

Another example is TotalAV, one of those grayware products that use the Avira engine. Sometimes it gets the lowest score out of all tested products, and sometimes it get's a perfect result, outscoring even top products like Kaspersky.

 

[MTUser]

No Cortex, no FortiEDR, no SentinelOne, bah...

 

[simmerskool]

I also moved the thread, since was posted in wrong section, probably that movement removed original link, so I added the current 2...

...sorry about that

 

[Silverwing]

This is the exact reason I don't trust any of these labs, as the results showcased by members like Shadowra aswell as my own testing that I sometimes do in my free timealmost always contradict these so called "independent' labs. A example being Kaspersky that consistently score 100% in terms of malware detection from testing conducted here, which I have also personally seen when testing it myself (I don't use it and i'm no fanboy, but it is a top notch product which we all know from testing.), and yet a couple of these labs show results as low as 97%. Yes, no product is perfect and can actually provide 100% protection, but inconsistencies this large make me very suspicious as to how they actually test products, or even if they intentionally scew the results.

Another example is TotalAV, one of those grayware products that use the Avira engine. Sometimes it gets the lowest score out of all tested products, and sometimes it get's a perfect result, outscoring even top products like Kaspersky.

These tests arent accurate at all. Bitdefender did horibble yet its the 2nd best thing tested here besides Kaspersky's EDR which is extremely good.

 

[Andy Ful]

It is worth mentioning that any test made on the MT forum cannot reliably suggest whether the SE Labs test is accurate or not.
In the same way, the SE Labs test cannot disprove the tests made by AV-Comparatives, AV-Test, etc.
Also, the tests made by SE Labs, AV-Comparatives, and AV-Test cannot disprove the results of tests made on the MT forum.

All these tests can show only a part of the truth. They are not in contradiction to each other, because when one wants to reliably compare the results, the statistical error is bigger than the differences of the results. The main reasons for that are very different testing methodologies, a small number of samples, and small differences in the results of tested AVs.

I cannot see any reason to prefer any particular methodology used in different tests.

 

[ScandinavianFish]

It is worth mentioning that any test made on the MT forum cannot reliably suggest whether the SE Labs test is accurate or not.
In the same way, the SE Labs test cannot disprove the tests made by AV-Comparatives, AV-Test, etc.
Also, the tests made by SE Labs, AV-Comparatives, and AV-Test cannot disprove the results of tests made on the MT forum.

All these tests show a part of truth. They are not in contradiction to each other, because when one wants to reliably compare the results, the statistical error is bigger than the differences of the results. The main reasons for that are very different testing methodologies, a small number of samples, and small differences in the results of tested AVs.

The main difference is that members on this forum can post video's confirming the results, while these labs just post a bunch of numbers without context or anything confirming the results. They are simply not transparent enough to warrant any kind of trust.

Another problem is that they have to use so many samples that once they begin "testing, many of those samples will be many days old, which could explain why even sub-par products get such high detection rates, while members here can get samples while they are still fresh (albeit not as many), which is a more realistic scenario. Afterall, cybercriminals rarely use the same malware for more than a week, as at that point too many products will be detecting it (unless it's inflated to several hundred megabytes.). We can also get many more different types of malware thanks to sources like Malware Bazaar, while many (if not most) of the samples used by independent testing labs are likely duplicates or just variants of known malware.

 

[Andy Ful]

The main difference is that members on this forum can post video's confirming the results, ...

Such videos can be used only when the number of samples is very small or a very specific kind of test is performed.
In the first case, the results are not statistically meaningful to compare AVs (sometimes the results can be useful for other things, like showing a possible weakness or demonstrating how the AV works).
In the second case, the methodology has nothing to do with the real scenario, so the results can be incorrect.

The most important part of the test is how representative is the pule of tested samples, compared to the samples in the wild. Unfortunately, this information is absent in all tests (also made by professional AV testing labs). Most people believe that professional AV testing labs can prepare more representative samples because they share samples with AV vendors and use the testing methodology approved by AMTSO.

For me, the advantage of professional tests follows from systematic testing. So, several AVs are tested for a long time month by month, with the same methodology. Such testing can be statistically verified and the comparison results can be statistically meaningful.

Comparing AVs without proper statistics is like crystal-ball gazing.

 

[Andy Ful]

Some people who test AVs for a long time can intuitively know that a few AVs can provide better protection. But, their conclusions are usually very similar to statistical considerations based on professional tests.

 

[Sandbox Breaker]

I cant ever take these seriously. MRG Effitas "used" to be good. I've broken ALL of these solutions whilst testing.

 

[Andy Ful]

The AVs do not protect people against all possible malware and all possible attacks. Their main purpose is to prevent the malware epidemic and decrease the profit gained via cyber attacks. So, breaking the AV (on default settings) by making a POC or modifying a known malware is not a great challenge. It is also not a problem to create several malware samples and gain some profit. The AV vendors do not bother to prevent such activities, just like the police do not bother to catch all petty criminals.

 

[Sandbox Breaker]

The AVs do not protect people against all possible malware and all possible attacks. Their main purpose is to prevent the malware epidemic and decrease the profit gained via cyber attacks. So, breaking the AV (on default settings) by making a POC or modifying a known malware is not a great challenge. It is also not a problem to create several malware samples and gain some profit. The AV vendors do not bother to prevent such activities, just like the police do not bother to catch all petty criminals.

Said like a true master!

 

[Andy Ful]

Said like a true master!

Thanks, master.