Matousec - Proactive Security Challenge 64 (2013.1.18)

[savit]

Matousec Proactive Security Challenge 64 (2013.1.18) - http://www.matousec.com/projects/proactive-security-challenge-64/results.php

Interpretation of results

The best product tested against the latest set of tests and the Proactive Security Challenge 64 leader is currently Comodo Internet Security Premium 6.0.260739.2674 with excellent 92% score. Privatefirewall 7.0.28.1 is second with very good 88% score, closely followed by Kaspersky Internet Security 2013 13.0.1.4190 and Outpost Security Suite Pro 7.5.2.3939.602.1809, both with 86 %. These are the only products that have been awarded with Proactive Security Challenge 64 awards. There are two products with the exactly same score on the fifth and the sixth place. These are Outpost Security Suite Free 7.1.1.3431.520.1248 and VirusBuster Internet Security Suite 4.1 with 71 %.

Proactive Security Challenge 64 makes a big difference between good products and the rest of the world. Most of the products are filtered in very low levels which means that they probably miss some critical features. There are plenty of products whose vendors claim that they can protect you against all kinds known and unknown threats. But in fact their products implement just a basic set of features that can prevent the most simple attacks only. If a malicious application uses more advanced approach to achieve its goal it is not blocked. This is very common in case of self-defense features, anti-leak protection, or system infection prevention.

Differences between quality of protection on the 64-bit platform (tested in Proactive Security Challenge 64) and the 32-bit platform (formerly tested in Proactive Security Challenge) are also very interesting. The nature of the 64-bit platform makes it uneasy for the products to protect against some of the advanced attacking techniques, but most of the techniques can be prevented by the very same mechanisms as on the 32-bit platform. This is why one would expect to see the 64-bit results of the products to be just slightly worse than the results on the 32-bit platform. This is not the case, however. The protection of many products is on a totally different level on the 64-bit platform, unfortunately much worse level.

 

[McLovin]

RE: Matousec - Proactive Security Challenge 64 (2013.1.18))

Oh, damn gurl, CO-MO-DO did well. I mean, yes, these tests should be taken with very seriously.

 

[Deleted member 178]

matousec results must be took with a huge container of salt

1- Matousec tests is a firewall test based on prevention, so if a product doesn't have any prevention features (HIPS/BB) it will fail.

2- Matousec tests are like an obstacle race with 11 obstacles, if you fail one obstacle , you cant continue, even if your can pass the other ones.

3- Matousec does the test for free at the first run , give the results before making them public; but if the vendor is not satisfied by the result, it can pay for another run (mostly after fixing where they failed).

The only good point of matousec is that it made me discovered when i was a beginner about Comodo, Online Armor, etc...

 

[Exterminator]

The only advantage I see is that it is a long list of products some of which you might not have known about.

Honestly look at it.....4 products recommended 3 with links to "Get It Now". Everything else "Not Recommended". It is actually a joke.

Taking this with a grain of salt is an understatement.

 

[savit]

2- Matousec tests are like an obstacle race with 11 obstacles, if you fail one obstacle , you cant continue, even if your can pass the other ones.

NO!
The product has to score at least 50% in the tests on this level to pass it. - http://www.matousec.com/projects/proactive-security-challenge-64/level.php?num=1

3- Matousec does the test for free at the first run , give the results before making them public; but if the vendor is not satisfied by the result, it can pay for another run (mostly after fixing where they failed).

i sended feedback to matousec. i waiting the reply.

 

[Deleted member 178]

The product has to score at least 50% in the tests on this level to pass it. - http://www.matousec.com/projects/proactive-security-challenge-64/level.php?num=1

exactly what i said, if it fail, it can't go further,

 

[savit]

The product has to score at least 50% in the tests on this level to pass it. - http://www.matousec.com/projects/proactive-security-challenge-64/level.php?num=1

exactly what i said, if it fail, it can go further,

sorry. I was misunderstood.

 

[Deleted member 178]

opps typo mistake, i mean if on a level a product can't get 50% it cant reach the next level.

 

[illumination]

I looked at this site a few times in the past "about choked on the amount of salt i took with it", but like Umbra, learned of the different firewalls i have not heard of before..

 

[tapoo]

why and How they testing VirusBuster??? isn't VirusBuster is closed, and sold to Agnitum??
http://www.virusbuster.hu/en

 

[malbky]

Hmm there was news that Matousec is behind the scenes a puppet of comodo. And wow Comodo scores 92% what happened to all that perfect 100% scores.

 

[Plexx]

PF is however a good firewall alternative, regardless of the report, but what bothers me in this test is this:

They test both OSS Free and Pro but it is known that Free is somewhat behind against Pro in terms of technology.

I am actually curious to see the same exact chart for when Pro was on the same exact version of the current Free version and I do wonder if the results were mirrored.

And then, we have ZA's spot far behind from ESET?

Like nearly all tests, didn't take it with a grain of salt but with a salt shaker full.

 

[Exterminator]

Vendors' responses
We have received following responses to Proactive Security Challenge 64:

Agnitum Ltd. – the vendor of Outpost Security Suite Pro
2012-03-27 (Outpost Security Suite Pro 7.5.2.3939.602.1809 scored 86 % and took 2nd place): Securing 32- and 64-bit systems is in many ways different due to Microsoft Kernel Patch Protection. According to the tests some products protect only 32-bit OS's. The technology behind our products lets us protect all PCs - both 32-bit and 64-bit.

Konstantin Kutikov
Agnitum's Head of QA Department

Comodo Security Solutions, Inc. – the vendor of Comodo Internet Security Premium
2012-01-16 (Comodo Internet Security Premium 5.9.219863.2196 scored 94 % and took 1st place): Thank you for including our product in your latest tests. Your open-source, open methodology tests are always very well thought out and unique. The latest results indicate the fact that there is a reason for us to be able to offer a 500 USD virus free guarantee in all operating systems. Keep up the good work!

Egemen TAS,
Director, Desktop Security Products,
COMODO

PWI, Inc. – the vendor of Privatefirewall
2012-08-07 (Privatefirewall 7.0.28.1 scored 88 % and took 2nd place): Thank you Matousec for this latest testing effort for Privatefirewall and your continued commitment to security software vendors and their customers. Overall, we are pleased with Privatefirewall's improved score of 88 % and #2 ranking, but now have our attention focused on addressing the COM based and other vulnerabilities that remain.

Greg Salvato
CEO - Privacyware

2012-01-16 (Privatefirewall 7.0.25.5 scored 56 % and took 3rd place): While a #3 ranking is not very satisfying when combined with the Protection level earned by Privatefirewall in this initial round of the new x64 PSC, we are aware of the work required on our end to dramatically improve our performance and the effort to do so is already underway. We thank the Matousec organization for their continued commitment to helping vendors improve their products and ensuring that the consumers and businesses that rely on them are able to realize the greatest levels of system protection.

Greg Salvato
CEO - Privacyware

Only 4 Vendor responses who all just happen to be in the top 4 and are amazingly the only 4 recommended.Pay attention to the 2 responses by private firewall up from #3 to #2. I wonder where all the other Vendor responses are.I am sure there were a few more :s

I think the top 4 are all good products but there are way too many good products not recommended and to add a "Get It Now" link and recommend only 4 out of that whole list really just taints this whole report.

 

[savit]

3- Matousec does the test for free at the first run , give the results before making them public; but if the vendor is not satisfied by the result, it can pay for another run (mostly after fixing where they failed).

I received the reply from matousec.

Hello,

thank you for your message.

We do not do that in Proactive Security Challenge 64.
What we have here are two models of testing - paid and free.

Every vendor has a right to request a paid testing or a free testing.
If no requests are made, we select products ourselves and test them
and publish the results without prior notice to the vendor.
The only exception is in a case that there is a suspicion
that the results we have obtained are not right due to misconfiguration
of the product or in a case that there is some problem with the testing
process that can be solved with the assistance of the vendor.

If there is a request for a free test, we test the product for free
and publish the results without prior notice to the vendor. After
the results are published there is one month period in which
the published results can not be replaced even if a paid test is
requested.

If there is a request for a paid test, we do test the product
and send the results to the vendor and in case the vendor
is interested to publish the results then they are published,
otherwise they are not published.

Kind Regards,

 

[Venustus]

Why didn't they test Emsisoft?:huh:
I mean OAM is one of the best products on the market!

 

[arsenaloyal]

Why didn't they test Emsisoft?:huh:
I mean OAM is one of the best products on the market!

Here is your answer

http://www.matousec.com/info/?news=146-Online_Armor_temporarily_disqualified

 

[anitac]

Yeah, they punish Emsisoft for the money they claim Tall Emu owes them.

Mike Nash (Tall Emu) and Christian Mairoll (Emsisoft) have made a few comments on it at http://www.wilderssecurity.com/showthread.php?t=281529.

 

[Littlebits]

Matousec is a complete joke, don't take their tests serious.
Awhile back there was a post on Wilder's forum claiming that Comodo was the actual owners of Matousec. Although Comodo denied that they owned Matousec, there were several suspicious pointers that linked Comodo to the websites whois. Later the whois got protected by Domains By Proxy, LLC, so you couldn't tell who actually owns Matousec or see what links to it.

What kind of respectful trusted testing site would hide their whois info?

Therefore we don't know who actually controls and runs the testing or if they are even qualified to make accurate tests since they hide who they are from us.

Just like other testing sites they are fueled by money from their sponsors and can be influenced to tamper with testing results to make their sponsors happy and get more money.

They give many excellent products poor ratings and the advanced products that most users would never use or understand how to use the highest ratings.

Their testing is nothing more then advertisements for their paid vendors or maybe their own product (Comodo).

Thanks.

 

[Deleted member 178]

Why didn't they test Emsisoft?:huh:
I mean OAM is one of the best products on the market!

as said Abba "Money... money... money...in the rich man world..."

And i say: "Only the fools trust Test labs"