Link:
Matousec Proactive Security Challenge Analyzed
6. Final Thoughts
Of course any experienced user can use most of the same tests used in Matousec testing since they are located on the website for a free download (
http://www.matousec.com/downloads/). Therefore, money can't plausibly influence the validity of the actual tests since the tests are available to everyone.
The test results are linked by a PDF file and anyone can see the types of tests a product fails or passes. Since the raw data is posted to the site, you can ignore the overall score and just look at the tests passed or failed. However, the PDF has little value when it doesn't list enough testing levels to allow readers to make sound interpretations of the results. They might as well not even list level 1 products.
I'm suspicious of many scoring practices in the Proactive Security Challenge. For example, I find it problematic that they give products 0% for levels not tested and that they score products by the number of possible tests (when many of the tests were not actually administered). I found it confusing that they compare products based on the total number of possible tests. And the claim that their results validate (or invalidate) the security claims of vendors is false.
However, no other similar testing service (for proactive security or outbound protection testing) exists as far as I know, so Matousec has little competition. And, as stated at the beginning, I appreciate the thoroughness and technical details of their service. It should be noted that their website is informative and detailed about their testing methods.
