64-bits are uneasy to protect for many [Matousec]

[savit]

64-bits are uneasy to protect for many - http://www.matousec.com/info/?news=156-__-bits-are-uneasy-to-protect-for-many

Sixteen products have been tested in Proactive Security Challenge 64 so far, so there are still many top products that are missing in the results, but it seems that the message we get is simple – security vendors find it hard to protect 64-bit Windows machines. Compared to the 32-bit platform, the results of the tested products are much worse. Yes, the overall difficulty of Proactive Security Challenge 64 is higher but that would justify only minor differences in results.

The only product that confirmed its quality on 64-bit platform so far is Comodo Internet Security Premium 5.9. In Proactive Security Challenge it reached the perfect 100% score, its result in Proactive Security Challenge 64 is just 6 % lower, which is reasonable and still an excellent result. Other products, however, lose tens of percents in the score. They fail even the techniques that they passed in the 32-bit testing and that can be protected with the very same code on both 32-bit and 64-bit platforms. This is quite disturbing.

We hope to find more products that will succeed on 64-bit platform during the upcoming tests.

The only product that confirmed its quality on 64-bit platform so far is Comodo Internet Security Premium 5.9. In Proactive Security Challenge it reached the perfect 100% score, its result in Proactive Security Challenge 64 is just 6 % lower, which is reasonable and still an excellent result.

Proactive Security Challenge 64 Result - http://www.matousec.com/projects/proactive-security-challenge-64/results.php

 

[ZeroDay]

Thanks for this post : ) Well done Comodo!

 

[Hungry Man]

I still dont understand why.

Patchguard prevents antimalware products from hooking the kernel directly. But it also prevents malware form hooking the kernel directly. They both get knocked up a notch on the OSI... so why can't AM simply use application layer API instead of kernel/NT?

 

[Littlebits]

matousec.com is not a reliable source, search on Wilder's security forum.

Some highlights:

1. Whois info is hidden with Domains By Proxy, LLC because nobody knows who actually owns or runs the testing site.

2. Back in 2006, matousec.com announced that they were bought by another unknown company that had no history on any web searches that is when their whois started to get hidden. Some people were saying that Comodo was the company that bought them but no way to verify who was their true owners. What was really strange about the whole thing was it was right after PC Tools Firewall Plus and Online Armor got a higher score than Comodo.

3. Ethical issues, they way that they charge their clients for testing fees, the more they pay the better results they will get.

4. Controversy with Tall Emu (developer of Online Armor) and Emsisoft. They claimed that when Online Armor was still owned by Tall Emu, the testing fees were not paid. Tall Emu said they paid matousec.com but they wanted more money because Online Armor achieved a high score so then refused to pay more. At this time Emsisoft had already bought Online Armor and was not responsible for the dispute between Tall Emu and matousec.com, but matousec.com still tried to get some money for them as well. When Emsisoft also refused to pay, matousec.com deleted all previous test results for Online Armor including the ones that were paid for. That's why you don't see Online Armor is their tests now.

5. Testing mythology- Is known to tests software that are not designed to pass many of their selected tests. Uses many methods in their testing not ever known to be used by any form of malware, if malware doesn't use those methods and why should you worry about your software passing these tests? Their testing mythology can lead to false impressions making users think they have to use Comodo in order to be completely protected.

But there is even much more about matousec.com which even makes it more unreliable if you are willing to search you can find it.

Thanks.

 

[ZeroDay]

Thanks for the info Littlebits

 

[Valentin N]

matousec.com is not a reliable source, search on Wilder's security forum.

Some highlights:

1. Whois info is hidden with Domains By Proxy, LLC because nobody knows who actually owns or runs the testing site.

2. Back in 2006, matousec.com announced that they were bought by another unknown company that had no history on any web searches that is when their whois started to get hidden. Some people were saying that Comodo was the company that bought them but no way to verify who was their true owners. What was really strange about the whole thing was it was right after PC Tools Firewall Plus and Online Armor got a higher score than Comodo.

3. Ethical issues, they way that they charge their clients for testing fees, the more they pay the better results they will get.

4. Controversy with Tall Emu (developer of Online Armor) and Emsisoft. They claimed that when Online Armor was still owned by Tall Emu, the testing fees were not paid. Tall Emu said they paid matousec.com but they wanted more money because Online Armor achieved a high score so then refused to pay more. At this time Emsisoft had already bought Online Armor and was not responsible for the dispute between Tall Emu and matousec.com, but matousec.com still tried to get some money for them as well. When Emsisoft also refused to pay, matousec.com deleted all previous test results for Online Armor including the ones that were paid for. That's why you don't see Online Armor is their tests now.

5. Testing mythology- Is known to tests software that are not designed to pass many of their selected tests. Uses many methods in their testing not ever known to be used by any form of malware, if malware doesn't use those methods and why should you worry about your software passing these tests? Their testing mythology can lead to false impressions making users think they have to use Comodo in order to be completely protected.

But there is even much more about matousec.com which even makes it more unreliable if you are willing to search you can find it.

Thanks.

From what I have once heard and read, they test without fee every 6month, if I don't remember wrong. If people want the to test their products again then they need to pay. From what I know comodo has done that quite often, but that doesn't mean that comodo buy their result. I don't find that ethical problem; look at the 32bit, there you will see that there are other that are good. Besides CIS does well in other proactive protection test, which still shows that Matousec is not that wrong.

The best thing is to test yourself with malware and other leaktests.

 

[illumination]

From what I have once heard and read, they test without fee every 6month, if I don't remember wrong. If people want the to test their products again then they need to pay. From what I know comodo has done that quite often, but that doesn't mean that comodo buy their result. I don't find that ethical problem; look at the 32bit, there you will see that there are other that are good. Besides CIS does well in other proactive protection test, which still shows that Matousec is not that wrong.

The best thing is to test yourself what with malware and other leaktests.

These are the type of post's that make my short visit's to the forum enjoyable! If one is to take into account all angles, instead of specific aspects, it will most times paint an entire different picture!

The last sentence being the over-all best way to form an objective opinion!

 

[Valentin N]

From what I have once heard and read, they test without fee every 6month, if I don't remember wrong. If people want the to test their products again then they need to pay. From what I know comodo has done that quite often, but that doesn't mean that comodo buy their result. I don't find that ethical problem; look at the 32bit, there you will see that there are other that are good. Besides CIS does well in other proactive protection test, which still shows that Matousec is not that wrong.

The best thing is to test yourself what with malware and other leaktests.

These are the type of post's that make my short visit's to the forum enjoyable! If one is to take into account all angles, instead of specific aspects, it will most times paint an entire different picture!

The last sentence being the over-all best way to form an objective opinion!

Glad to hear

I dislike that people give wrong or misleading information, that lead people making wrong opinion, when it may not be true. An average user can't control or check how the methodology is really done. I wonder who really can? Not to go offtopic but some people (I am part of it), at comodo forums, made a community based testing group, we only stared. one of the point is to record the testing, so that the user see what's happening.

What's important is to get fact that are not emotionally loaded and then put the fact to a test; you can get it and test. This is good to see if they tell the truth.

 

[illumination]

What's important is to get fact that are not emotionally loaded and then put the fact to a test; you can get it and test. This good to see if they tell the truth.

Exactly, and agree 100%

 

[NSG001]

Best way to test is in-house.
i.e. on your own machines.
I trust no one
Comodo's security products are extremely solid products if set up correctly.

 

[Hungry Man]

Matousec's methodology is bs lol they test malware in levels.

So if you pass level 1 you go to level 2 and if you pass 2 you go to 3 and then you fail 4 and you stop.

But what if you fail level 1 and stop even though you could have passed 2 and 3 and 4?

It's nonsensical.

 

[Littlebits]

Matousec's methodology is bs lol they test malware in levels.

So if you pass level 1 you go to level 2 and if you pass 2 you go to 3 and then you fail 4 and you stop.

But what if you fail level 1 and stop even though you could have passed 2 and 3 and 4?

It's nonsensical.

I forgot to mention that, yes there level testing just don't make sense. If they would just drop the levels and test all software against all of their tests then they would get a more accurate result.
If a products fails in level 1 then it is a complete dismissal even though it could have passed all of the other levels with flying colors. So it would be so easy to present false results to make one product look better than another product. All you would have to do is move the levels around in a different order.

Example product A fails Level 1 and passes the other levels, but that product paid for a second test and better score.
Simply move Level 1 to Level 10, now that product gets a better score.

Just ignorant testing methods.

Thanks.

 

[illumination]

Now don't get me wrong, I am not defending Matousec's methodology, but what about the other "independent testing" companies that only test a portion of the whole suite to come up with results for some of the "bigger name" security corporations? Are their testing methods flawed as well?

It is why I agree that testing yourself, is the best method!

 

[Valentin N]

Matousec's methodology is bs lol they test malware in levels.

So if you pass level 1 you go to level 2 and if you pass 2 you go to 3 and then you fail 4 and you stop.

But what if you fail level 1 and stop even though you could have passed 2 and 3 and 4?

It's nonsensical.

I assume that the higher the more complex they are and I think that they think: if they can't pass lvl1 (or higher), they won't pass the next lvl.

 

[HeffeD]

Matousec's methodology is bs lol they test malware in levels.

Almost any test you do will be flawed in some way. That is the reason I've never given any serious thought to any malware tests I see published by any testing organization. Especially those that state 'detection rates'...

This is why AMTSO was created. Without testing standards, any sort of rating derived from malware testing is by-and-large useless.

 

[Hungry Man]

I agree that no test is perfect. I don't really care about tests.

But of all of the tests, matousec has the least credibility. It just doesn't even make sense.

 

[Deleted member 178]

I agree with HeffeD, i dont care anymore on testing organization, when i was a beginner they helped me to know which ones were at the top so i can make a choice, then i relied on my own tests and experiences when using them.

 

[Littlebits]

Matousec's methodology is bs lol they test malware in levels.

Almost any test you do will be flawed in some way. That is the reason I've never given any serious thought to any malware tests I see published by any testing organization. Especially those that state 'detection rates'...

This is why AMTSO was created. Without testing standards, any sort of rating derived from malware testing is by-and-large useless.

I totally agree with you, do you know any testing sites that are AMTSO certified?

Many things you need to consider when looking at a testing organization:

1. Are they really qualified to conduct an accurate test, do they have the knowledge and skills needed.
2. Are they using good samples or testing methodologies.
3. Is it really an independent organization not influenced with money offered by clients.
4. Is the organization trustworthy? who are the testers (a group, individuals, students, amateurs, etc.) Whois info about the website is a good way to find out, only shady websites hide their whois info. including sites that (have illegal content, steal intellectual properties, spread malware, have fake or misleading content, etc.) No respectful website hides its whois info.
5. Who sponsors the organization? If the testing organization is sponsored by a single client that has a product in the testing, it is a conflict of interest.

Thanks.

 

[Littlebits]

I agree that no test is perfect. I don't really care about tests.

But of all of the tests, matousec has the least credibility. It just doesn't even make sense.

Completely agree, matousec is the worst testing site that I have ever seen.

It was pretty good when it first came out and only tested firewall (network control) but after it was taken over by another unknown company (Different Internet Experience Ltd.) the testing results, methodology, etc. kept getting less creditable. Many members at Wilder's forum believe the testing site to be a sick joke.

Thanks.

 

[HeffeD]

I totally agree with you, do you know any testing sites that are AMTSO certified?

No, unfortunately I'm not aware of any testing organizations that are AMTSO certified.

I think most organizations are resistant to follow the guidelines for the obvious reasons.

Some organizations such as Virus Bulletin are AMTSO members, but their testing methodologies aren't AMTSO certified. :s