- Content source
- http://www.amtso.org/PR20150506
When you read an anti-malware product review, you expect an honest representation of the security product. The reviewer also expects that the software submitted for review has not been tampered or falsified from the product available to customers in any way. When these rules of engagement are not met, hard-earned reputations become tarnished and customers are put at unnecessary risk. In the last few weeks, the dishonest actions of a few security vendors has not only impacted the reputation of respected security testing bodies, but impacted the industry as a whole.
How? By e.g. "submitting a different product for review than what was actually offered their users" or by "having optimizations in the product only to perform better in a performance test".
This situation is not unlike someone buying a car based on a review highlighting its great NCAP rating for safety, only to find that the model purchased does not even include an airbag. Not only are the reputations of the car manufacturer, sellers and testing bodies are all negatively impacted, but the security of the purchaser is also put at risk.
If the security product delivered to consumers or businesses differs from the one reviewed by a reputable testing body, it could give the buyer a false sense of security. The results for the user could vary from malicious attackers accessing sensitive data to total disruption of the system and user experience.
Cheating goes against everything AMTSO stands for. Our mission is to improve the relevance and accuracy of anti-malware products testing. It is something that benefits all the community:
Conversely, bad testing harms us all:
Testing plays an important role in benchmarking a vendor’s solution for both existing users and prospects, but it is vital that submitted products are not tampered with for the benefit of achieving a better testing result. This approach is not only dishonest and unethical, but it puts everyone - from tester, to seller, to buyer, to user - at risk.
Reputable testers set up an environment that emulate the real-world as closely as possible. The tester hammers the security product against real-world threats and recording the behavior - both good and bad - for the vendor. This approach allows the anti-malware vendor to review the findings and improve the solution for the benefit of its users.
In the case where the product delivered to the testers is falsified or tampered with in some way in order to achieve a better test result, the test cannot reflect the anti-malware product’s true capability. Testers waste their time analysing findings of a bogus product and readers of those tests are misguided about the product’s true capabilities.
Ultimately, ethics play a central role for both the testers and vendors. Unethical behavior cannot be tolerated and decisions need to be made about how to penalise those who have bypassed the rules of engagement in order to falsely achieve a better testing result.
Stripping dishonest vendors of previously earned testing certifications and awards is a viable approach. Going public when a vendor breaks the rules of engagement is also important: it sends a clear warning to other vendors about the importance of ethical behavior.
The AMTSO Board
How? By e.g. "submitting a different product for review than what was actually offered their users" or by "having optimizations in the product only to perform better in a performance test".
This situation is not unlike someone buying a car based on a review highlighting its great NCAP rating for safety, only to find that the model purchased does not even include an airbag. Not only are the reputations of the car manufacturer, sellers and testing bodies are all negatively impacted, but the security of the purchaser is also put at risk.
If the security product delivered to consumers or businesses differs from the one reviewed by a reputable testing body, it could give the buyer a false sense of security. The results for the user could vary from malicious attackers accessing sensitive data to total disruption of the system and user experience.
Cheating goes against everything AMTSO stands for. Our mission is to improve the relevance and accuracy of anti-malware products testing. It is something that benefits all the community:
- product testing drives anti-malware vendors to improve their solutions:
- encourages innovation so vendors can offer better solutions than their competitors
- finds product issues and bugs to be resolved by vendors before products are dispatched to users
- better solutions provides more resilient threat protection for users; and
- better testing provides users with data to make an informed decision about what solutions fit their specific needs in order to get the best protection.
Conversely, bad testing harms us all:
- anti-malware vendors focus on ways to obtain better testing results, rather than researching and developing better protection for users;
- bad testing provides consumers and businesses with misleading information; and
- the reputations of all involved are tarnished.
Testing plays an important role in benchmarking a vendor’s solution for both existing users and prospects, but it is vital that submitted products are not tampered with for the benefit of achieving a better testing result. This approach is not only dishonest and unethical, but it puts everyone - from tester, to seller, to buyer, to user - at risk.
Reputable testers set up an environment that emulate the real-world as closely as possible. The tester hammers the security product against real-world threats and recording the behavior - both good and bad - for the vendor. This approach allows the anti-malware vendor to review the findings and improve the solution for the benefit of its users.
In the case where the product delivered to the testers is falsified or tampered with in some way in order to achieve a better test result, the test cannot reflect the anti-malware product’s true capability. Testers waste their time analysing findings of a bogus product and readers of those tests are misguided about the product’s true capabilities.
Ultimately, ethics play a central role for both the testers and vendors. Unethical behavior cannot be tolerated and decisions need to be made about how to penalise those who have bypassed the rules of engagement in order to falsely achieve a better testing result.
Stripping dishonest vendors of previously earned testing certifications and awards is a viable approach. Going public when a vendor breaks the rules of engagement is also important: it sends a clear warning to other vendors about the importance of ethical behavior.
The AMTSO Board
