- Apr 25, 2013
- 5,358
A new study reveals that anti-malware solutions are not as good as first thought, with most unable to detect new 'in the wild' malware. But some of the vendors in question have hit back.
Delta Testing, a specialist in anti-malware testing methodologies and research, released a new report on Thursday highlighting shortcomings in the detection rates of leading players in the advanced threat solution market.
In its report, the company found ‘significant' discrepancy between the seven vendors tested, with FireEye the top-performing vendor with its NX7500 product detecting 99.14 percent of new threats. Others were not quite as efficient – Fidelis solutions (XPS Edge 200, XPS Sensor, XPS Command Post) detected just 5.17 percent of malware.
Delta Testing said that rather than testing blackbox systems against legacy malware like viruses, worms and malware repositories, it looked for ‘in the wild' malware sourced for the firm's network of incident response teams, security researchers and organisations that have been breached.
“By using malicious code already in the wild, yet recent enough that no signatures currently exist, products are exposed to an unpredictable environment,” said the firm in a press release, adding that this approach of using malicious malware code with packet captures was better than testing against ‘synthetic' lab-approved malware.
In the test, packet captures were stripped of customer data and replayed through the latest published software version of the solutions. The tested products were given time to analyse the packets in their virtualised environments, or to perform look-ups via threat intelligence or in the cloud.
Each product's interface was subsequently checked to see if an alert had been triggered to determine detection, and detection rates were calculated as percentages by dividing the total number of detection against the number of samples sent
Mark Thomas, commercial director, Delta Testing, said that the results prove that malware defences are always reactive to threat actors.
“It would appear that most mainstream vendors are on par when it comes to previously known attacks. However, as soon as you move towards a methodology that exposes the products to fresh malware samples taken from actual customer environments, detection rates start to show significant variations based on the technology and detection methodology used by the vendor,” he said in an email.
“While only one malware sample tested was missed by all seven products, the fact that no vendor scored 100 percent in identifying all of the samples shows that some attacks are still getting through the net. To meet the shortfall, organisations need to consider a holistic approach to security which doesn't just rely on detection but addresses threat prevention and response tactics too.”
But speaking to SCMagazineUK.com before the report was announced, Thomas said that the evolution of malware is hard to keep up with as it is ‘cat and mouse', but stressed that some of the solutions will to evolve – or will improve detection when paired with other products.
As an example, he pointed to Fidelis solutions, which he said were stronger on preventing data exfiltration, and to McAfee's– which gave a ‘fuller picture' when working with IPS.
He said that cyber-criminals are increasingly hiding their activities using encryption, and are often ‘kicking off the process' by exploiting vulnerabilities in Java or JavaScript, before moving to a second-stage of a remote access Trojan (RAT) which reports back to command and control (C&C) servers.
“If CISOs are looking at this, a co-ordinated threat prevention strategy has to be front of mind,” said Thomas, adding that they too would need to be diligent in looking at information sources and case studies before buying solutions.
“If someone is looking at this, they need to consider a more holistic approach and that is not just about detection. It shouldn't be taken in isolation is the story.”
But some of the named security vendors hit back at the study's results with Trend Micro VP of security research Rik Ferguson telling SCMagazineUK.com that FireEye's involvement as sponsor raised questions about its credibility.
“Yes, ‘FireEye-sponsored test gives FireEye the lead" is a slightly less credible and slightly more realistic headline I think,” he said in an email to SC, before later tweeting FireEye directly to say that the test “smacks of desperation.”
“To be honest, independent and unsponsored testing does not bear out the conclusions of this PR exercise at all,” he said pointing to the recent NSS Labs Breach Detection Comparison Report.
“Quite aside from testing, what really counts is the value that is returned to the user of these technologies, and the fact that we are consistently winning head-to-head deals against every other BDS (Breach Detection System) vendor is proof enough for me (of the validity of) our coverage of more than 80 different protocols and applications on all network ports, using customisable, on-site sandboxing and our deep integration with other industry leading technologies such as TippingPoint, QRadar, ArcSight and Splunk, all while achieving independently tested and verified industry-leading protection.”
Other companies in the test had not responded to our requests for comment at the time of writing. We will update this article if/when we receive this information.
Source
Delta Testing, a specialist in anti-malware testing methodologies and research, released a new report on Thursday highlighting shortcomings in the detection rates of leading players in the advanced threat solution market.
In its report, the company found ‘significant' discrepancy between the seven vendors tested, with FireEye the top-performing vendor with its NX7500 product detecting 99.14 percent of new threats. Others were not quite as efficient – Fidelis solutions (XPS Edge 200, XPS Sensor, XPS Command Post) detected just 5.17 percent of malware.
Delta Testing said that rather than testing blackbox systems against legacy malware like viruses, worms and malware repositories, it looked for ‘in the wild' malware sourced for the firm's network of incident response teams, security researchers and organisations that have been breached.
“By using malicious code already in the wild, yet recent enough that no signatures currently exist, products are exposed to an unpredictable environment,” said the firm in a press release, adding that this approach of using malicious malware code with packet captures was better than testing against ‘synthetic' lab-approved malware.
In the test, packet captures were stripped of customer data and replayed through the latest published software version of the solutions. The tested products were given time to analyse the packets in their virtualised environments, or to perform look-ups via threat intelligence or in the cloud.
Each product's interface was subsequently checked to see if an alert had been triggered to determine detection, and detection rates were calculated as percentages by dividing the total number of detection against the number of samples sent
Mark Thomas, commercial director, Delta Testing, said that the results prove that malware defences are always reactive to threat actors.
“It would appear that most mainstream vendors are on par when it comes to previously known attacks. However, as soon as you move towards a methodology that exposes the products to fresh malware samples taken from actual customer environments, detection rates start to show significant variations based on the technology and detection methodology used by the vendor,” he said in an email.
“While only one malware sample tested was missed by all seven products, the fact that no vendor scored 100 percent in identifying all of the samples shows that some attacks are still getting through the net. To meet the shortfall, organisations need to consider a holistic approach to security which doesn't just rely on detection but addresses threat prevention and response tactics too.”
But speaking to SCMagazineUK.com before the report was announced, Thomas said that the evolution of malware is hard to keep up with as it is ‘cat and mouse', but stressed that some of the solutions will to evolve – or will improve detection when paired with other products.
As an example, he pointed to Fidelis solutions, which he said were stronger on preventing data exfiltration, and to McAfee's– which gave a ‘fuller picture' when working with IPS.
He said that cyber-criminals are increasingly hiding their activities using encryption, and are often ‘kicking off the process' by exploiting vulnerabilities in Java or JavaScript, before moving to a second-stage of a remote access Trojan (RAT) which reports back to command and control (C&C) servers.
“If CISOs are looking at this, a co-ordinated threat prevention strategy has to be front of mind,” said Thomas, adding that they too would need to be diligent in looking at information sources and case studies before buying solutions.
“If someone is looking at this, they need to consider a more holistic approach and that is not just about detection. It shouldn't be taken in isolation is the story.”
But some of the named security vendors hit back at the study's results with Trend Micro VP of security research Rik Ferguson telling SCMagazineUK.com that FireEye's involvement as sponsor raised questions about its credibility.
“Yes, ‘FireEye-sponsored test gives FireEye the lead" is a slightly less credible and slightly more realistic headline I think,” he said in an email to SC, before later tweeting FireEye directly to say that the test “smacks of desperation.”
“To be honest, independent and unsponsored testing does not bear out the conclusions of this PR exercise at all,” he said pointing to the recent NSS Labs Breach Detection Comparison Report.
“Quite aside from testing, what really counts is the value that is returned to the user of these technologies, and the fact that we are consistently winning head-to-head deals against every other BDS (Breach Detection System) vendor is proof enough for me (of the validity of) our coverage of more than 80 different protocols and applications on all network ports, using customisable, on-site sandboxing and our deep integration with other industry leading technologies such as TippingPoint, QRadar, ArcSight and Splunk, all while achieving independently tested and verified industry-leading protection.”
Other companies in the test had not responded to our requests for comment at the time of writing. We will update this article if/when we receive this information.
Source
