Forget signatures for malware detection. SparkCognition says AI is 99% effective

Exterminator

Community Manager
Thread author
Verified
Staff Member
Well-known
Oct 23, 2012
12,527
The notion of detecting malware by looking for malicious file signatures is obsolete. Depending on which source is cited, anywhere from 300,000 to one million new malware files are identified every day.

Kaspersky Lab says it finds 323,000 files daily, AV-TEST claims to discover more than 390,000 new malicious programs every day, and Symantec says it uncovers almost a million new threats per day. No matter how you count it, that’s a lot of malicious software being unleased into the wild day after day.

Most of these “new” files are actually clones of each other, with perhaps just one character that is different. Given that every digital file has a unique signature, this one character difference means that two otherwise identical files still have different signatures.

Malware researchers continuously scour the Internet to look for malicious files. They use honeypots and other techniques to attract the files. When they come across a new sample they compute an MD5 and/or SHA256 hash and add it to their database of signatures. Anti-virus (AV) and anti-malware (AM) products that get installed on endpoint computers compare the hashes of all the files on an endpoint to the hashes in the signature database. If there is a match, the AV/AM software generates an alert about the malicious file.

This process was rather effective until a few years ago, when the amount of malware being produced daily skyrocketed. Now it is practically impossible for any research team to keep up with the volume of malware variants and to generate and distribute the necessary hashes to detect the malware in real-time. The efficacy of AV/AM products that rely solely on signatures is dropping precipitously.

Luckily an alternative is rising in its place: malware detection that uses artificial intelligence (AI) to identify malicious files by characteristics rather than by signatures. Products based on AI are said to be more effective in detecting malware in all its mutations, quickly and with few false positives.

SparkCognition, an Austin-based AI company, has a new entrant in the malware detection marketplace. DeepArmor Enterprise is a machine learning-based malware detection engine. SparkCognition has trained its algorithms on hundreds of thousands of clean files and malicious files to learn the characteristics of a file that is benign versus a file that is malicious. The characteristics are an indicator of what the actual intent is of those files. When the system reads a new file, it’s able to read those characteristics, make a determination and provide a confidence score on whether the file is malicious or benign.

SparkCognition doesn’t define the characteristics for its malware detection engine; that would be tantamount to giving it signatures. Instead, the machine learning model leverages an ensemble of algorithms that pore over a thousand or more characteristics per file to learn how to classify the file as clean or benign.

DeepArmor Enterprise is an endpoint security product. It utilizes a kernel level driver that works in two ways to try to stop threats at the endpoint. One, it monitors all new file activity on the system and scans all new executable files to determine whether they are malicious or benign. Two, it can pause the execution of files for a millisecond in order to run them through machine learning models based in the cloud, and come back with a determination of whether they are malicious or benign. If a file is malicious, then DeepArmor can stop its execution and block the file as well as auto-quarantine it. Thus, DeepArmor provides true protection and not just alerts.

DeepArmor Enterprise uses a small, low-profile endpoint agent on Windows systems to monitor the file activity and freeze new files so they can be checked. The detection engine is based in the cloud, so any new file that the endpoint agent hasn’t seen before is sent up to the cloud-based threat detection engine to be scanned and give a prediction back as to whether it is malicious or benign.

The kernel level driver is one of the first drivers executed upon Windows being launched, meaning it starts running before any other applications are started up within the start sequence. This makes it difficult for hackers to get something launched before DeepArmor protection is executed. Once DeepArmor is running, it monitors all the execution activity on the system.

In addition to the Windows version of DeepArmor Enterprise, there is an Android version and the company says it will eventually offer a Linux version. SparkCognition says the goal is to provide unified protection for clients, servers, mobile devices and IoT devices. The vendor understands that customers don’t want to have to use multiple solutions to protect a range of endpoints.

SparkCognition has a multi-directional strategy for IoT devices. The current Windows version is able to protect devices based on Windows 10 IoT Core. The endpoint agent can run in a headless mode; it is specifically designed to have no user interface, so the agent is very small. It is for devices such as point-of-sale and other types of devices that have low power system-on-chip circuits. DeepArmor can protect these devices without putting overhead on them.

The Android version includes Android Things, so SparkCognition says that DeepArmor can be set up to protect virtually any IoT device running Android or Android Things. Beyond Windows and Android, the next move will be on to Linux, which is the backbone for a number of IoT devices. All the IoT versions of DeepArmor will work headless within the devices.

One distinction SparkCognition has from other AI-based malware detection systems is the machine learning models are not put on the endpoints. The machine learning processing is done in the cloud which makes this solution a good fit for IoT protection.

As for the efficacy of DeepArmor, SparkCognition claims its models are 99% effective against malware and that there is less than a 1% false positive rate. The threat detection engine is constantly adapting and learning about new threats. The vendor continuously tests its system and updates its training sets if there are new pieces of clean files or new approaches to malware.

A second way that SparkCognition is taking this product to market is through a micro service, which includes a software developer’s kit and SparkCognition’s cloud-based threat detection engine. This version of the product allows other security vendors to incorporate SparkCognition’s machine learning technology into their security stack.
 

Parsh

Level 25
Verified
Honorary Member
Top Poster
Malware Hunter
Well-known
Dec 27, 2016
1,480
Most of the upcoming 'AI' products first curse the old methods used by AV products and their declining efficacy and then present their next-gen features to give the feel of Gods are coming!
AVs have risen well above the traditional signature-based outlook, though there will always be a rift between them and the intelligently crafted new malware rising progressively. They probably won't catch up with the best of threats or zero days.

The main difference is the training aspect.
Coming to DeepArmor, I wonder how they make the group of algorithms to decide on the benign and malicious characteristics of tons of files.
SparkCognition doesn’t define the characteristics for its malware detection engine; that would be tantamount to giving it signatures. Instead, the machine learning model leverages an ensemble of algorithms that pore over a thousand or more characteristics per file to learn how to classify the file as clean or benign.
Any wrong classification done for any sample, and the equilibrium between the good and the bad space gets disturbed further. That can affect the judgement for the remaining files being trained too.
They must be using some very careful approach for the same.

The numbers are only good on paper. False alerts don't count for less than 1% but the true positives in DA's detection I've seen is quite impressive for their work.
 
Last edited:

Parsh

Level 25
Verified
Honorary Member
Top Poster
Malware Hunter
Well-known
Dec 27, 2016
1,480
There's a simple difference.
The anti-malware products are deterministic. They react to a type of threat in the same way always (unless technically - their technology or model change over time). They don't swing on decisions.
Man is not deterministic. His knowledge, needs, emotions, confusion, confidence, patience, experiences all determine his action over the current judgement to be taken and that might differ the next time he has to act.
Man O Man!
 

Xsjx

Level 13
Verified
Feb 21, 2017
613
The notion of detecting malware by looking for malicious file signatures is obsolete. Depending on which source is cited, anywhere from 300,000 to one million new malware files are identified every day.

Kaspersky Lab says it finds 323,000 files daily, AV-TEST claims to discover more than 390,000 new malicious programs every day, and Symantec says it uncovers almost a million new threats per day. No matter how you count it, that’s a lot of malicious software being unleased into the wild day after day.

Most of these “new” files are actually clones of each other, with perhaps just one character that is different. Given that every digital file has a unique signature, this one character difference means that two otherwise identical files still have different signatures.
Anddd Avira discovers more than 3 million a day with sigs+ cloud detects some more ;)

So why they never say anything about avira?
 
Last edited:

Parsh

Level 25
Verified
Honorary Member
Top Poster
Malware Hunter
Well-known
Dec 27, 2016
1,480
I said it wrong discovers^
Edited it ;)
I find this at Avira :)
IMG_20170423_123906.jpg
 

_CyberGhosT_

Level 53
Verified
Honorary Member
Top Poster
Content Creator
Well-known
Aug 2, 2015
4,286
DeepArmor produces lots and lots of FPs.
You have said this many times, and again, I will say:
It does at first as it is learning, but as with VoodooShield, the FP's go away.
As an actual user I can tell you I don't hear anything FP related from it these days.
I did for the first "approximately" 90 days maybe and it was intermitten not constant FP's.
I compare it to the FP's of VS because it was a similar pattern, and duration but as we know they
are two very different software. Just wanted to make myself very clear ;)
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top