FP galore AKA AV Comparatives March AV Test results

mekelek

Level 28
Thread author
Verified
Well-known
Feb 24, 2017
1,661
AV-Comparatives - Independent Tests of Anti-Virus Software - Real World Protection Test Overview

drVEMP.png


so ehm, who wants to bet with me that 90% of the samples were PUP?(nevermind looking at the ESET results)
327 FP with Panda, I think FSecure has a contender! :ROFLMAO:

I can see that 166 from Trend Micro, those are all "Suspicious unknown program blocked".
 

mekelek

Level 28
Thread author
Verified
Well-known
Feb 24, 2017
1,661
Bitdefender - 99.99 % :rolleyes:

Kaspersky - 99.98% :rolleyes:

Why can't both of them rounded to 100%?!
it's bad enough 80% of the test results are 100%, imagine if they started rounding.
I wish the malware hub would have results like these, all green on all pages with these products
but oh hey, the general consumer has to be tricked somehow to pay for something.(and pick the wrong one)
 

ZeroDay

Level 30
Verified
Top Poster
Well-known
Aug 17, 2013
1,905
100% for built in protection is pretty dam good I have to say.
Me too. You can't beat that for free and built in. At least when people's 3 month trials of Norton run out they'll still have solid protection.
 
  • Like
Reactions: Electr0n

mekelek

Level 28
Thread author
Verified
Well-known
Feb 24, 2017
1,661
It looks like Avira did the best overall. Not 100% but close and only 1 FP. WD is staying consistent protection wise.
bare in mind this test is a joke and Avira scores nowhere near decent, they're still living in stone age with only signatures.
tho those signatures are fast, they're not competitive against other fast signatures and dynamic modules
 

ZeroDay

Level 30
Verified
Top Poster
Well-known
Aug 17, 2013
1,905
bare in mind this test is a joke and Avira scores nowhere near decent, they're still living in stone age with only signatures.
tho those signatures are fast, they're not competitive against other fast signatures and dynamic modules
I like Avira but I would only use it alongside some solid zero day protection and even then only for it's signatures. Last Time I ran it with CF my system was very responsive and they played well together. Avira still have some of the very best signatures in the world. So, Avira teamed up with say CF or VS would bve a great setup. I find on a lot of AV-Comparatives tests and AV-TEST's that the HUB semms to actually support some of their findings.

You can't compare ANY signatures to dynamic modules. As I said - I'd happily use Aviras alongside CF and it would be an extremely good combo. I'm currently running WD with CF and it's the lightest setup I've run for a while.
 

ZeroDay

Level 30
Verified
Top Poster
Well-known
Aug 17, 2013
1,905
I also find AV-TEST results to be pretty much in line with the hub's overall results. On another machine I have WD+CF+AppGuard just to see how they run together and they run great. I can't think of 1 single security suite with multiple modules that would even match that setup let alone beat it.
 

ZeroDay

Level 30
Verified
Top Poster
Well-known
Aug 17, 2013
1,905
bare in mind this test is a joke and Avira scores nowhere near decent, they're still living in stone age with only signatures.
tho those signatures are fast, they're not competitive against other fast signatures and dynamic modules
I seem to Remember Rezjor running an Avira Pro AV test and it did extremely well, especially with it's cloud detections and that was running the files not just static scans.
 

mekelek

Level 28
Thread author
Verified
Well-known
Feb 24, 2017
1,661
I seem to Remember Rezjor running an Avira Pro AV test and it did extremely well, especially with it's cloud detections and that was running the files not just static scans.
when a new pack gets released i will toy around with it a bit and see if it improved at all
currently trying to get my hands on Cyclane but these damn resellers only accept CC
 
D

Deleted member 65228

You can't compare ANY signatures to dynamic modules
Generic signatures can be applied dynamically as well, and it is a positive thing because it can be beneficial for matching a detection prior to the sample being able to attempt to carry out any malicious activity on the system, which may or may not have been caught by a completely dynamic component. I know what you meant though (and that your post wasn't even referring to memory scanning, but it's a perfect opening for me to talk about the topic so hopefully you won't mind), and please don't be alarmed even though this post is started with a quote from you, I too agree that dynamic components are mandatory for good zero-day protection.

For example, packing is a common technique which is used by criminals to surpass static signature detection performed by traditional real-time and on-demand scanners from security packages (e.g. "Anti-Virus"/"Anti-Malware" products). A security package may attempt to unpack a sample if it has an appropriate unpacking algorithm after attempting to identify the packer being used on the sample; this is not always a successful operation, but if the operation is successful, then the generic signatures may become effective once again for the static-based scan operation for the targeted sample. For dynamically applying signatures, a memory scanner could be appropriate - this can be accompanied by emulation support in some scenarios. The idea behind it focuses on scanning the memory of the newly started process with generic signatures which were not originally effective whilst being applied to the static binary due to the binary being packed (prior to execution of the sample).

If we have a generic signature named XYZ for example purposes which can match a detection when used on a sample named XYZ.exe, and then we were to pack XYZ.exe with UPX for example purposes, the signature used for the XYZ detection may no longer be effective when used with the packed version of XYZ.exe. The reason for this is because the section of code which was identified during the reverse-engineering stage and was a good candidate for the signature creation may have since become encrypted during the packing procedure - when the packed version of sample is ran, the loader for the packer will be executed and the portion of code which would have triggered the generic signature named XYZ prior to the packing procedure may now become decrypted in-memory, and therefore if the memory scanner were to do a re-scan using the generic signatures (this time dynamically as opposed to statically), the XYZ signature may now be capable of matching a detection.

It can be very difficult to develop a good memory scanner depending on the requirements. A simple memory scanner could literally scan the process's memory after it has been executed, although the timing of the memory scan and the timing of the decryption procedures can cause the results of the memory scanning procedure to become affected by an extraordinary amount. Different scenarios can cause results to be affected at different levels. To elaborate my point and for example purposes, if the decryption routine is held off for 10-20 seconds and the process decides to go into a sleeping state during that time, but at the same time the security package decides to go ahead and activate its memory scanner, then it may be completely ineffective... since if the packer were to function by encrypting parts of code present within the .text section of the Portable Executable (PE) and is setup to decrypt those encrypted parts of code when the PE is executed (e.g. the packer may alter the Entry-Point of the PE to redirect execution flow once the PE is started up in-memory and its main thread is resumed, providing the process with the green-light to start executing code), and the security package performs a memory scan prior to the decryption routine carrying out its real operation and continuing the execution flow of the program afterwards, then the bytes being used for the signature which would have been matched prior to the packing procedure being applied may still be in an encrypted-format in-memory at the time of the memory scan.

Additions accompanying memory scanning functionality such as emulation or virtualization can be extremely beneficial since it provides the security package a good opportunity to safely perform dynamic forensics on the sample, but even with such additions, the above example regarding the holding off on the decryption routine for X amount of time can still cause such techniques to become ineffective. Features like emulation or virtualization being primarily utilised for quick forensic analysis prior to main on-host execution being granted will typically have a maximum time-frame otherwise waiting times for the user can last a long time, which is a disadvantage.

A memory scanner may also be triggered at random intervals throughout the execution of an "untrusted" program (or an alike mechanism) in some situations depending on the security package and how it works. An example of a mechanism like this would be interception of virtual memory operations on-going within the newly started process and activation of memory scanning techniques when a virtual memory operation which involved large memory manipulation (locally) is performed.

I have seen security packages rely on memory scanning for "Anti-Exploit capabilities" in the past.

One idea behind using signatures for exploit mitigation purposes (dynamically) is to create signatures for known exploits (e.g. using the disassembly of a routine in a loader which deploys exploitation of a vulnerability) and take action if the memory scanner gets a hit on the signature after being triggered (e.g. by a specifically filtered operation in the process locally). Such mechanisms can be implemented on a case-by-case basis for specific software packages and vulnerabilities.

If we have a program called ZeroDay.exe which when exploited by a specific vulnerability will cause the process executing the code for the ZeroDay.exe image to invoke specific APIs, then those specific APIs can be watched and a memory scanner mechanism can be triggered when the watch for the invocation of those APIs become triggered (filtering can be applied to make it more accurate - such as parameter data). In a specific scenario like this, the thread responsible for invoking those specific APIs upon the exploitation attempt will be temporarily hijacked (the EIP/RIP could have been redirected to a callback routine related to the interception of those specific APIs), potentially providing the memory scanner which has just been triggered a green-light to scan the process's memory for specific signatures before the exploitation phase has a chance to escalate.

Mechanisms like these do not always provide only positive results to the table, negatives can be brought along as well. They aren't really applicable to be thrown into a security package and utilised 24/7, they are sensitive and delicate and would need to be implemented and activated when appropriate only - however whether it would be responsible and appropriate for such mechanisms to be activated depend on the scenario (e.g. the type of vulnerability and how exploitation of the vulnerability works, what the target/s for exploitation is/are and how it affects the target, etc.) and would need to be agreed on/discussed properly.

It would be inaccurate of me to say that memory scanning is a full-proof technique against packing techniques; this is simply not true, but it does not mean that the technique is useless either. Some packers are stronger than others and not all packers will work like in the examples above (e.g. a portion of code used for signature creation may only be decrypted in-memory after several other operations which were malicious have already occurred on the system, and in such a scenario, a lot of damage could have already been done).
 

ZeroDay

Level 30
Verified
Top Poster
Well-known
Aug 17, 2013
1,905
Generic signatures can be applied dynamically as well, and it is a positive thing because it can be beneficial for matching a detection prior to the sample being able to attempt to carry out any malicious activity on the system, which may or may not have been caught by a completely dynamic component. I know what you meant though (and that your post wasn't even referring to memory scanning, but it's a perfect opening for me to talk about the topic so hopefully you won't mind), and please don't be alarmed even though this post is started with a quote from you, I too agree that dynamic components are mandatory for good zero-day protection.

For example, packing is a common technique which is used by criminals to surpass static signature detection performed by traditional real-time and on-demand scanners from security packages (e.g. "Anti-Virus"/"Anti-Malware" products). A security package may attempt to unpack a sample if it has an appropriate unpacking algorithm after attempting to identify the packer being used on the sample; this is not always a successful operation, but if the operation is successful, then the generic signatures may become effective once again for the static-based scan operation for the targeted sample. For dynamically applying signatures, a memory scanner could be appropriate - this can be accompanied by emulation support in some scenarios. The idea behind it focuses on scanning the memory of the newly started process with generic signatures which were not originally effective whilst being applied to the static binary due to the binary being packed (prior to execution of the sample).

If we have a generic signature named XYZ for example purposes which can match a detection when used on a sample named XYZ.exe, and then we were to pack XYZ.exe with UPX for example purposes, the signature used for the XYZ detection may no longer be effective when used with the packed version of XYZ.exe. The reason for this is because the section of code which was identified during the reverse-engineering stage and was a good candidate for the signature creation may have since become encrypted during the packing procedure - when the packed version of sample is ran, the loader for the packer will be executed and the portion of code which would have triggered the generic signature named XYZ prior to the packing procedure may now become decrypted in-memory, and therefore if the memory scanner were to do a re-scan using the generic signatures (this time dynamically as opposed to statically), the XYZ signature may now be capable of matching a detection.

It can be very difficult to develop a good memory scanner depending on the requirements. A simple memory scanner could literally scan the process's memory after it has been executed, although the timing of the memory scan and the timing of the decryption procedures can cause the results of the memory scanning procedure to become affected by an extraordinary amount. Different scenarios can cause results to be affected at different levels. To elaborate my point and for example purposes, if the decryption routine is held off for 10-20 seconds and the process decides to go into a sleeping state during that time, but at the same time the security package decides to go ahead and activate its memory scanner, then it may be completely ineffective... since if the packer were to function by encrypting parts of code present within the .text section of the Portable Executable (PE) and is setup to decrypt those encrypted parts of code when the PE is executed (e.g. the packer may alter the Entry-Point of the PE to redirect execution flow once the PE is started up in-memory and its main thread is resumed, providing the process with the green-light to start executing code), and the security package performs a memory scan prior to the decryption routine carrying out its real operation and continuing the execution flow of the program afterwards, then the bytes being used for the signature which would have been matched prior to the packing procedure being applied may still be in an encrypted-format in-memory at the time of the memory scan.

Additions accompanying memory scanning functionality such as emulation or virtualization can be extremely beneficial since it provides the security package a good opportunity to safely perform dynamic forensics on the sample, but even with such additions, the above example regarding the holding off on the decryption routine for X amount of time can still cause such techniques to become ineffective. Features like emulation or virtualization being primarily utilised for quick forensic analysis prior to main on-host execution being granted will typically have a maximum time-frame otherwise waiting times for the user can last a long time, which is a disadvantage.

A memory scanner may also be triggered at random intervals throughout the execution of an "untrusted" program (or an alike mechanism) in some situations depending on the security package and how it works. An example of a mechanism like this would be interception of virtual memory operations on-going within the newly started process and activation of memory scanning techniques when a virtual memory operation which involved large memory manipulation (locally) is performed.

I have seen security packages rely on memory scanning for "Anti-Exploit capabilities" in the past.

One idea behind using signatures for exploit mitigation purposes (dynamically) is to create signatures for known exploits (e.g. using the disassembly of a routine in a loader which deploys exploitation of a vulnerability) and take action if the memory scanner gets a hit on the signature after being triggered (e.g. by a specifically filtered operation in the process locally). Such mechanisms can be implemented on a case-by-case basis for specific software packages and vulnerabilities.

If we have a program called ZeroDay.exe which when exploited by a specific vulnerability will cause the process executing the code for the ZeroDay.exe image to invoke specific APIs, then those specific APIs can be watched and a memory scanner mechanism can be triggered when the watch for the invocation of those APIs become triggered (filtering can be applied to make it more accurate - such as parameter data). In a specific scenario like this, the thread responsible for invoking those specific APIs upon the exploitation attempt will be temporarily hijacked (the EIP/RIP could have been redirected to a callback routine related to the interception of those specific APIs), potentially providing the memory scanner which has just been triggered a green-light to scan the process's memory for specific signatures before the exploitation phase has a chance to escalate.

Mechanisms like these do not always provide only positive results to the table, negatives can be brought along as well. They aren't really applicable to be thrown into a security package and utilised 24/7, they are sensitive and delicate and would need to be implemented and activated when appropriate only - however whether it would be responsible and appropriate for such mechanisms to be activated depend on the scenario (e.g. the type of vulnerability and how exploitation of the vulnerability works, what the target/s for exploitation is/are and how it affects the target, etc.) and would need to be agreed on/discussed properly.

It would be inaccurate of me to say that memory scanning is a full-proof technique against packing techniques; this is simply not true, but it does not mean that the technique is useless either. Some packers are stronger than others and not all packers will work like in the examples above (e.g. a portion of code used for signature creation may only be decrypted in-memory after several other operations which were malicious have already occurred on the system, and in such a scenario, a lot of damage could have already been done).
I simply meant one can not compare static detection to dynamic detection alone. I've worked and carried out work for a number of security companies Aviare included. Although that was a long time ago now at least in the tech world. I then went into government employment in the UK and now work in the private sector. Your above post is a fantastic post. However - What I meant is that when Mekelek mentioned Avira being weak I simply pointed out that, when Avira's static signatures are teamed up with some solid zero day protection the person using said combination would be very well protected. I'm honestly surprised that Avira's signatures and indeed their engine is not used by more products that use dual-engine protection. I try to keep my replies short and sweet on forums because once people discover your level of expertise the demand for knowledge increases and instead of just enjoying a security forum you become people's personal Google.

The difficulty in creating a memory scanner, or indeed any scanner only depends on the person or indeed teams expertise and resources. If MS wasn't restricted by the same rules as the rest of the AV industry they could build something truly amazing security product wise. But then they'd get sued by every other security company for having a monopoly as they have full and complete access to parts of the OS that other security vendors simply don't have, or at least legally can't reverse engineer.

Again - Thank you for a great reply.
 

simmerskool

Level 37
Verified
Top Poster
Well-known
Apr 16, 2017
2,627
I'd love to get my hands on Cyclane too.

I was curious enough to buy a cylance license Been running it for several months. I like it. Very light and trouble-free, system impact, if any, is not noticeable. I can't really comment on detection, as I don't intentionally throw malware at it. I do not get false positives either. Plays nice with other security apps, but not all. It did block Panda Cloud Cleaner from running, so you might consider that a false positive, but I did not, I saw it as cylance protecting itself from getting "corrupted" by incompatible software. Not mean as a negative comment on Panda, just my observation.
 

About us

  • MalwareTips is a community-driven platform providing the latest information and resources on malware and cyber threats. Our team of experienced professionals and passionate volunteers work to keep the internet safe and secure. We provide accurate, up-to-date information and strive to build a strong and supportive community dedicated to cybersecurity.

User Menu

Follow us

Follow us on Facebook or Twitter to know first about the latest cybersecurity incidents and malware threats.

Top