Although a couple of months old "study", nevertheless an interesting read:
Assessing the Effectiveness of Antivirus Solutions
Assessing the Effectiveness of Antivirus Solutions
- Oct 23, 2012
- 12,527
Quite a study and a very interesting one at that.Sure brings to light some very valid concerns about the true effectiveness of security software.
Maybe Umbra isnt so paranoid after all
Maybe Umbra isnt so paranoid after all
Interesting study and good read
Paranoid when it comes to security software is a good thing,
If you are relying on one software to protect you its about the worst-thing any user can do,
A antivirus or internet security should always be run with on-demand scanners.
malware today is changing too fast for any one security software company to keep up.
Paranoid when it comes to security software is a good thing,
If you are relying on one software to protect you its about the worst-thing any user can do,
A antivirus or internet security should always be run with on-demand scanners.
malware today is changing too fast for any one security software company to keep up.
exterminator20 said:
No, the study sheds to light the lack of effectiveness of antiviruses, not OTHER security technologies.
Bo
On the very first page of the study, the following is something that caught my eye before I went through the whole document:
exterminator20 said:Quite a study and a very interesting one at that.Sure brings to light some very valid concerns about the true effectiveness of security software.
Maybe Umbra isnt so paranoid after all
hahaha ^^ i keep repeating that !
Umbra's name alone, suggest he may know a thing or two about viruses
seriously though, as Biozfear pointed out in the beginning Document, is added layers are needed to keep up with the fast, ever changing landscape of malware. Behavior Blockers/ HIPS, added along with the signatures of AV's.
One thing that i had a problem with in this document, is the numbers, the whole 5% thing. After watching so many videos's on testing of products against zero day URL's, and seeing so many products flag these links, it is hard to believe that the signature based detection of zero day malware is actually that low and or lower as they indicate. Of course i may be wrong, but these malware from sites like MDL are supposed to be within 24 hours of being detected in the wild correct?
seriously though, as Biozfear pointed out in the beginning Document, is added layers are needed to keep up with the fast, ever changing landscape of malware. Behavior Blockers/ HIPS, added along with the signatures of AV's.
One thing that i had a problem with in this document, is the numbers, the whole 5% thing. After watching so many videos's on testing of products against zero day URL's, and seeing so many products flag these links, it is hard to believe that the signature based detection of zero day malware is actually that low and or lower as they indicate. Of course i may be wrong, but these malware from sites like MDL are supposed to be within 24 hours of being detected in the wild correct?
illumination said:
5% looks like an honest number to me. I know is hard to swallow but is the truth. Expecting an antivirus company to release a definition for a new virus that was born 5 hours ago and it hasn't infected more than only a few people up to that point is not something realistic. IMO.
To take care of Zero day threats, people have to use programs that don't use definitions. Thats the only way.
Bo
Gnosis
Level 5
- Apr 26, 2011
- 2,779
Makes me glad that I use TF and Sandboxie.
Judging from what I read, and what I have learned on my own, it looks as if we have been duped into allowing AV's to be graded on a curve thus treating a score of 79% as being a straight A. Competition between AV companies' varying malware detectioin rates has become the bar, when the bar should have been set pertinent to head-to-head ratings based on malware encounters, not other vendors' best performances.
Did McAfee achieve 100% while having three times the false positives of the others?
So apparently McAfee is underrated.
In that test, the lowest lag times appeared to be 1.5 weeks for definitions. That is not very helpful, even if this only occurs in light of less than 5% of the malware out there at any given time. But like they said, the AV is still necessary, but sandboxes and BB's are VERY NECESSARY.
Illuminated and Biozfear are making me think hard. I like that.
It is hard to disagree with either one of you, but while reading your comments I pondered about government/military/intelligence agency malware, and that mindset really put the debate in perspective. I believe that the gov't backed malware out there represents most of the failure rates and signature delays, or at least it will in the near future.
Regardless, this is a fantastic topic and discussion. Cheers.
Judging from what I read, and what I have learned on my own, it looks as if we have been duped into allowing AV's to be graded on a curve thus treating a score of 79% as being a straight A. Competition between AV companies' varying malware detectioin rates has become the bar, when the bar should have been set pertinent to head-to-head ratings based on malware encounters, not other vendors' best performances.
Did McAfee achieve 100% while having three times the false positives of the others?
So apparently McAfee is underrated.
In that test, the lowest lag times appeared to be 1.5 weeks for definitions. That is not very helpful, even if this only occurs in light of less than 5% of the malware out there at any given time. But like they said, the AV is still necessary, but sandboxes and BB's are VERY NECESSARY.
Illuminated and Biozfear are making me think hard. I like that.
It is hard to disagree with either one of you, but while reading your comments I pondered about government/military/intelligence agency malware, and that mindset really put the debate in perspective. I believe that the gov't backed malware out there represents most of the failure rates and signature delays, or at least it will in the near future.
Regardless, this is a fantastic topic and discussion. Cheers.
I certainly would not mind watching one of our testers here put this to the test. Taking some of the more popular anti virus programs, and using zero day malware, calculate the percentage to see if indeed, they are only capable of maintaining a 5% mark or lower in over all detection, using the freshest malware samples they can find.
As far as needing more then just an AV now days, i completely agree.. It is why i have HIPS/BB as well as Virtualization along with my AV&FW.
As far as needing more then just an AV now days, i completely agree.. It is why i have HIPS/BB as well as Virtualization along with my AV&FW.
The % figures in the report do look valid, considering this extensive test was only done with 82 samples. It is more in line with Real World environment than loading up a VM/Junk pc with a pack of over 200 files.
If we consider the methods used:
It is a known fact that AV companies depend on several methods to keep up to date with new infections. However the draw back is the time it will take to release an update to cover the infection in the database:
Methods used are for example:
Cloud technology
Automatic data submission from the user system to it's servers
Honeypots
Dedicated Malware Hunters
Heuristics to flag suspicious files
Submission of samples manually via the solution client
Now if the solution has a HIPS/BB for example, what can be unknown in the actual database should be trigger an alert which in turn communicates with the servers (in theory). But at the same time it should technically speaking, stop the possible infection.
We then have at least one solution that uses the rollback feature. Such feature monitors the process and if "eventually" found as an infection, a scan is done (considering default settings) and it should roll back to the point prior to that specific infection. Unfortunately, such approach is more problematic than a HIPS/BB component.
There are flaws as well on HIPS/BB/Specific methods of protection for example: too much alerts and the users will simply ditch the solution. How do companies overcome such issue? Make it more silent and automated. Unfortunately that comes with a price: Incorrect automatic decisions can be made. An example is BitDefender's Auto Pilot mode. Others choose to enable such component on default settings and then tend to forget it exists or that it has flaws, for example ESET's famous default HIPS settings (automatic mode) as well as the flaw in the Interactive mode ( Interactive mode: Follow the rules. If there is no rule for such action, user will get a pop up to answer (allow, deny and learn rule). If user fails to answer the pop up dialog after a specific time, action will be allowed).
AV solutions vendors should focus more in the deployment and tune up of zero day protection first (HIPS/BB/Heuristics). This unfortunately has not been seen on at least 3 vendors in a course of 2 versions:
*ESET: Since the introduction of version 5, only on the 2nd or 3rd patch they included the one and only HIPS rule: Load system drivers. Nothing has been done in that component since, including on version 6 which has been recently released and still keep the Automatic mode on ( Follow the rules. If there is no rule for suct action, automatically allows its operation.).
*avast!: Since version 7, Behavior Shield is set with Auto Decide, yet it seems non existent. Auto Sandbox was changed from Ask to Auto Decide, yet if an infection is not known to avast!, it will allow the user to run again outside sandbox. Behavior Shield is also required for Script Shield (another protection module) but no mention is made that disabling Behavior Shield will "break" Scrip Shield.
*AVG: Since version 2012, its Behavior Blocker (Identity Protection) has improved but still causes false positives which in turn makes the actual AV solution crash occasionally upon removal. It is to note that although improvements have been made since then, it still happens.
*Panda Cloud AV: Although it contains a Behavioural analysis protection module, it doesn't fully intercept infections (prime example the system I had to recently clean: It intercepted partially 2 fake AV's but system still got infected).
If we consider the methods used:
It is a known fact that AV companies depend on several methods to keep up to date with new infections. However the draw back is the time it will take to release an update to cover the infection in the database:
Methods used are for example:
Cloud technology
Automatic data submission from the user system to it's servers
Honeypots
Dedicated Malware Hunters
Heuristics to flag suspicious files
Submission of samples manually via the solution client
Now if the solution has a HIPS/BB for example, what can be unknown in the actual database should be trigger an alert which in turn communicates with the servers (in theory). But at the same time it should technically speaking, stop the possible infection.
We then have at least one solution that uses the rollback feature. Such feature monitors the process and if "eventually" found as an infection, a scan is done (considering default settings) and it should roll back to the point prior to that specific infection. Unfortunately, such approach is more problematic than a HIPS/BB component.
There are flaws as well on HIPS/BB/Specific methods of protection for example: too much alerts and the users will simply ditch the solution. How do companies overcome such issue? Make it more silent and automated. Unfortunately that comes with a price: Incorrect automatic decisions can be made. An example is BitDefender's Auto Pilot mode. Others choose to enable such component on default settings and then tend to forget it exists or that it has flaws, for example ESET's famous default HIPS settings (automatic mode) as well as the flaw in the Interactive mode ( Interactive mode: Follow the rules. If there is no rule for such action, user will get a pop up to answer (allow, deny and learn rule). If user fails to answer the pop up dialog after a specific time, action will be allowed).
AV solutions vendors should focus more in the deployment and tune up of zero day protection first (HIPS/BB/Heuristics). This unfortunately has not been seen on at least 3 vendors in a course of 2 versions:
*ESET: Since the introduction of version 5, only on the 2nd or 3rd patch they included the one and only HIPS rule: Load system drivers. Nothing has been done in that component since, including on version 6 which has been recently released and still keep the Automatic mode on ( Follow the rules. If there is no rule for suct action, automatically allows its operation.).
*avast!: Since version 7, Behavior Shield is set with Auto Decide, yet it seems non existent. Auto Sandbox was changed from Ask to Auto Decide, yet if an infection is not known to avast!, it will allow the user to run again outside sandbox. Behavior Shield is also required for Script Shield (another protection module) but no mention is made that disabling Behavior Shield will "break" Scrip Shield.
*AVG: Since version 2012, its Behavior Blocker (Identity Protection) has improved but still causes false positives which in turn makes the actual AV solution crash occasionally upon removal. It is to note that although improvements have been made since then, it still happens.
*Panda Cloud AV: Although it contains a Behavioural analysis protection module, it doesn't fully intercept infections (prime example the system I had to recently clean: It intercepted partially 2 fake AV's but system still got infected).
Littlebits
Retired Staff
- May 3, 2011
- 3,893
We all know it is impossible for any AV to detect everything that's why you should always follow the guidelines when downloading file or visiting unknown sites. Another excellent way to keep your system protected from all of the zero-day infections is to learn how to effectively use UAC. So far I have not read any reports about vulnerabilities in UAC, however there have been many vulnerabilities found in top security products.
Awhile back there was some malware samples found that could bypass HIPS, sandboxing and virtualization. However UAC was able to stop them in their tracks. So many users turn off UAC thinking their security setup is better or ignorantly just click approve without even knowing what they did approve.
I could care less about how many samples an AV detected or didn't detect in a test. If the samples are not wide spread and are extremely remote most of us will never come in contact with them unless you like hunting them down and playing with them. And if we did come in contact we should be able to deal with them effectively.
Actually getting malware samples that are still active in the wild and widely distributed is not as easy as what many think. Most malware samples found on forums and malware download sites are usually rare malware that are not widely distributed or old samples mixed with false positives. It takes a lot of work and research to get a good malware sample collection and within 30 days or less that collection may no longer be any good.
Thanks.
Awhile back there was some malware samples found that could bypass HIPS, sandboxing and virtualization. However UAC was able to stop them in their tracks. So many users turn off UAC thinking their security setup is better or ignorantly just click approve without even knowing what they did approve.
I could care less about how many samples an AV detected or didn't detect in a test. If the samples are not wide spread and are extremely remote most of us will never come in contact with them unless you like hunting them down and playing with them. And if we did come in contact we should be able to deal with them effectively.
Actually getting malware samples that are still active in the wild and widely distributed is not as easy as what many think. Most malware samples found on forums and malware download sites are usually rare malware that are not widely distributed or old samples mixed with false positives. It takes a lot of work and research to get a good malware sample collection and within 30 days or less that collection may no longer be any good.
Thanks.
You have both brought up valid points, it was what i was seeking..
This has been some of the issue i want to bring up, as most times, i hear others talking about AV's, and how useless they are, they talk of the signatures alone. This form of standard AV has come to past, most have some form of Heuristics now as well as the other things you mentioned. Although some of these forms of Heuristics can be a little too aggressive resulting in many False Positives, the detection rates of these being implemented, have increased.
I agree 100% with this statement. AV's although not 100% effective by any means, still do a good job of detecting and keeping a lot of the malware out of our systems. Other factors such as the UAC do contribute greatly as you pointed out.
This is something i think we will see more of in the future as malware evolves. It is one of the reasons i keep my UAC set on max.
This makes me question the detection rates of these articles/documents as well. How were the samples collected, did they pertain to the geographic areas of the vendors, and the likely hood that we would even come into contact with these samples..
These rare samples, not seen so often, are another reason i question, most of these have been detected by the test of vendors we have seen on this forum, although not all, the detection rate is still much higher then they claim in this document. It is my nature to question such a lucrative environment as malware, motives, as well as methods.
Thank you both for your your insights on this topic, it has been covered in just about every aspect.
Bottom line is, we need something to compliment the AV's such as the HIPS/BB/Virtualization as well as using common sense/good habits, and this in it's self still does not guarantee 100% coverage from malware, but increases the odds of coverage.
Biozfear said:
This has been some of the issue i want to bring up, as most times, i hear others talking about AV's, and how useless they are, they talk of the signatures alone. This form of standard AV has come to past, most have some form of Heuristics now as well as the other things you mentioned. Although some of these forms of Heuristics can be a little too aggressive resulting in many False Positives, the detection rates of these being implemented, have increased.
Littlebits said:
I agree 100% with this statement. AV's although not 100% effective by any means, still do a good job of detecting and keeping a lot of the malware out of our systems. Other factors such as the UAC do contribute greatly as you pointed out.
Littlebits said:
This is something i think we will see more of in the future as malware evolves. It is one of the reasons i keep my UAC set on max.
Littlebits said:
This makes me question the detection rates of these articles/documents as well. How were the samples collected, did they pertain to the geographic areas of the vendors, and the likely hood that we would even come into contact with these samples..
Littlebits said:
These rare samples, not seen so often, are another reason i question, most of these have been detected by the test of vendors we have seen on this forum, although not all, the detection rate is still much higher then they claim in this document. It is my nature to question such a lucrative environment as malware, motives, as well as methods.
Thank you both for your your insights on this topic, it has been covered in just about every aspect.
Bottom line is, we need something to compliment the AV's such as the HIPS/BB/Virtualization as well as using common sense/good habits, and this in it's self still does not guarantee 100% coverage from malware, but increases the odds of coverage.
Gnosis
Level 5
- Apr 26, 2011
- 2,779
Agreed. You would need a second OS (maybe on external drive), that includes your custom settings, to be used to check for "false logic" against the primary OS that is thought to be infected with malware, imho.
I believe that this is due to their fears that most users cannot effectively train BB or HIPS without allowing malicious actions.
You have addressed one of my biggest potential concerns that I was not 100% informed about. The auto-sandbox is a great idea, but like you implied, if Avast cannot recognize malware while observing its behavior in the auto-sandbox, it is going to auto-allow a malicious nasty.
I am not UAC expert, but if UAC simply asks you "are you sure?!" over and over, I would absolutely not tolerate it, but I could see where a novice or careless user might need it to keep them conscious of their decisions as to avoid potential infections. How many "are you sure's" are you going to respond to before it prevents you from allowing a malicious download that an AV or BB could detect much more gracefully, while not nagging you every single time you want to download something? Sandboxie does make me think twice, because a download is quarantined and I have to manually confirm release from the sandbox to utilize said download. But I am not expert on UAC and may be missing the whole point. I have based my opinion on an assumption, so feel free to fire back. Fire away. LOL
UAC by default can cause some minor headaches to users, specially on beginner entry level. As a result, they either google or ask people how to turn it off.
That puts one of windows protection out of the window unfortunately, no matter how bullet proof it may appear to be.
That puts one of windows protection out of the window unfortunately, no matter how bullet proof it may appear to be.
Biozfear said:
The UAC for those insisting on running their computers with Admin accounts benefit form the presence of it. The UAC basically provides an application control centered on limiting the application to standard user privileges until the user allows the application to have Admin privileges.
The tasks that the UAC cover, from system wide changes to configuring parental controls, is immense.
I have seen many that will turn it off, due to the headaches, although it is among my list of must have's. It is an added layer of protection, that i do not mind taking a couple seconds to deal with. Combined with a HIPS or BB it provides the over lapping field of protection i speak of often.
Littlebits
Retired Staff
- May 3, 2011
- 3,893
UAC on default settings only prompts you when a process tries to change your system's configuration. This will block any malware since they will have to make changes to your system in order to infect it.
UAC on default settings will not bother you when running normal programs that don't try to make changes.
You also can right click on a program and click "Run as Administrator" to bypass the UAC prompts. You also can right click on a program, go to the Compatibility tab under Privilege Level and check "Run this program as Administrator" and it will always run without any prompts from UAC.
It is very simple once you learn how to use it, much more simple then trying to configure a product like Comodo, Sandboxie, Online Armor, etc.
Unlike products like Comodo, Sandboxie, Online Armor, etc. that prompt you or auto decide, block or sandbox a process that may not even be trying to change your system's configuration. I never understood why any security software would try to block a process that doesn't try to make system changes unless the security product is not able to tell.
UAC offers much better protection because it works at the OS level whereas all software only works at the software level which is much easier to bypass for advanced malware.
Enjoy!!
UAC on default settings will not bother you when running normal programs that don't try to make changes.
You also can right click on a program and click "Run as Administrator" to bypass the UAC prompts. You also can right click on a program, go to the Compatibility tab under Privilege Level and check "Run this program as Administrator" and it will always run without any prompts from UAC.
It is very simple once you learn how to use it, much more simple then trying to configure a product like Comodo, Sandboxie, Online Armor, etc.
Unlike products like Comodo, Sandboxie, Online Armor, etc. that prompt you or auto decide, block or sandbox a process that may not even be trying to change your system's configuration. I never understood why any security software would try to block a process that doesn't try to make system changes unless the security product is not able to tell.
UAC offers much better protection because it works at the OS level whereas all software only works at the software level which is much easier to bypass for advanced malware.
Enjoy!!
Littlebits
Retired Staff
- May 3, 2011
- 3,893
Umbra Corp. said:UAC at max is almost equal to a built-in HIPS
UAC on default settings is better than HIPS. (OS Level)
Max settings just give you a lot of extra prompts for harmless processes.
Rule #1 if a malicious process can not change your system's configuration then it can not infect your system. In other words the malicious file may still be on your system but can not run to do its dirty work. So it is just a dormant infected file.
Thanks.
Similar threads
