When i test something, i test every aspect of it. I watch for how modules work together and react to threats as well as usability for daily driving, bugs, system impacts, frequency of program updates, issues that arise from those, ect,ect.
- Feb 17, 2018
- 870
- Feb 17, 2018
- 870
Have not made it that far yet, i start with usability and bug issues first, because it will not matter how strong the suite is, if it renders productivity.
- Feb 24, 2017
- 1,661
before i switched back to KIS
http://dl.surfright.nl/hmpalert-test.exe
take this on a spin, every single exploit attempt gets blocked by KIS for example
GData stays silent during the whole test
You shouldn't rely on simulation only to test exploit mitigation capabilities in my opinion, I don't think it is a realistic approach unless you know the internals of the exploit mitigation component you're testing - which would require cooperation with the vendor or (potentially illegal and violating the Terms of Service) reverse engineering the component.
I advise you go on a hunt for samples relying on an in-house zero-day exploit or at-least use samples which are exploiting a vulnerability (which is yet to be patched) but is recent. That would provide you with more accurate test results, assuming that if you did this you understood what vulnerability was being exploited and how it was possible to focus more on the strengths and weak-points of the exploit mitigation component.
For example, such research may show results that product X from vendor X has good prevention ratio of Heap Spray exploitation and didn't manage to block any of the Buffer Overflow exploitation samples. Alternatively, product X from vendor X managed to prevent UAC exploit CVE-............ while it failed to prevent CVE-.............. and so on.
Moving on from that though, an exploit mitigation component might focus on preventing exploitation attacks which have been already been discovered and disclosed as opposed to identification of potential attempts generally speaking. This would be an example of why HitmanPro.Alert has a lot of FPs compared to exploit mitigation from some other vendors.
Not all vendors take the same approach and have an identical exploit mitigation component. The simulator test from Surfright (now SOPHOS) won't be applicable for all exploit mitigation components depending on what makes them tick and their goals (not to mention that it could actually be white-listed internally due to where it comes from). In the real world, the results may be the complete opposite to those from a simulation test. From my experience with exploit mitigation development and component testing, it just does not paint a picture of a "real world scenario" illustration of the exploit mitigation capabilities depending on how the component was designed to work.
Just some food for thought.
I advise you go on a hunt for samples relying on an in-house zero-day exploit or at-least use samples which are exploiting a vulnerability (which is yet to be patched) but is recent. That would provide you with more accurate test results, assuming that if you did this you understood what vulnerability was being exploited and how it was possible to focus more on the strengths and weak-points of the exploit mitigation component.
For example, such research may show results that product X from vendor X has good prevention ratio of Heap Spray exploitation and didn't manage to block any of the Buffer Overflow exploitation samples. Alternatively, product X from vendor X managed to prevent UAC exploit CVE-............ while it failed to prevent CVE-.............. and so on.
Moving on from that though, an exploit mitigation component might focus on preventing exploitation attacks which have been already been discovered and disclosed as opposed to identification of potential attempts generally speaking. This would be an example of why HitmanPro.Alert has a lot of FPs compared to exploit mitigation from some other vendors.
Not all vendors take the same approach and have an identical exploit mitigation component. The simulator test from Surfright (now SOPHOS) won't be applicable for all exploit mitigation components depending on what makes them tick and their goals (not to mention that it could actually be white-listed internally due to where it comes from). In the real world, the results may be the complete opposite to those from a simulation test. From my experience with exploit mitigation development and component testing, it just does not paint a picture of a "real world scenario" illustration of the exploit mitigation capabilities depending on how the component was designed to work.
Just some food for thought.
Last edited by a moderator:
- Feb 24, 2017
- 1,661
I had asked Lockdown about this prolly a year ago, and he said that there is probably no proper way of testing exploit mitigation.
for me, having one AV react to something and an other not tells a whole lot, sure it's barely considered a proper test, but still.
You can test products however you'd like to, the nice thing about testing security software is that there's no golden rule-book like with the laws of physics which declare whether you're right or wrong; by no means am I saying you are wrong for what you're doing if that is the impression my post stank of.
People I know used to tell me that Malwarebytes Anti-Exploit wasn't very good and that they'd had never seen it react in the real-world. I took their word for it for a long time... Until I investigated it myself. My own findings after concluding what made it trigger and how it was designed to work was that in the real world, it can actually be a very beneficial addition.
Exploit prevention components can be very tricky to test because even if you conducted a test with 100 samples exploiting 100 different vulnerabilities, and all of which were zero-day (that'd be phenomenally impressive to find 100 zero-day vulnerabilities being exploited in the world at once like that, very unrealistic), and all exploitation attempts were deployed successfully... A week later that same component might stop a different exploit attack.
In my opinion, the only way to really understand if the exploit prevention component is going to be beneficial or not in the real-world is to know the internals of it. This way you'd be able to think about how it was designed to work and how it does work and understand the benefits and downsides of it. Although, such is not applicable to most and learning about how such a component works internally can be non-applicable if you do not work for the vendor and they do not want to share the details with you (not to mention that through reverse engineering you can be breaking the law and violating Terms of Service depending on the situation).
As an example of my above point, let's switch over to Windows Defender Exploit Guard (WDEG) for a minute. While we haven't reverse-engineered it or asked Microsoft for additional details, we know that it allows you to enforce policies such as Address Space Layout Randomization should the software package not have it enabled on the binaries since such is exposed as an individual feature; this can cause the targeted software package to break depending on how it was built internally (e.g. hard-coding which will no longer function when ASLR support relocates images in-memory) though.
Now, with WDEG and the ASLR enforcement policy in mind, we can investigate ASLR and then understand that in the real world it can be beneficial because it can cause exploit attacks which are not very flexible and are relying on hard-coded addresses/offsets to be unsuccessful in deployment. We'll understand that while the enforcement feature can be beneficial, it certainly won't be full-proof and that when enforced on specific software packages, the software may not function correctly if it was not built to support ASLR (e.g. intentionally disabled).
It becomes more difficult when a vendor is focusing on preventing exploitation without disclosing details of how the component works. Believe it or not, some vendors may market their product as having "exploit mitigation" and while you as a customer may be thinking in your mind, "It's great they have a nice exploit mitigation component to help stop zero-day exploit attacks!", it may actually be based on signature scanning.
That's a small example, but I hope it'll elaborate on what I was trying to say in this post.
People I know used to tell me that Malwarebytes Anti-Exploit wasn't very good and that they'd had never seen it react in the real-world. I took their word for it for a long time... Until I investigated it myself. My own findings after concluding what made it trigger and how it was designed to work was that in the real world, it can actually be a very beneficial addition.
Exploit prevention components can be very tricky to test because even if you conducted a test with 100 samples exploiting 100 different vulnerabilities, and all of which were zero-day (that'd be phenomenally impressive to find 100 zero-day vulnerabilities being exploited in the world at once like that, very unrealistic), and all exploitation attempts were deployed successfully... A week later that same component might stop a different exploit attack.
In my opinion, the only way to really understand if the exploit prevention component is going to be beneficial or not in the real-world is to know the internals of it. This way you'd be able to think about how it was designed to work and how it does work and understand the benefits and downsides of it. Although, such is not applicable to most and learning about how such a component works internally can be non-applicable if you do not work for the vendor and they do not want to share the details with you (not to mention that through reverse engineering you can be breaking the law and violating Terms of Service depending on the situation).
As an example of my above point, let's switch over to Windows Defender Exploit Guard (WDEG) for a minute. While we haven't reverse-engineered it or asked Microsoft for additional details, we know that it allows you to enforce policies such as Address Space Layout Randomization should the software package not have it enabled on the binaries since such is exposed as an individual feature; this can cause the targeted software package to break depending on how it was built internally (e.g. hard-coding which will no longer function when ASLR support relocates images in-memory) though.
Now, with WDEG and the ASLR enforcement policy in mind, we can investigate ASLR and then understand that in the real world it can be beneficial because it can cause exploit attacks which are not very flexible and are relying on hard-coded addresses/offsets to be unsuccessful in deployment. We'll understand that while the enforcement feature can be beneficial, it certainly won't be full-proof and that when enforced on specific software packages, the software may not function correctly if it was not built to support ASLR (e.g. intentionally disabled).
It becomes more difficult when a vendor is focusing on preventing exploitation without disclosing details of how the component works. Believe it or not, some vendors may market their product as having "exploit mitigation" and while you as a customer may be thinking in your mind, "It's great they have a nice exploit mitigation component to help stop zero-day exploit attacks!", it may actually be based on signature scanning.
That's a small example, but I hope it'll elaborate on what I was trying to say in this post.
Exactly what i meant by testing all Modules and how they are designed to work together, the underlined being an important aspect.
An old saying i had back in my heavy testing days... Do not wrap the product around the test "one size fits all" but rather wrap the test around the product. Meaning to test the product based upon how it is designed to work to see the products full capabilities.
- Feb 17, 2018
- 870
Not true, i tested it with KIS and also very silent
- Feb 24, 2017
- 1,661
yes but the exploits didn't go through and you got an access denied on every one of them.Not true, i tested it with KIS and also very silent
with GData, calculator got opened with every exploit.
- Feb 17, 2018
- 870
Well the calculator happy popped up here with KIS enabled, same for Norton no sign on interuption.
- Jan 25, 2018
- 155
i tried this test with bitdefender and it blocked them all. bitdefender show a notification (exploit attack blocked).
- Feb 24, 2017
- 1,661
I apologize, I re-run the test and turns out the target process i was using was crashing.
tho, it only works with KIS if it's done with the default process, aka the hmp test exe, if I select for example adobe acrobat reader, calculator doesnt show up
- Apr 28, 2015
- 8,664
Hum... hmpalert-test.exe is completely trusted in KSN, so it will run without any limitations/restrictions...
Proably You should put it at least, in low restriction, and test in Interactive Mode...
Anyway, probably We are hijacking @PathFinder's config thread
hmpalert-test.exe is completely trusted in KSN, so it will run without any limitations/restrictions...Proably You should put it at least, in low restriction, and test in Interactive Mode...
Anyway, probably We are hijacking @PathFinder's config thread
Last edited:
It's all good, as it allowed a few members to hash out experiences as well as hopefully learning something along the way.Anyway, probably We are hijacking @PathFinder's config thread
- Feb 24, 2017
- 1,661
i set it to low restricted, same resultsHum...
Proably You should put it at least, in low restriction, and test in Interactive Mode...
Anyway, probably We are hijacking @PathFinder's config thread
- Apr 28, 2015
- 8,664
Sincerely I don't know why that test tool does not trigger System Watcher (PDM.Exploit) and/or Heur.Exploit (File AV) Kaspersky technology.
There is not much info (or none) about their implementations, but the truth is that almost every week I test samples in the Hub, KTS detects by System Watcher (PDM.Exploit) some on demand undetected samples (usually MsOffice docs, and sometimes other types of files) or on demand by Heur.Exploit, so I don't mind if that exploit tool designed for HMP.A does not trigger any Kaspersky Anti-Exploit modules, since against real malware Kaspersky Anti-Exploit usually is effective, as demonstrated in Hub results
There is not much info (or none) about their implementations, but the truth is that almost every week I test samples in the Hub, KTS detects by System Watcher (PDM.Exploit) some on demand undetected samples (usually MsOffice docs, and sometimes other types of files) or on demand by Heur.Exploit, so I don't mind if that exploit tool designed for HMP.A does not trigger any Kaspersky Anti-Exploit modules, since against real malware Kaspersky Anti-Exploit usually is effective, as demonstrated in Hub results
- Feb 17, 2018
- 870
Same with Norton Security, does not trigger anything.
Removed: Gdata
Added:Appguard & Eset IS
Gdata while seeming to be much lighter then the last time i ran/tested it, still produces just enough resource impact for me to chose not to use it. I noticed a start up impact on the machine "which is not that big of deal, as i start up slowly myself now days..lol", but also notice a huge impact upon scanning, maxing HHD 100% through out scans leaving the system unusable and very hard on the drive. I understand i can lesson the impact by disabling 1 of the engines, but what is the point in that, why run a dual engine, if you have to disable one for scans.
I have placed Appguard back on, which i was itching to do, as my system just does not seem right without it... Seriously though, after testing and finding my older machine is vulnerable to Spectre, which i understand is harder to exploit, but also understand that Appguards memory protections will be a must, meaning Appguard is here to stay on this machine.
Eset, well, the Firewall and Network Protection are among the best i feel consumers can get their hands on. The ability to monitor the whole network, and protect it from attacks, as well as scan it for vulnerabilities is just about as irreplaceable as Appguard. The 2 combined make for a solid defense, and will be what i run from this point forward.
Added:Appguard & Eset IS
Gdata while seeming to be much lighter then the last time i ran/tested it, still produces just enough resource impact for me to chose not to use it. I noticed a start up impact on the machine "which is not that big of deal, as i start up slowly myself now days..lol", but also notice a huge impact upon scanning, maxing HHD 100% through out scans leaving the system unusable and very hard on the drive. I understand i can lesson the impact by disabling 1 of the engines, but what is the point in that, why run a dual engine, if you have to disable one for scans.
I have placed Appguard back on, which i was itching to do, as my system just does not seem right without it...
Seriously though, after testing and finding my older machine is vulnerable to Spectre, which i understand is harder to exploit, but also understand that Appguards memory protections will be a must, meaning Appguard is here to stay on this machine. Eset, well, the Firewall and Network Protection are among the best i feel consumers can get their hands on. The ability to monitor the whole network, and protect it from attacks, as well as scan it for vulnerabilities is just about as irreplaceable as Appguard. The 2 combined make for a solid defense, and will be what i run from this point forward.
Similar threads
