Recent claims have led to plan and execute a video-recorded real-world test that will test the solutions below:
- Webroot - highly advocated product. User claims that the product includes "sandbox"/"isolation"
- Trend Micro - claims include that me and Shadowra have tested it in unrealistic conditions
- ZoneAlarm - product was criticised and its users were deemed to have "insufficient evaluation skills"
- Eset - this will be included just to regulate the test -- it is a highly reputable and regarded product.
So the framework:
On the first stage, the products will be tested with several malicious and phishing links. I will put all links in an HTML page so I can just drag and drop on top of the browser and start clicking, instead of copy/pasting from a document. I will take extra care to remove PUPs and will pre-analyse everything on various sandboxes, to ensure it is malicious.
On the second stage, products will be tested against malware that I am actively hunting on my mac. This malware will be uploaded to a sharing portal and downloaded. I consider this to be a valid real world scenario -- attackers can for example, take over an Instagram account (happens very frequently) and share a malicious link there. You believe it is a trusted document coming from an acquaintance, so you rush to open it.
This malware as well will be in an HTML table so I can just click.
On the third stage, products will be tested against phishing and malicious documents that I am creating myself. These will be downloaded too. They will be as well on an HTML table, so I can just easily click.
No malware will be introduced through unrealistic means, such as malware packs.
If malware has been allowed to execute (not deleted right away), system will be monitored with Process explorer, potentially wireshark and will be scanned with Norton Power Eraser.
The framework predicts 2 verdicts only: pass and fail. Every product will be allowed to miss 1 phishing website and 0 malware samples to pass. (feedback?)
In terms of settings, I need your opinion.
ZoneAlarm (the criticised product) is a fully automated solution with no tweaks.It only allows components to be turned on and off but does not offer settings such as heuristics aggressiveness, anti-phishing aggressiveness, etc, My question is, should I in this case, tweak other products. How many users tweak their products really? I need feedback before I kickstart.
Talk about a hard and thankless job! 
