I did an unofficial test of CIS 5.8.199581.2037 Beta (mainly to investigate it's performance for myself). In general, I am not concerned about a security program's signature based malware detection. If you do not have a signature, you are not protected (if a real-time scanner detects 99%, it takes only one malware to slip through and cause major problems). So, I want to see how well a security program can protect me even if the real-time signature based scanner is disabled.
Test set up
I ran the tests on VMware with a clean install of XP SP3 32 bit. Just to be sure, I ran the most up to date versions of CCE, Malwarebytes, Hitman Pro, and Kaspersky TDSS Killer and all showed no infections.
I started with 61 malware files from Malwaredomain list and malcode database (about 50% of the malware was posted in the last 48 hours, and the rest were from within the last month).
Test of On demand scanner
I started with on demand scans using the most recent version and most recent malware database for CIS, HMP (hitman pro), and MWB (Malwarebytes). I scanned the malware folder sequentially with the 3 scanners (each time I scanned in a different order). Here are the results:
61 files: CIS removed 52 files --> MWB removed 7 more --> HMP removed 2 more
61 files: MWB removed 57 files --> CIS removed 2 more --> HMP removed 2 more
61 files: HMP removed 55 files --> CIS removed 4 more --> MWB removed 2 more
This is a good demonstration that using only signature based protection will expose you to malware that is unknown to your scanner (and this is why I think the signature based detection rate is not very important). The results above also show the importance of using more than one on-demand scanner to check a PC for malware.
Test of CIS with real-time antimalware scanner disabled
I then ran the 61 malware files with CIS 5.8 Beta installed (no other antimalware or security programs were installed). The CIS settings were:
Antivirus - DISABLED
Configuration = proactive
All alert options were set to allow me to chose what to do (no autoblocking). I responded to alerts as follows: Virus alert response was "clean". If the D+ alert indicated that I should block or sandbox, I did.
Rest of settings were default (firewall safe, D+ safe, sandbox enabled)
Here is what happened (the numbers represent the # of malware files that generated a given result):
40 - Almost immediate Cloud AV scanner alert and sandbox alert.
8 - Almost immediate Cloud AV scanner alert and D+ alert saying block this known malware.
7 - Sandbox alert only.
2 - Sandbox alert, but files would not run (not a valid win32 application error)
2 - D+ alert alone, and text in alert was clear enough to recommend blocking, even to a noob.
1 - D+ alert that application was signed by a trusted vender, but not yet whitelisted - okay to allow, so I allowed it. Virustotal detected this as "Gen.Variant.Kazy" malware on 16/43 scanners - not sure if this is actually malware (file submitted to comodo)
1 - ran without any alerts, trusted by comodo. Virus total scan showed malware on 1/43 scanners - may not be malware.
Ran CCleaner, then rebooted. Then I got the following D+ warnings about 30 seconds after reboot (see pics below), but no other alerts. Personally, I think these warnings are not very helpful, and many users will likely allow these requests since the alert is rather neutral. I blocked these alerts. Then, I checked the files that generated these alerts with virus total:
processindex.exe -17/43 scanners detected it as Gen:Variant.Kazy. (possible malware).
stpp.exe - 1/43 scanners detected it as adware (likely not malware).
2 other files were dropped in the same folder as stpp.exe (program files>stip)
rmip.exe - 1/43 virustotal scanners detected it as adware (likely not malware).
stip.exe - 0/43 virustotal scanners detected it (likely not malware).
Then I scanned with MWB, and it showed 8 malicious items: 6 executable files, 1 folder, and 1 registry entry. No malware was running in memory. Also, the files listed above (processindex, stpp, rmip, and stip) were not detected as malware. These INACTIVE files were likely left over because of incomplete virtualization of the automatic sandbox.
I then ran the 6 executable files that MWB detected and CIS gave the following alerts:
1 - Almost immediate Cloud AV scanner alert and D+ alert saying block this known malware.
1 - Almost immediate Cloud AV scanner alert and D+ alert saying block this known malware. But, when I selected block on the D+ alert, the file ran in the sandbox, took focus (window remained in front of everything else), and I could not bring the AV alert to the front. To access the AV alert, I had to open the task manager and terminate the malware process. Comodo needs to find a way to prevent a sandboxed malware from taking focus (full screen or not)!
2 - D+ alert saying block because heuristics detects as malware. But, when I selected block on the D+ alert, the file ran in the sandbox, took focus (window remained in front of everything else), and I could not bring the AV alert to the front. To access the AV alert, I had to open the task manager and terminate the malware process.
1 - Sandbox alert
1 - could not find the file despite viewing hidden files (could not find the folder it was in either)
Scanned next with Hitman Pro. It found 5 additional executable malware files that were not detected by MWB (but nothing malicious was running in memory). I ran the 5 executable files that HMP detected and CIS alerts were:
2 - Almost immediate Cloud AV scanner alert and sandbox alert.
3 - sandbox only (these files were detected by virus total 14/43, 18/43, and 16/43).
My conclusions:
1) Even with the CIS on-demand scanner turned off, the D+ cloud scanner will alert the user to the majority of malware almost immediately. I know it is signature based, but this shows the effectiveness of the cloud scanner in CIS.
2) Comodo needs to find a way to prevent a sandboxed malware from taking focus (full screen or not). When this happens, it often makes the PC unusable, especially when an undetected malware goes full screen (an undetected malware can take focus even after reboot). In my case, the heuristics detected the malware, and even though I blocked it, it ran in the sandbox and stole focus.
3) Comodo needs to minimize the number of alerts per given malware file. For a single executed malware file, I often got 2 or 3 cloud AV alerts, a sandbox alert, and 2 or 3 quarantined alerts (5-7 alerts for one file!). In other cases, I would also get multiple cloud AV alerts and a D+ alert for a single file. These multiple alerts are confusing. Comodo needs to find a way to prioritize the alerts, so that you get one alert for the detection and then one confirming quarantine (or other action that was taken).
4) CIS must be very careful when trusting programs. I am concerned about the integrity of the whitelist, the integrity of the trusted vendors list, and the validity of signatures. An alert that says "application is signed by a trusted vender, but not yet whitelisted - okay to allow" is not an acceptable alert if a file is malware (not sure the file was indeed malware, but it was detected by 16 scanners on virus total). As suggested many times before, Comodo needs a way to let the user control the trusted vendors list.
5) Hopefully we will see full virtualization of the auto sandbox.
