One reason it that those tests are synthetic and not real world based, the most applicable test he/she conducted is the keylogger one. The only thing that was really tested is which software shows the most popups.
–
–
@Rod McCarthy
That published test is a bit controversial. I am NOT saying the results ARE NOT correct. I am NOT saying the results ARE correct.
Are the test results a cause for concern - yes, they certainly are - or at least something that should be considered.
Are the test results a source of confusion - oh, most definitely - and I will explain further.
The bottom line is that Comodo's default configuration - Internet Security - is not really all that secure. In fact, this is official Comodo stance since with Internet Security configuration, HIPS is disabled. In CIS, there are many protections that are dependent upon HIPS being enabled. Disable HIPS and user has effectively hobbled CIS and made their system much less secure.
That default Internet Security configuration does not provide the most robust protections is well established. The gentleman's test results confirm this fact. Essentially, it is no surprise.
With HIPS enabled there are some obvious potential problems. After reading the gentleman's test methodology carefully many times, there is only one conclusion that can be reached - when the gentleman enabled HIPS, it is unclear which method he used to activate HIPS.
Test author states: 'Comodo's HIPS module does not install by default - but if you enable it deep down in installer, it goes into "Safe Mode".' Unfortunately, this statement does not indicate precisely how he enabled HIPS - as there are various ways to do so. And that is a big problem - since activating HIPS each way provides different levels of protection.
- Did he install default Internet Security configuration and then enable HIPS by Advanced Settings > Security Settings > Defense+ > HIPS > HIPS Settings > tick Enable HIPS (default for this setting is Safe Mode).
or
???
- Did he activate Proactive Security configuration - which enables HIPS automatically by default - by switching from default Internet Security to Proactive Security configuration by Advanced Settings > General Settings > Configuration > Right-Click on Proactive Security > Select Activate.
The short of it. Protection-wise the 1st method of enabling HIPS protects the system to a lesser extent than by enabling HIPS using the second method. Why is there a difference? Because HIPS rules for each configuration are different; Internet Security configuration built-in HIPS rules are less secure than Proactive Security configuration built-in HIPS rules.
So, if he used the 1st method to enable HIPS, it could be the reason Comodo did not protect against those specific Matousec SSTS64 utilities.
If he used the 2nd method to enable HIPS, it could be an indication of a security hole.
The fact that the gentleman used Matousec SSTS64 utilities is a really important point BECAUSE...
Matousec vigorously tested Comodo Internet Security Premium 7.0.317799.4142 against the very same utilities - along with 50 or so additional ones - and Comodo performed spectacularly. The test results were essentially a repeat of multiple testings of Comodo Internet Security by Matousec over the years. Every time Matousec tested Comodo, it performed spectacularly.
The problem I have with the Matousec test results is figuring out exactly what Comodo Internet Security settings the Matousec testing team used:
Here is a very small part of their published testing:
Methodology and rules
Installation and configuration
The tested products are installed on a virtual machine running Windows 7 Service Pack 1 with Internet Explorer 9 set as the default browser and with UAC turned off. The products are configured to their highest usable security settings and tested with this configuration only. We define the highest usable security settings as follows. The user must be able to do the configuration of the product without need of expert knowledge of the operating system and the computer security. This means that the user, with the skills and knowledge we assume, is able to go through all forms of the graphic user interface of the product and enable or disable or choose among several therein given options, but is not able to think out names of devices, directories, files, registry entries etc. to add to various table of protected objects manually, not even if such a configuration is suggested on the product's support forum or website. The product is configured to interact with the user as much as possible, reducing the number of automatic decisions made by the product as much as possible. To meet the usability requirement it must be possible to use the computer with the configured product for all legitimate tasks as if there was no security product installed. It is also required that the user is not forced to predict behavior of any unknown application and that under normal circumstances (i.e. no malware attack is in the progress) and once the product is set up properly, the product does not bother the user too often.
IF ANYONE CAN TRANSLATE THE ABOVE PARAGRAPH INTO ACTUAL COMODO INTERNET SETTINGS - PLEASE CALL ME.
Final Matousec SSTS64 test results can be found here: Results and comments - www.matousec.com
Products tested against the suite with 110 tests
Product Product score Level reached Protection level Recommendation Report Award
Comodo Internet Security Premium 7.0.317799.4142FREE 97 % 11+ Excellent GET IT NOW!
Outpost Security Suite Pro 9.1.4643.690.1951 90 % 11 Excellent GET IT NOW!
Kaspersky Internet Security 2015 15.0.0.463 89 % 11 Very good GET IT NOW!
SpyShelter Firewall 9.2 89 % 11 Very good GET IT NOW!
Privatefirewall 7.0.30.3FREE 88 % 11 Very good N/A
Outpost Security Suite Free 7.1.1.3431.520.1248FREE 71 % 11 Good Not recommended
VirusBuster Internet Security Suite 4.1 71 % 10 Good Not recommended
ESET Smart Security 8.0.304.0 67 % 11 Good Not recommended
Jetico Personal Firewall 2.1.0.13.2471 58 % 10 Poor Not recommended
ZoneAlarm Extreme Security 2013 11.0.780.000 34 % 6 Very poor Not recommended
ZoneAlarm Free Antivirus + Firewall 13.1.211.000FREE 34 % 6 Very poor Not recommended
Total Defense Internet Security Suite 9.0.0.134 30 % 6 Very poor Not recommended
Dr.Web Security Space 10.0.0.12011 24 % 4 None Not recommended
Webroot SecureAnywhere IS Complete 8.0.4.104 23 % 4 None Not recommended
Bitdefender Total Security 2014 17.28.0.1191 19 % 4 None Not recommended
BullGuard Internet Security 2014 14.0.279.6 16 % 3 None Not recommended
eScan Internet Security Suite 14.0.1400.1381 14 % 3 None Not recommended
Avira Internet Security 2014 14.0.6.552 9 % 2 None Not recommended
K7 TotalSecurity 2014 14.1.0.217 9 % 2 None Not recommended
Norton Internet Security 2014 21.3.0.12 9 % 2 None Not recommended
avast! Internet Security 2015.10.0.2208 8 % 2 None Not recommended
.
.
.
.
.
.
.
.
.
.
.
.
.
IF you read through all the infos carefully you begin to see problems with how the gentleman enabled HIPS - which affects protection.
He says CIS does not protect against BITStest, Kill5, SchedTest3 - which are Matousec SSTS64 utilities.
Matousec says CIS protects against BITStest, Kill5, SchedTest3 - for years running - BUT - we don't know precisely what configuration or settings they used !
Knowing Matousec - which is/was closely affiliated with Standford University during the time of testing - I tend to give more weight to Matousec results. And also because their testing was very consistent over the years.
I just wish I had their damn Comodo settings...
PS - Deploying Matousec SSTS64 suite is NOT trivial; it is not a simple click on an installer - but rather a long drawn-out process that involves creation of a testing container\environment. Done improperly - which is easy to do - will produce erroneous results.
proactive mode
@Rod McCarthy
That published test is a bit controversial. I am NOT saying the results ARE NOT correct. I am NOT saying the results ARE correct.
Are the test results a cause for concern - yes, they certainly are - or at least something that should be considered.
Are the test results a source of confusion - oh, most definitely - and I will explain further.
The bottom line is that Comodo's default configuration - Internet Security - is not really all that secure. In fact, this is official Comodo stance since with Internet Security configuration, HIPS is disabled. In CIS, there are many protections that are dependent upon HIPS being enabled. Disable HIPS and user has effectively hobbled CIS and made their system much less secure.
That default Internet Security configuration does not provide the most robust protections is well established. The gentleman's test results confirm this fact. Essentially, it is no surprise.
With HIPS enabled there are some obvious potential problems. After reading the gentleman's test methodology carefully many times, there is only one conclusion that can be reached - when the gentleman enabled HIPS, it is unclear which method he used to activate HIPS.
Test author states: 'Comodo's HIPS module does not install by default - but if you enable it deep down in installer, it goes into "Safe Mode".' Unfortunately, this statement does not indicate precisely how he enabled HIPS - as there are various ways to do so. And that is a big problem - since activating HIPS each way provides different levels of protection.
- Did he install default Internet Security configuration and then enable HIPS by Advanced Settings > Security Settings > Defense+ > HIPS > HIPS Settings > tick Enable HIPS (default for this setting is Safe Mode).
or
???
- Did he activate Proactive Security configuration - which enables HIPS automatically by default - by switching from default Internet Security to Proactive Security configuration by Advanced Settings > General Settings > Configuration > Right-Click on Proactive Security > Select Activate.
The short of it. Protection-wise the 1st method of enabling HIPS protects the system to a lesser extent than by enabling HIPS using the second method. Why is there a difference? Because HIPS rules for each configuration are different; Internet Security configuration built-in HIPS rules are less secure than Proactive Security configuration built-in HIPS rules.
So, if he used the 1st method to enable HIPS, it could be the reason Comodo did not protect against those specific Matousec SSTS64 utilities.
If he used the 2nd method to enable HIPS, it could be an indication of a security hole.
The fact that the gentleman used Matousec SSTS64 utilities is a really important point BECAUSE...
Matousec vigorously tested Comodo Internet Security Premium 7.0.317799.4142 against the very same utilities - along with 50 or so additional ones - and Comodo performed spectacularly. The test results were essentially a repeat of multiple testings of Comodo Internet Security by Matousec over the years. Every time Matousec tested Comodo, it performed spectacularly.
The problem I have with the Matousec test results is figuring out exactly what Comodo Internet Security settings the Matousec testing team used:
Here is a very small part of their published testing:
Methodology and rules
Installation and configuration
The tested products are installed on a virtual machine running Windows 7 Service Pack 1 with Internet Explorer 9 set as the default browser and with UAC turned off. The products are configured to their highest usable security settings and tested with this configuration only. We define the highest usable security settings as follows. The user must be able to do the configuration of the product without need of expert knowledge of the operating system and the computer security. This means that the user, with the skills and knowledge we assume, is able to go through all forms of the graphic user interface of the product and enable or disable or choose among several therein given options, but is not able to think out names of devices, directories, files, registry entries etc. to add to various table of protected objects manually, not even if such a configuration is suggested on the product's support forum or website. The product is configured to interact with the user as much as possible, reducing the number of automatic decisions made by the product as much as possible. To meet the usability requirement it must be possible to use the computer with the configured product for all legitimate tasks as if there was no security product installed. It is also required that the user is not forced to predict behavior of any unknown application and that under normal circumstances (i.e. no malware attack is in the progress) and once the product is set up properly, the product does not bother the user too often.
IF ANYONE CAN TRANSLATE THE ABOVE PARAGRAPH INTO ACTUAL COMODO INTERNET SECURITY SETTINGS - PLEASE CALL ME.
Final Matousec SSTS64 test results can be found here: Results and comments - www.matousec.com
Products tested against the suite with 110 tests
Product Product score Level reached Protection level Recommendation Report Award
Comodo Internet Security Premium 7.0.317799.4142FREE 97 % 11+ Excellent GET IT NOW!
Outpost Security Suite Pro 9.1.4643.690.1951 90 % 11 Excellent GET IT NOW!
Kaspersky Internet Security 2015 15.0.0.463 89 % 11 Very good GET IT NOW!
SpyShelter Firewall 9.2 89 % 11 Very good GET IT NOW!
Privatefirewall 7.0.30.3FREE 88 % 11 Very good N/A
Outpost Security Suite Free 7.1.1.3431.520.1248FREE 71 % 11 Good Not recommended
VirusBuster Internet Security Suite 4.1 71 % 10 Good Not recommended
ESET Smart Security 8.0.304.0 67 % 11 Good Not recommended
Jetico Personal Firewall 2.1.0.13.2471 58 % 10 Poor Not recommended
ZoneAlarm Extreme Security 2013 11.0.780.000 34 % 6 Very poor Not recommended
ZoneAlarm Free Antivirus + Firewall 13.1.211.000FREE 34 % 6 Very poor Not recommended
Total Defense Internet Security Suite 9.0.0.134 30 % 6 Very poor Not recommended
Dr.Web Security Space 10.0.0.12011 24 % 4 None Not recommended
Webroot SecureAnywhere IS Complete 8.0.4.104 23 % 4 None Not recommended
Bitdefender Total Security 2014 17.28.0.1191 19 % 4 None Not recommended
BullGuard Internet Security 2014 14.0.279.6 16 % 3 None Not recommended
eScan Internet Security Suite 14.0.1400.1381 14 % 3 None Not recommended
Avira Internet Security 2014 14.0.6.552 9 % 2 None Not recommended
K7 TotalSecurity 2014 14.1.0.217 9 % 2 None Not recommended
Norton Internet Security 2014 21.3.0.12 9 % 2 None Not recommended
avast! Internet Security 2015.10.0.2208 8 % 2 None Not recommended
.
.
.
.
.
.
.
.
.
.
.
.
.
IF you read through all the infos carefully you begin to see problems with how the gentleman enabled HIPS - which affects protection.
He says CIS does not protect against BITStest, Kill5, SchedTest3 - which are Matousec SSTS64 utilities.
Matousec says CIS protects against BITStest, Kill5, SchedTest3 - for years running - BUT - we don't know precisely what configuration or settings they used !
Knowing Matousec - which is/was closely affiliated with Standford University during the time of testing - I tend to give more weight to Matousec results. And also because their testing was very consistent over the years.
I just wish I had their damn Comodo settings...
PS - Deploying Matousec SSTS64 suite is NOT trivial; it is not a simple click on an installer - but rather a long drawn-out process that involves creation of a testing container\environment followed by utilities testing. It requires a precise order of multiple steps. Done improperly - which is easy to do - will produce erroneous results. In other words, SSTS64 suite is a convoluted mess for someone who is not well-versed in its use...
Oh and can I have link to both of you guys settings... Running out door, or is it in this post?
