Testing Firewalls for Outbound Attacks Protection
Testing firewalls was done on a set of 64 test cases for each product separately (detailed results for each of the products can be found
in an Excel report).
To remind you, in accordance with the used
award scheme, we gave 1 point (+) if the attack was automatically blocked, the protection of the program tested was not disarmed. 0.5 points (or +/-) was given if the attack was blocked only in some circumstances (for example, when the user would choose the right action by a question of the tested program). And, in the end, in case if the attack was completely successful, or if it partially disabled the protective shield, the points were not awarded. The maximum number of possible collected points in this test was 64.
Table 1-2 and picture 1-2 show firewall testing results separately from the standard and maximum settings. For a better picture, the results for each firewall are broken down into two groups: protection from basic difficulty level attacks and protection from more advanced level attacks.
Table 1: Test Results of Firewalls on Standard Settings
FirewallBasic Level Attacks (max. 56 points)Advanced Level Attacks (max. 8 points)Totally Points (max. 64)
%
Points%% of the amountPoints%% of the amount
Comodo5395%82,8%675%9,4%5992%
Online Armor5089%78,1%7,594%11,7%57,590%
Norton4580%70,3%675%9,4%5180%
Jetico4682%71,9%4,556%7,0%50,579%
Outpost4580%70,3%2,531%3,9%47,574%
Trend Micro4275%65,6%338%4,7%4570%
Kaspersky4275%65,6%2,531%3,9%44,570%
Dr.Web42,576%66,4%225%3,1%44,570%
TrustPort4377%67,2%0,56%0,8%43,568%
G DATA4275%65,6%113%1,6%4367%
Avast4173%64,1%113%1,6%4266%
Eset4173%64,1%113%1,6%4266%
Bitdefender4173%64,1%113%1,6%4266%
AVG4173%64,1%00%0,0%4164%
McAfee4173%64,1%00%0,0%4164%
PC Tools4173%64,1%00%0,0%4164%
Avira4071%62,5%00%0,0%4063%
Microsoft4071%62,5%00%0,0%4063%
F-Secure31,556%49,2%113%1,6%32,551%
Panda3054%46,9%00%0,0%3047%
Kingsoft2748%42,2%113%1,6%2844%
Picture 1: Test Results of Firewalls on Standard Settings
Protection from outbound attacks on the recommended by the vendor settings leaves much to be desired. Only three firewalls were able to overcome the threshold of 80% on standard settings. That's Comodo, Online Armor and Norton. Jetico products (79%) and Outpost (74%) are scoring quite close to them. The results of the other firewalls turned out to be significantly worse.
Compared to the results of the past testing all leaders confirmed their high results, there were only small movements within the leading group, for example, Outpost and Jetico switched their positions. The only surprise was Norton product that showed the results of 45% in the past testing and stayed in the bottom part of the table, and took the third place with 80% in this testing.
The results achieved are related to the fact that many vendors tweak the standard settings in such a way that the number of messages a user reacts to is lower. This is also confirmed by the test results: firewalls asked users questions only in 5.4% of attacks with standard settings and in 9.2% of attacks with the maximum settings. This, however, impacts the protection level that stays silent in a situation when a malicious program will imitate doing very legitimate actions in the system.
It's also important to note two trends. First, the percentage of preventing complex types of attacks overall is much worse than for the attacks at the basic complexity level. More than half of these attacks have been deflected only by four products - Comodo, Online Armor, Norton and Jetico. Four more products have been included in the borderline group having deflected from 25% to 38% of these attacks - that's Outpost, Trend Micro, Kaspersky and Dr.Web. All other products shielded from not more than one complex attack. Secondly, the indicators of basic attack protection improved. If in the past test 11 (50%) products shielded from fewer than 50% of attacks, then there were only 3 (14%) such products in this test.
Table 2: Test Results of Firewalls on Maximum Settings
FirewallBasic Level Attacks (max. 56 points)Advanced Level Attacks (max 8 points)Totally Points (max. 64)Total
%
Points%% of the amountPoints%% of amount
Comodo56100%87,5%8100%12,5%64100%
Bitdefender56100%87,5%8100%12,5%64100%
Online Armor5395%82,8%8100%12,5%6195%
Kaspersky5395%82,8%788%10,9%6094%
Norton50,590%78,9%8100%12,5%58,591%
PC Tools49,588%77,3%5,569%8,6%5586%
Outpost4988%76,6%5,569%8,6%54,585%
Eset4988%76,6%5,569%8,6%54,585%
Dr.Web46,583%72,7%563%7,8%51,580%
Jetico4682%71,9%4,556%7,0%50,579%
Trend Micro4377%67,2%338%4,7%4672%
TrustPort4377%67,2%2,531%3,9%45,571%
G DATA4275%65,6%338%4,7%4570%
Avira41,574%64,8%225%3,1%43,568%
Avast4173%64,1%1,519%2,3%42,566%
AVG4173%64,1%00%0,0%4164%
McAfee4173%64,1%00%0,0%4164%
Microsoft4071%62,5%00%0,0%4063%
F-Secure31,556%49,2%113%1,6%32,551%
Panda3054%46,9%00%0,0%3047%
Kingsoft2748%42,2%113%1,6%2844%
Picture 2: Test Results of Firewalls on Maximum Settings
When the maximum settings are on, the protection quality from outbound attacks for many tested firewalls has significantly gone up. This is especially noticeable for the products showing very average results. All leaders of the previous testing showed high results in this test as well. We should note the Bitdefender product for the changes made on it, and, together with Comodo, it showed 100% results and the Norton product that moved to the leading group.
The results for a number of products testing for the standard and the maximum settings turned out to be the same. This is related to the fact that these products don't have any special settings that can impact the results of our test.
Comparing Quality of Protection with Standard and Maximum Settings
In accordance with the logic of this test we are not going to add or average out the results of one and the same product with different settings. To the contrary, we want to compare them and show significant differences in the quality of protection for the tested products depending on the settings used.
For illustration we are giving end test results for firewalls with the standard and the maximum settings in table 3 and in picture 3.
Table 3: Overall results for the firewall testing with standard and the maximum settings
Product
Standard SettingsMaximum Settings
Comodo92%100%
Online Armor90%95%
Norton80%91%
Jetico79%79%
Outpost74%85%
Trend Micro70%72%
Kaspersky70%94%
Dr.Web70%80%
TrustPort68%71%
G DATA67%70%
Avast66%66%
Eset66%85%
Bitdefender66%100%
AVG64%64%
McAfee64%64%
PC Tools64%86%
Avira63%68%
Microsoft63%63%
F-Secure51%51%
Panda47%47%
Kingsoft44%44%
Picture 3: Summary Firewall Test Results on Standard and Maximum Settings
Picture 3 very clearly illustrates the difference in test results depending on the chosen settings.
First of all, only two products – Comodo and Online Armor show close to the highest security protection indicators with both standard and maximum settings.
Secondly, when changing the standard settings, suggested by the vendor, a number of products show a significantly better level of protection. This is best seen on such products as Bitdefender, Kaspersky, Eset, F-Secure and PC Tools.
Thirdly, as we have noted above, some of the tested products don't have any settings that could in any way impact the results of the test. That's why their results at all types of settings are the same in this test. This group includes Jetico, Avast, AVG, McAfee, F-Secure, Panda, Kingsoft and Microsoft.
The resulting number of points doesn't take into account the situations when the attack has been rejected, but there were issues with the product user interface. In the majority of the cases the issues were in «falling» out of the interface for a short period of time (from 2 to 10 seconds) or until the next loading of the operating system. Despite the fact that when there were problems with the user interface the products continued to provide protection, the fact that these problems existed is perceived negatively and can impact the preferences in the product choice. The number of problems with the user interface are shown in table 3 and in picture 3. The errors analyzed were the ones appearing during the 1st level attacks, and the total number of them was 41.
Table 4: The Number of Errors with Firewall's GUI on Standard and Maximum Settings
ProductStandard SettingsMaximum Settings
Errors%Errors%
McAfee3483%3483%
Microsoft3380%3380%
Kingsoft2049%2049%
F-Secure1946%1946%
Panda1741%1741%
Jetico1639%1639%
PC Tools1332%1332%
Trend Micro1229%1229%
AVG1024%922%
TrustPort922%922%
G DATA922%922%
Bitdefender820%820%
Norton615%615%
Avast512%512%
Outpost512%512%
Eset512%410%
Comodo512%00%
Avira25%25%
Dr.Web25%25%
Kaspersky12%12%
Online Armor12%12%
Picture 4: The Number of Errors with GUI on Standard and Maximum Settings
The results we got show that the problems with the user interface with McAfee and Microsoft products have occurred in the majority of attacks (more than 80%). This can be called an unacceptable level, because practically any deflected attack will lead to problems. Pretty bad results, that are in the range of 30% to 50% are shown by products Kingsoft, F-Secure, Panda, Jetico and PC Tools. When they are being used, each second-third attack will lead to problems with the interface. Another whole lot of products show the results from 10% to 30%, that can be called satisfactory. The products Avira, Dr.Web, Kaspersky and Online Armor showed good results, the problems for them occurred in the range of 2-5% of attacks. The only product that hasn't had any issues with user interface was Comodo on maximum settings, that can be considered to be an excellent result. However, with the standard settings, Comodo results get worse (12%), which tells about the fact that using this product requires certain knowledge about its settings.
Final Test Results and Awards
Just like in the previous testing we haven't averaged out the results of one and the same product with different settings, but looked at the results independent of each other. Therefore, each of the tested products can get two awards, one for each setting type.
In accordance with the award plan, the best firewalls get the award specifying the settings used for that purpose, see table 4.
Table 5: Final Firewall Test Results on Standard and Maximum Settings
ProductSettings TypeBlocking Attacks [%]Total
[%]Award
Basic LevelAdvanced Level
ComodoMax100%100%
100%
Platinum Firewall Outbound
Protection Award
BitdefenderMax100%100%
100%
Online ArmorMax95%100%
95%
Gold Firewall Outbound
Protection Award
KasperskyMax95%88%
94%
ComodoStandard95%75%
92%
NortonMax90%100%
91%
Online ArmorStandard89%94%
90%
PC ToolsMax88%69%
86%
OutpostMax88%69%
85%
EsetMax88%69%
85%
NortonStandard80%75%
80%
Dr.WebMax83%63%
80%
JeticoMax82%56%
79%
Silver Firewall Outbound
Protection Award
JeticoStandard82%56%
79%
OutpostStandard80%31%
74%
Trend MicroMax77%38%
72%
TrustPortMax77%31%
71%
Trend MicroStandard75%38%
70%
KasperskyStandard75%31%
70%
Dr.WebStandard76%25%
70%
G DATAMax75%38%
70%
TrustPortStandard77%6%
68%
Bronze Firewall Outbound
Protection Award
AviraMax74%25%
68%
G DATAStandard75%13%
67%
AvastMax73%19%
66%
AvastStandard73%13%
66%
EsetStandard73%13%
66%
BitdefenderStandard73%13%
66%
AVGMax73%0%
64%
AVGStandard73%0%
64%
McAfeeMax73%0%
64%
McAfeeStandard73%0%
64%
PC ToolsStandard73%0%
64%
MicrosoftMax71%0%
63%
MicrosoftStandard71%0%
63%
AviraStandard71%0%
63%
F-SecureMax56%13%
51%
No Award
F-SecureStandard56%13%
51%
PandaMax54%0%
47%
PandaStandard54%0%
47%
KingsoftMax48%13%
44%
KingsoftStandard48%13%
44%
