The matousec tests are HIPS tests in reality so I wouldn't really count on them. Testing firewalls is quite the task imo since a lot of them aren't just firewalls anymore but include a plethora of extra features and gimmicks.Here I hope experienced users could introduce how to measure and evaluate the capability of a firewall.
If we were talking about an antivirus engine, maybe we could measure its capability with some straightforward indicators, such as detection rate & false positive rate, and evaluate it according some features that are not difficult to understand, such as the capability of detection packed viruses & the capability of removing active viruses.
But regarding the firewall... How to evaluate a firewall? The only thing I know is the Matousec tests: Results and comments - www.matousec.com
Now this is an excellent answer. I see a lot of reply's stating "it's this one" with no reasons as to why that particular user believes it to be.The matousec tests are HIPS tests in reality so I wouldn't really count on them. Testing firewalls is quite the task imo since a lot of them aren't just firewalls anymore but include a plethora of extra features and gimmicks.
Over the years it has become obvious that a standard firewall isn't needed anymore since almost all consumer routers include them anyways and even the Windows firewall is enough for blocking inbound(!) traffic. Now here comes the real deal: Outbound traffic. How do you test the protection of firewalls against outbound traffic? In reality, you really can't. There are hundreds if not thousands of ways to masquerade, hide and obscure traffic, because of this they started including HIPS features. Prime example here is Comodo, they provide insane amounts of control over quite a lot of things in their HIPS but is it really enough? Even the mighty Comodo has been breached and it will happen again and again until you have to manually allow even the movement of the mouse cursor. Even then it will happen since every software has flaws and coding errors, so why even bother with it? Software firewalls are insecure by design since the host can be compromised and hardware firewalls are too picky and require a lof of attention and knowledge to configure and keep safe and even then they can and will fail.
Long story short: Test all four and see which one fulfills your requirements of a firewall.