This week, we are posting a new product family chart. A PDF version is located here on our website, and is also shown below for convenience. The goal is to give our website visitors a quick side-by-side comparison of Netgate appliances against one another. While most of this data (and more) exists on individual product pages, we believe the chart makes it fast and easy for viewers to quickly determine which appliance is best for their needs.
That said, there is some new data within. About 3/4 of the way down the table, we’ve inserted a performance comparison section. In this section, you’ll find a pretty interesting matrix of performance tests for each product: The matrix has two dimensions:
- Packet Size: iPerf3 and IMIX
- Secure Networking Function: Routing (Forwarding), Firewall, VPN
In our view, this provides a very clear manner by which products can be compared - and under different levels of user-experienced traffic conditions. We see this as crucial given our user base varies, literally, from home consumers (with relatively light bandwidth and firewall needs) all the way to sophisticated enterprises (who demand predictable performance under the most strenuous encryption and packet mix conditions).
We must stress, however, that these performance numbers remain “bell curve guidelines”. Tight bell curves, hopefully, but bell curves nonetheless. In other words, YMMV (your mileage may vary). Why? A quick look at auto speed testing provides a good analogy.
A Porsche GT2 RS is a rocket of a car. But how fast, exactly, can it go? The answer is, “it depends”. The car configuration, track, weather, altitude, driver, fuel, tires and more will significantly impact test results. Failure to state exact test parameters and conditions means any two tests - even of the exact same car - can yield wildly varying results.
This presents a slippery slope (pun intended) for a vendor trying to make secure networking performance claims. If we state our performance based only on a single metric, e.g., L3 forwarding of iPerf3 traffic, a user who deploys an appliance processing loads of encrypted, varied packet-size traffic will be sorely disappointed. If we go to the other extreme - every possible permutation of product configuration, feature activation and traffic type - not only would we never finish testing, our controlled lab environment will never (I repeat, never) yield the same results as are likely to be seen in real-world conditions.
So we picked a happy medium - a digestible array of test conditions that fairly indicate low and high water marks of traffic throughput potential for a given product - and a quick, easy way to compare it to neighboring Netgate appliances that vary by CPU, number of ports, and internal architectures. If you’d like a primer on our products’ internal architectures, see this recent blog.
As you consider these test results, an understanding of each test parameter can be helpful:
-
iPerf3: Max throughput of an appliance using 1460 bytes of TCP framed packets. This is a good test if you are a home user where your most strenuous traffic is watching movies from the likes of Netflix, Hulu, Youtube, etc. - all of whom stream video content using TCP and a few seconds of buffering, instead of using UDP, since the delay is not crucial and TCP transfers are easily accomplished over HTTP and web browsers.
-
IMIX: We use the Simple IMIX standard, a test which is generally well-suited for medium to large size networks, where traffic is statistically spread over a relatively large number of concurrent users. But, home users, telecommuters, and small office home office (SOHO) users who routinely access multiple corporate applications at a remote data center or in the cloud can also experience traffic patterns akin to Simple IMIX.
-
L3 Forwarding: Layer 3 forwarding is the movement of a packet from one internal architecture point to another using an IP header. This measurement is a good approximator of raw routing performance. A secure networking appliance must do this work really fast, since it is not burdened with any firewall, VPN, or other security inspection/enforcement responsibility.
-
Firewall: Appliance speed will start to drop as you subject traffic to a firewall access control list (ACL) which specifies which users or system processes are granted access to objects, as well as what operations are allowed on given objects. We use a benchmark standard of 10,000 ACLs - as that is a level of computational work which burdens CPU and memory resources to the point where they must perform under stress.
-
IPsec VPN: Internet Protocol Security (IPsec) is a secure network protocol suite that authenticates and encrypts the packets of data sent over an Internet Protocol network. It is used in virtual private networks. Different encryption schemas can be employed - the tradeoff being higher security (harder to crack) vs. speed (greater computational processing). Our products also use different crypto offload mechanisms, so we test to the highest security and best performing encryption modes available to each.
In summary, the new product family chart provides a fast, easy, and comprehensive compare and contrast of Netgate secure networking appliances. And for the first time, we are providing a detailed matrix of performance tests that cover a useful range of traffic types and applications.
The final point to be made here is that the above performance testing data is based only on pfSense® software, specifically Release 2.4.4-p3. If you are running a different version, YMMV. Note also, the SG-5100, XG-1537, and XG-1541 (so far) are also capable of running TNSR™ software, which substantially improves packet processing performance - particularly for more demanding use cases characterized by smaller packets and/or encryption. We’ll keep you informed as that test data comes to fruition.
Netgate always strives to provide objective and informative product information that aids buyers in determining the best Netgate product for their needs. We believe our new product family chart is a solid addition to that heritage. And the next time you’re watching a Porsche round the last turn at the Nürburgring, think of us!