Netgate adds Export of Packet Flow Data using NetFlow v5 & IPFIX to pfSense Plus Software Version 24.03

As attacks on network infrastructure, both on-premises and in the cloud, become more advanced, the ability to send both metadata and data to collectors for visualization or analysis by AI-enabled tools will become one of the most important assets in your arsenal to protect your sensitive data. This functionality shouldn’t come as a high-performance burden to your security infrastructure.

Netgate® is pleased to announce that pfSense® Plus software version 24.03 will be able to directly export flow data to one or more external collectors, using either the NetFlow v5 or IPFIX protocol, by using the pflow(4) feature in pf(4). The data will be collected directly from firewall states and does not require a separate daemon, service, or add-on package.

An admin will have the flexibility to control the scope of exported data, as flows can optionally be tracked and exported for individual firewall rules. pfSense Plus software version 24.03 will support up to 16 different export configurations (combining different external collectors and formats/flow protocols).

Being able to capture, visualize, and analyze network flow data is critical for network management. Network flow data includes information about how data moves through the network, including endpoints, packets & bytes transferred, and protocols, without inspecting the data itself. Flow-based metrics can help with application response issues, usage-based accounting, traffic profiling, traffic engineering, attack/intrusion detection, QoS monitoring, and more.

Netgate chose to port pflow(4) from OpenBSD rather than using ng_netflow, a similar but less flexible and less performant technology in FreeBSD, for several reasons:

• Faster, scalable performance

• pflow(4) is significantly faster than ng_netflow and scales much better, as, like pf, it is able to leverage multiple CPU cores.
• ng_netflow, which is based on netgraph, is single-threaded and will bottleneck on a single core.  Actual export of netflow or IPFIX data additionally requires copying netflow or IPFiX frames to a Netgraph ng_knsocket node, which like all Netgraph nodes, is (also) single-threaded.

• Less processing overhead

• pflow(4)’s architecture has far less overhead since it only does the work to export flow data when a pf state expires, not for every single data packet. Using  ng_netflow requires copying every packet off the monitored interface, parsing it, and then copying it back to the monitored interface.
• pflow(4) integrates directly with pf, so unnecessary packet parsing isn’t forced (as it is with ng_netflow).
• pflow(4) does not require a separate daemon, service, or add-on package.

• Extra flexibility

• pflow(4) gives admins the flexibility to selectively export flow data that matches specific criteria, as defined by a pf rule.
• By using pflow(4), we also gain the ability to export IPFIX (netflow v10) and RFC8158 NAT44 information.

• Reporting from the packet filter

• ng_netflow combined with ng_knsocket can be adequate for getting network metadata with moderate performance impact
• By using pflow(4), we gain a full netflow sensor, the data, and the metadata without performance impact to the firewall appliance. 

Netgate also upstreamed the pflow(4) code to FreeBSD in January 2024, so all FreeBSD users have access to it.  For configuration of this feature, please refer to the documentation.

Netgate continues to listen to our customers, enhancing the pfSense Plus software experience to add capabilities while maintaining the industry’s best price/performance ratio and the lowest TCO. We continue to upstream code to FreeBSD to benefit everyone in the community.

Q&A

• What is Netflow?
  Netflow is a network protocol originally developed by Cisco for collecting IP network traffic as it enters or exits an interface. By analyzing flow data, Netflow provides valuable information about network traffic patterns, volume, and routing, which helps in bandwidth management and network performance optimization.

• Does pfSense support IPFIX for network traffic monitoring and analysis?
  Yes, pfSense supports IPFIX, which is an extension of the Netflow protocol, for network traffic monitoring and analysis. IPFIX allows for more detailed and flexible collection of traffic flow data, including IPv4 and IPv6 traffic, enhancing network security and monitoring capabilities.

• Does pfSense support exporting IPFIX records for network traffic monitoring?
  Yes, pfSense supports the exporting of IPFIX records for network traffic monitoring. This functionality enables pfSense to send detailed flow data to a designated flow collector for further analysis, helping in identifying network usage patterns and potential security threats.

• Does pfSense support exporting flow data using the IPFIX protocol?
  Yes, pfSense supports exporting flow data using the IPFIX protocol. This feature allows for comprehensive monitoring and analysis of network traffic by sending flow data to external collectors, which can then be used for detailed network analysis and troubleshooting.

• How can I enable IPFIX exporting on pfSense to monitor network traffic?
  To enable IPFIX exporting on pfSense, you can install and configure softflowd, an open-source Netflow exporter available from the pfSense package repository. After installing softflowd, configure it to capture and export flow data using the IPFIX template to your specified flow collector over UDP or TCP.

• Does pfSense support IPFIX for network traffic monitoring, and if so, how can it be configured?
  Yes, pfSense supports IPFIX for network traffic monitoring. It can be configured by installing a flow exporter like softflowd from the pfSense package manager. Once installed, you can set up softflowd to collect and export flow data using the IPFIX format to an external flow collector, specifying details such as the collector's IP address, port, and the desired transport protocol (UDP/TCP).

• What are the differences between SNMP, NetFlow, and IPFIX?
  SNMP (Simple Network Management Protocol) is used for managing devices on IP networks. It can query devices for various metrics like bandwidth usage but doesn't provide detailed flow data. NetFlow, developed by Cisco, and IPFIX (Internet Protocol Flow Information Export), which is an IETF standard that extends NetFlow, both provide mechanisms to export network flow data (source and destination IP addresses, port numbers, and more) from routers and switches. NetFlow and IPFIX allow for more detailed analysis of network traffic patterns and volumes than SNMP, with IPFIX offering more flexibility and a standardized template-based approach for flow data.

• What are the best practices for securing network monitoring tools?
  To secure network monitoring tools, it's essential to use strong authentication and encryption for data in transit and at rest. Implement access controls to limit who can view or alter monitoring data. Regularly update and patch your monitoring tools and underlying operating systems, whether they are Linux, Windows, or FreeBSD. For tools like pfSense Plus firewall, ensure that the firewall rules and VPN configurations are correctly set to protect the monitoring traffic, especially if it traverses public networks.

• How can you visualize network traffic in real-time?
  To visualize network traffic in real-time, you can use open-source tools like Wireshark for packet analysis or more comprehensive solutions like Softflowd on FreeBSD or Linux that can export flow data (using NetFlow or IPFIX) to a flow collector and visualizer. pfSense, an open-source firewall and router platform, also offers packages and built-in functionality for real-time traffic monitoring and visualization, using tools like the Darkstat package or through integration with external flow collectors and visualization platforms.

• What are the challenges of managing network traffic in large-scale networks?
  Managing network traffic in large-scale networks involves challenges like handling the sheer volume of data, ensuring network security, maintaining performance, and achieving efficient routing and bandwidth management. Tools and protocols like SNMP, NetFlow/IPFIX, and routing protocols need to be scaled and configured correctly across devices from different vendors like Netgate (pfSense), and others. The complexity of IPv4 and IPv6 addressing, alongside the need to monitor both ingress and egress traffic, adds to the challenge.

• How can cloud services be monitored effectively?
  Cloud services can be monitored effectively by using built-in cloud provider tools and third-party solutions that offer deep visibility into cloud workloads, network traffic, and performance metrics. Integration with NetFlow/IPFIX for flow data collection, along with the use of SNMP for basic device monitoring, can provide comprehensive visibility. Additionally, cloud-native monitoring tools often support functionalities like DNS query monitoring, TCP/UDP traffic analysis, and the ability to monitor encrypted traffic using advanced inspection techniques.

• What are the implications of encrypted traffic on network monitoring?
  Encrypted traffic, such as traffic secured by SSL/TLS, poses challenges for network monitoring because it hides packet payloads, making it difficult to analyze the content of network communications. While encryption enhances privacy and security, it limits the visibility that tools like NetFlow, IPFIX, and packet analyzers have into the specifics of network traffic. Solutions include using SSL/TLS decryption at network boundaries or focusing on metadata analysis (e.g., endpoints, timestamps) and pattern recognition to infer the nature of encrypted communications.

• What tools are available for monitoring network traffic on virtual machines?
  For monitoring network traffic on virtual machines, tools like Wireshark for packet analysis, and Softflowd or other NetFlow/IPFIX exporters for flow data can be used. Many virtualization platforms also offer native monitoring solutions that integrate with the hypervisor to provide visibility into VM traffic. For environments running on Windows, Microsoft's built-in monitoring tools and third-party solutions can offer detailed insights. Additionally, products like pfSense Plus software can be deployed as virtual appliances to act as firewalls and routers, providing network monitoring functionalities within virtualized environments.