Update 2021-03-18: Pending further updates from Netgate, we recommend users of WireGuard do not configure jumbo frames on WireGuard interfaces (MTU >= 1420). The blog post text below has been updated to reflect this recommendation.

I’ve been involved in the open source community dating back to the first time I downloaded and installed 386BSD 0.1 in 1992 while a freshman in college. The collaborative community, the transparency of the plain-to-see source code, and the opportunities to learn, grow, and be mentored by others spoke to me on a deep level. Twenty nine years later, I’m still learning, still growing, and still seeking out positive collaboration with others.

I talked about WireGuard in this blog a couple of months ago. My team and I were proud of the work, proud of the results, and eager to share it with the pfSense and FreeBSD communities. One of the inherent advantages to using open source software is that a community will often form around what you’re doing, and that community has a multiplier effect on improving the code and reducing the burden of maintenance. It’s a symbiotic relationship, and the whole becomes greater than the sum of its parts. That’s certainly happened with Netgate and pfSense on a large scale, and we were hoping that it would happen with WireGuard on a smaller scale.

Writing kernel code is hard, and writing security-focused kernel code is even harder. It winds up being a collaborative effort by necessity; even the best developers need code reviews, design feedback, and constructive criticism. Even then, mistakes can slip through, ranging from being simple but overlooked, to being complex and intertwined in the structure of the code. So security is also an iterative and interactive process. The first review might miss a bug, but the second review will catch it. The important things are to always operate openly, collaborate in good faith, and leave your ego at the door.

For WireGuard, our developer started the work in 2019 and put it out for private review in May 2020. In August 2020 he put it out for public review. A lot of iterative feedback and fixes happened during that time; I think I counted 92 exchanges on the public review. When it finally was submitted to the FreeBSD source tree in November 2020, we all felt it was in a state that would be useful for others. During the code review, and all the way through our pfSense Plus and CE release cycles in February 2021, we tested it internally and we encouraged the community to test it as well. There were bugs that were reported, found, and fixed during this time, and by the end, we felt pretty good about it. There were some unresolved issues, but they seemed to either be minor and able to be worked around, or they were in use cases that didn’t apply to pfSense software. In particular, the code was not working well in FreeBSD’s “jail” container environment. We take all bug reports seriously, but we also prioritize them. Since jails are not a normal use-case for pfSense, we deferred the problem for the release.

Around the same time as the release, interest in the FreeBSD community around WireGuard started to grow and attract the attention of other developers. This is exactly what we had hoped for; many hands make light work, and collaboration strengthens us all. However, open collaboration on security code requires thoughtful handling when that code has already been published. Fixing a bug sometimes means exposing a vulnerability and highlighting an exploit. Sometimes it means taking ownership of a careless mistake or a bad design. There are social norms and community practices, though, for working in this kind of environment that don’t compromise openness and transparency and also don’t put users at undue risk. The lessons learned in the IT industry on this topic, good and bad, what to do and what not to do, can easily fill a book or two. However, the guiding principles that I’ve found in my 29 years are to always be transparent, always be respectful and empathetic towards others, and to always keep your ego in check. Omitting any of these principles results in unnecessary pain, conflict, confusion, and distrust.

We are taking the public discussion from the past week about WireGuard and FreeBSD very seriously. The uncoordinated publication caught us off-guard, which is unfortunate and not the norm in the security community. However, every issue that has been disclosed to us is being investigated and evaluated. – Right now, we have not found any issues that would result in a remote or unprivileged vulnerability for pfSense users who are running WireGuard. – We’ve identified several low-risk issues that are unlikely to be exploitable, except by an attacker who has already compromised the admin permissions of the system. The use of Jumbo Frames appears to be a concern, and we strongly recommend users avoid manually configuring a WireGuard interface with an MTU larger than its own default of 1420 bytes, but this is not a typical use case for most networks and most users, and most Internet connections are not capable of carrying such large frames. Again, we take these discussions seriously, we are developing and testing fixes right now, and we will disclose our findings as soon as possible.

Unfortunately, the public discussion has also veered into vague claims and slanderous attacks. This is where the lack of transparency, the lack of respect, and the inflation of ego is damaging and unproductive. We had hoped for a better collaboration than this, and it makes me doubt the motives of the attackers. And yes, I make deliberate use of the word “attacker” here, because that’s what this is, an attack on Netgate and on the FreeBSD and pfSense communities. Beware of anyone who says that they have all the answers. I also worry about the integrity of those who make vague statements and blanket, over-the-top accusations.

Why did this blow up? It blew up because the attackers broke the process and procedure for progressing an open source project. Not just any project, but a well-established, solid operating system project. A project that should not be ruled by the “move fast and break things” process. It blew up because it surprised people who expected stability and gravitas. It blew up because of a disrespect for our developers, our testers, and our users. We at Netgate, and I personally, tried to engage their effort, only to be rebuked by them.

By following the normal, well understood security disclosure process this entire incident could have been handled quickly and efficiently through normal channels. We have yet to see a full description of the problems claimed; their choice to do a complete rewrite obscures the evidence of what they believe they were fixing, and they have yet to submit their work through the normal FreeBSD Phabricator process for review. That said, we do look forward to the bug reports and subsequent evaluation of the code through this review process. Code development is an iterative process, and one that we continue to strive to be better at. In the end, we will all benefit.

So what have I learned from this? I’ve learned to be a little less trusting. I’ve learned to be more proactive in defending against people who have ulterior motives. I’ve learned that people who emphatically say that they’re here to help often aren’t. This was definitely not the positive collaborative experience that I alluded to at the beginning of this blog. Does that mean that I don’t believe in community collaboration anymore? I hope not. Enduring an attack this insidious needs the strength that comes from the community. We need everyone’s help to continue to improve both FreeBSD and the pfSense software and build a strong security community. We need to work together, be transparent, be respectful, and leave our egos at the door. We continue to be committed to quality, community, transparency, and security. Please join us in this effort.