Multisite Wireguard VPN Mesh with OpenBSD

0. Introduction

For many years, I relied on OpenVPN to connect distant sites, configuration was so hectic that I ended up writing my own tool to automate the process. As I'm often working in different (but fix) places - I have been considering setting up a mesh VPN for some time - but it always looked quite complex to manage.
With the rise of Wireguard, its small attack surface and easy configuration, I have been using it for over a year and am very happy. As configuration sligthly changed with OpenBSD 6.9 (it is even easier to setup now) - the mesh VPN now looks like something I could achieve! Actually I'm not sure I will get it working, but I'm sure that if I don't start documenting the process from the start, I will never do so.
In this post, I will wander through my ideas, my first tests look promising, but with every answer come many questions... so let's start here!

The idea:

The idea of the feasability and requirements came to my mind while reading ianix.com Howto: WireGuard on OpenBSD:

1. The keys don't have to be generated separately on server and client like indicated on most tutorials. Let's create all files centrally, and call each member of the mesh a node
2. For each node need to create:
   1. a wg0.conf server file
   2. a hostname.wg0 server file
   3. various wgX.conf client files
   4. various hostname.wgx client files
   5. pf rules
   6. routing rules

1. First prototype:

One late night (yesterday), I did a short shellscript, as a prototype... it looks promising - but is still incomple and "probably" has hidden bugs!
Running it is fast and straightforward as you can see:

Here is an overview of generated configuration files:

The files in light grey are not generated to avoid routing hell (aka host2 connects to host4 as 10.0.4.2 and host4 connects to host2 as 10.0.2.4)!
To illustrate this, let's look at an image, with arrows representing client connections.
The smaller the host number, the most incoming VPN connections to the mesh, the highest the host number, the more outgoing VPN connections to the mesh.
This "algorithm" is the simplest and most efficient I could think of: just put your most reliable sites in front of the list when configuring.

2. routing and pf:

Shellcheck almost agrees with my code now 🕺
I wasn't sure about how to manage pf nor routing rules.
Then the solution came to my mind while taking a bath... it would actually be very easy if I use predictible IP ranges for my internal subnets!

It all works alone because I work on single digits in the code, with $clientcount and $servcount. This is also why there is a limit to 9 nodes: at 10 we would be routing 172.210.X (public range) - and hard to debug issues might popup in your network at some point in the future! So if you want to use more than 9 nodes, either remove the 172.2X range from the code (254 max nodes then) - OR update the whole code to be able to use 172.16/12 (more work)

1. Regarding routing, at some point, I thought about using obgpd or ospfd - but then looking at the big picture, I realised that it was not necessary at all: as long as a node is running, every other node has a direct route to it! This means I can just add static routes to my inner networks, and not care about state: if a node is down route won't work - if it's up it will - + my routes are static and predictable (cfr ip adressing)

2. Regarding packet filtering:
   Let's consider the following image representing the typical network:
   
   It is clear that only a restricted number of machines on this network should be able to communicate with the network behind node2. Now it would be foolish to try to filter that traffic on the wg interfaces: indeed we don't want to filter our VPN, we want to filter what gets on it.
   This is why I decided to put wg* interfaces on "pass in quick, pass out quick" - but enable acces to the VPN to some hosts through authpf. In this way, the sysadmin (me), or the backup server have a temporary pass in rule when they need it. This feels much safer - and no need to filter traffic on wg interfaces.
   

Update 20210829 Squashed a few bugs, you can now get wgenconf from bitbucket.

Note: for the domain names, I use ddclient, which allows me to have hostnames on dynamic IP.
As you can see, it generates a clear hierarchy of files for each server, from there, you just scp the tar.gz on the right node, cp hostname.wg* and the wireguard folder in /etc - include the pf lines where they belong in your pf.conf (can't be automated) - and add the contents of routing.info to /etc/rc.local - either restart services or reboot and this node is up.

I hope it helps ;)

3. Conclusion:

I was absolutely amazed by the ease of finishing this script. It is one of the first scripts I ever write in ksh - and while I expected this project to keep me busy for a couple of weeks, it only lasted for a couple of days. The fact that I was able to do this in two days with only 60 lines of code and that it only takes a few seconds to generate a complete mesh config (keys, ip, routing, filtering) with a single command - is a testimony to the high quality of the work of the OpenBSD team and Jason A. Donenfeld (wireguard's developer): Thank you!