nebula - the VPN you never knew you wanted

March 15, 2022 9-minute read

So you're out looking for a VPN for all your networking needs and somehow stumbled across this page. Well let me present Nebula to you!

Nebula is an overlay network which allows you to join machines into a single virtual network. It is also incredibly simple to set up and understand. The underlying cryptographical protocol is the same as WireGuard (Noise Protocol Framework) with the advantage of being easier to manage.

This article will explain the core concepts behind Nebulas architecture and get you started in setting up your own network. I will also compare it to WireGuard in some places, and elaborate why you would want to use Nebula in its place.

VPN Basics

If you already know what a VPN is (not the YouTube sponsor kind), feel free to skip this section

A VPN (Virtual Private Network) is a rather loosely defined term in the space of network engineering. On the most basic level, it provides a way to interconnect multiple machines without having to physically wire them together.

Even from this basic example, multiple use cases arise:

Connecting geographically distributed datacenters
Connecting clients to a protected network
Tunneling internet traffic through a designated gateway or firewall

This last use case has become the most widely spread meaning for VPN. It is also possible to build something like this using Nebula, however the focus of this article lies in the use case of connecting multiple datacenters or clouds.

From a technical perspective, VPNs work by encapsulating network packets on a higher level. To facilitate this, a network device is created and configured using routing rules. Whenever the interface receives a packet from the host, it encapsulates it into its own application level data packet and sends it of to its destination. The destination has to run the VPN software as well. On the receiving side, the packet is unpacked and sent from the virtual interface to the operating system.

This of course results in an overhead on top of regular IP communication but in most cases, this performance penalty is negligible.

How Nebula works

Nebula is a so called Mesh Network in which nodes talk directly to each other without passing a central gateway. This works by clever exploitation of networking properties.

A regular networking flow, which happens thousands of times per day looks (absurdly simplified) like this:

When establishing a connection, the client opens a port for responses from the server. Firewalls and routers between the client and server know about this process and ensure that traffic back to the client is permitted.

Nebula uses this open port to cleverly traverse firewalls and connect clients directly to each other. A special node, called Lighthouse in nebula terminology, is used to have clients discover each other. This special node is the only network member which has to have a publicly reachable IP address. The client first connects to the lighthouse and opens a local port. The lighthouse however, does not directly respond but rather instructs the target client to respond on the newly opened port. This process is repeated for the other side as well.

Please note that this is a very simplified diagram. In reality, a lot more spoofing and trickery is going on to enable seamless NAT traversal.

Authentication

One big advantage over WireGuard, is the way Nebula handles authentication. WireGuard expects the administrator to integrate the public key of new clients into the configuration of each existing node. While being simple in theory, this approach does not scale very well and requires additional tooling to automatically roll out new nodes. Tailscale and consorts are good ways to manage this complexity but come at the price of introducing additional services and abstractions.

Nebula utilizes Public Key Infrastructure (PKI) concepts to address this issue. A central certificate authority (CA) signs certificates of clients. These certificates are used for cryptographic purposes as well as administrative ones. The signature contains the IP address of the node, as well as other metadata like allowed subnets and groups.

Whenever a new client registers with the lighthouse, it verifies the certificate with the certificate authority and remembers the IP address stored in the certificate. Since the CA itself does not change, the lighthouse can verify clients which have been created after it has been configured, and it does not need to be adapted to these changes.

Our example setup

The setup we're discussing in this article is as follows:

earth: On-premise server, located at my house, connected to the internet via a consumer ISP
moon: VPS located in a nearby datacenter. Has its own public IP and will be our lighthouse
jupiter[01:04]: VPS running on a cloud provider. They do not have permanent public IP addresses

If you want to follow along, you can use any machines you have access to. Google Cloud, Azure, AWS and Oracle Cloud all have free tiers for you to play around with.

Our goal is to have each machine on the same network, for reasons coming up in a future blog post.

Setting up Nebula

Nebula is provided as a statically compiled binary and can be downloaded on the official GitHub releases page. The release contains two important files: nebula and nebula-cert. To install these, please follow the usual procedure for your OS. Most likely you will want to these two binaries to a directory contained in your $PATH.

Certificates

Our first step is to set up the certificate authority:

nebula-cert ca -name univer.ze

This is the minimal command creates a new certificate authority with the name univer.ze. This CA on its own does not do very much. Make sure to keep the ca.key and ca.crt in a safe place as these files will allow you to onboard new clients onto your network. At the creation stage, it is also possible to restrict a few variables like IP subnets or groups. For this example, a minimal CA allows for the most flexibility.

Now onto registering the clients. Let's start with the lighthouse aka moon.univer.ze.

nebula-cert sign \
  -name "moon.univer.ze" \
  -ip "172.30.42.1/24" \
  -out-crt "moon.crt" \
  -out-key "moon.key" \
  -groups "trusted"

This command contains a lot of information. Here's a breakdown:

Parameter	Description
name	Friendly name of the client. Does not have to be the actual hostname of the target machine
ip	Designated IP address inside the VPN. You can freely choose any address here, but you have to keep track of the addresses yourself
out-crt	Output location for the certificate
out-key	Output location for the key
groups	Client groups. This is used for advanced firewall rules

Make sure you execute the command in the same directory as the ca.key and ca.crt files or specify these files using the ca-key and ca-crt parameters.

If you provide services to external users and want to introduce an additional layer of security, you can have them send you a public certificate generated by nebula-cert keygen and sign it on your machine. This way, you never see the private key.

After setting up the certificates for the lighthouse, we have to perform the same procedure for the clients.

Configuration

Now that our certificates are ready, we can begin configuring the machines. The first step is to download and install the nebula binary on the system. The second component is the configuration file.

Nebula is configured using a yaml file and an example can be found project repository. This example is very well documented and describes specific use cases for most options. Below is minimum configuration for our setup, but I urge you to read the configuration file yourself.

# Configuration file for the lighthouse moon.univer.ze
---
pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/moon.crt
  key: /etc/nebula/moon.key

lighthouse:
  am_lighthouse: true

listen:
  host: "[::]"
  port: 4242

tun:
  dev: nebula1

firewall:
  outbound:
    - port: any
      proto: any
      host: any
  inbound:
    - port: any
      proto: any
      groups:
        - trusted

If our lighthouse should not be a member itself, the tun interface can also be disabled. In this mode, the lighthouse is only used for discovery.

The firewall section can be used to restrict incoming and outgoing traffic. This is also the reason one can assign groups to hosts during the signing process. For this example, we allow all outbound traffic and only allow inbound traffic from the trusted group.

To run the lighthouse, execute nebula -config config.yaml as a user with enough access rights. Templates for service managers can be found here.

Pro Tip: If you add the cap_net_admin to the nebula executable, Nebula can be run as any user. The command to add this capability is: setcap cap_net_admin+ep nebula

Client Configuration

Now that the lighthouse is up and running, let's take a look at the client configuration file:

# Configuration file for the client earth.univer.ze
---
pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/host.crt
  key: /etc/nebula/host.key

static_host_map:
  "172.30.42.1": ["203.0.113.42:4242"]

lighthouse:
  am_lighthouse: false
  hosts:
    - '172.30.42.1'

listen:
  host: '0.0.0.0'
  port: 0
punchy:
  punch: true
tun:
  dev: nebula1

firewall:
  outbound:
    - port: any
      proto: any
      host: any
  inbound:
    - port: any
      proto: any
      groups:
        - trusted

While most sections are pretty straight forward, the static_host_map requires some further explanation. 203.0.113.42 is the publicly routable IP address of moon and 172.30.42.1 its internal one. The static host map is used to create a mapping between internal and external addresses. In our case, only the lighthouse IP has to be entered here. If multiple machines are reachable from the public internet, we could enter them here to gain a small performance boost as the lighthouse is not needed for the link setup.

In the lighthouse section, we can specify any number of lighthouse addresses. It is important to enter the internal address here rather than the external one. The mapping between internal and external addresses happens in the static_host_map block.

Two other interesting things happen in this config. The special port value 0 tells Nebula that this is a roaming node and dynamic port assignments shall be used. Another related setting is punchy. Behind this innocent name lies a subsystem designed to keep firewall NAT mappings alive by periodically "punching" new connections to the firewall. Further tuning options are available for more complex NAT setups.

Running the client works the same way as starting the lighthouse server:

nebula -config config.yaml

And just like that, the two hosts are able to reach each other!

[fedora@earth ~]$ ping 172.30.42.1
PING 172.30.42.1 (172.30.42.1) 56(84) bytes of data.
64 bytes from 172.30.42.1: icmp_seq=1 ttl=64 time=0.449 ms

[fedora@moon ~]$ ping 172.30.42.2
PING 172.30.42.2 (172.30.42.2) 56(84) bytes of data.
64 bytes from 172.30.42.2: icmp_seq=1 ttl=64 time=0.683 ms

The configuration for the jupiter nodes is exactly the same, and left out in this article for brevity.

Conclusion

Now that our network is up and running, we can see the mesh in action. Communication between the jupiter nodes is much faster and has lower latency while communication across the internet takes a bit more time. In a classical VPN setup, all traffic, regardless of hierarchy, would have gone through the lighthouse which would further slow down communications.

Of course there are a lot of more things you can do with Nebula (for example adding non-Nebula hosts) but this post should be enough to get you started and figure out the rest for yourself. Hopefully I convinced you to try out Nebula for yourself and simplify your network.