Getting over CGNAT (Wireguard + GCE)

First, let me ask you a question.

Did you spend the last couple of hours trying to figure out why port forwarding isn’t working?
Why is your public ip different on your router and on sites like these?

The answer is that your isp sucks…
Or more realistically, is likely a result of ipv4 address exhaustion showing its effects. Thankfully though, this predicament was foreseen many years ago and a solution is already here, ipv6, which if your isp allows can be a viable alternative that i will not be covering here.
However… Some internet service providers fail to implement ipv6 on their systems and instead resort to CGNAT, which basically means you and dozens of other people in your city with the same isp were assigned the same ip address, and traffic is forwarded to your home much like a router will forward traffic to your computer. CGNAT also has the side effect of adding more hops to your traffic making connections slower, so if possible, i would recommend switching isps instead.

There are many solutions to get around this problem like ssh, ngrok, serveo, onion addresses, ipv6 and more. But for this particular guide we will be using a wireguard vpn.

For this guide you will need:

  • Some linux know how.
  • A virtual machine with a publicly accesible ip.
    In the spirit of keeping this guide free we’ll be signing up for a google cloud account and getting a free GCE. But if you dont want to get in bed with google you can use a paid vps or wireguard service online.
  • Wireguard.
Google Compute Engine (GCE)

Ok, so first things first, we will be getting a google cloud account, which you can do here. NOTE You will need a credit card but you will not be charged, not now or ever, it even has a note in the registration page letting you know you won’t be automatically debited when the first year runs out.
Once you have your account set up, navigate to compute engine - vm instances then click on the top plus sign and create a new instance. Give it a name, Select the Region closest to you, for the machine type you can select micro or small (we don’t need much since this machine will only forward traffic), Select Debian for the linux distro, and check both tickboxes to allow http and https on the firewall, open the advanced options and add your ssh key under the security tab, and finally go on the networking tab change the ip forwarding box to on then click on edit network interfaces - and under external ip select ‘Create IP Address’ with the premium tickbox and follow the steps. This Virtual Machine will act as the frontend of the vpn tunnel to your machine thats sitting behind the cgnat. It will redirect all traffic that gets to the external ip you just set-up above into your machine at the other end of the tunnel.

Wireguard

Its time to set-up wireguard on both machines. Wireguard is an extremely simple and lightweight vpn that aims to replace ipsec and openvpn, it’s still a work in progress and shouldn’t be relied on for company solutions, but for our purposes as self hosted homelabbers it will do just fine. This part of the guide for setting up wireguard is heavily based on this other guide by angristan, the guy who maintains the openvpn-install scripts.

Server Set-up

We’ll start by setting it up on the GCE running debian, log in to your virtual machine either by using ssh on a terminal or connecting through the cloud console management website. Then type or copy the following commands one by one to install wireguard.

 sudo -i
 echo "deb http://deb.debian.org/debian/ unstable main" > /etc/apt/sources.list.d/unstable.list
 printf 'Package: *\nPin: release a=unstable\nPin-Priority: 90\n' > /etc/apt/preferences.d/limit-unstable
 apt update
 apt install wireguard

what this commands do in order is:

  • sudo -i will change your user to root so all the following commands will run as administrator. press ctrl-d, or type logout after running the commands to get back to regular user.
  • echo and printf commands will add the unstable repository to debian. think of this as adding an extended store to download apps from.
  • apt update and install will update the list with the apps from the newly added repository and then install wireguard.

Now that wireguard is installed we need to set up its interface. For that we will create a file /etc/wireguard/wg0.conf where we can specify the configuration options, you can do a custom configuration or use the following template.

Important note: There’s basically two ways of going about this whole thing:

Options Pros Cons
Option a You can see external ips on your logs ALL traffic will be forwarded through the vpn
Option b You wont be able to see external ips on logs Only the services traffic will use the vpn

I’ll be using option a in this guide but i’ll leave some notes at the end for those who want to go with option b.

[Interface]
Address = 10.66.66.1/24
ListenPort = 1194
PrivateKey = <Server_private_key>
PostUp = iptables -t nat -A PREROUTING -p tcp -i eth0 '!' --dport 22 -j DNAT --to-destination 10.66.66.2; iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source <local_interface_ip>
PostDown = iptables -t nat -D PREROUTING -p tcp -i eth0 '!' --dport 22 -j DNAT --to-destination 10.66.66.2; iptables -t nat -D POSTROUTING -o eth0 -j SNAT --to-source <local_interface_ip>

[Peer]
PublicKey = <peer_public_key>
AllowedIPs = 10.66.66.2/32

Explanation and steps:

  • To create this file you can use a basic terminal text editor like nano to type the template in or copy it by running sudo nano /etc/wireguard/wg0.conf. To close nano first save with ctrl-o then exit with ctrl-x.
  • Address specifies the ip address that we’re assigning to the GCE on the vpn, for this example i’m using 10.66.66.1/24 (you can also use ipv6 but i won’t, if you’re interested in that check out angristans blog linked above).
  • PrivateKey Is the private key for the server which you can generate by running the following command on the terminal wg genkey then copy the key given and replace <Server_private_key> on the conf file with it.
  • PostUp/PostDown are commands to be executed after the interface comes up and after its taken down respectively. The commands i included in there are iptables commands that will redirect all incoming traffic hitting the GCE to 10.66.66.2 (the ip we’ll assign to the natted server soon) except for port 22 (which is ssh, this is so you can still ssh into the vm normally) and to mask outgoing traffic to the GCE ip (this is so that nat works, i won’t be going in depth into how this works but just know its needed).
    NOTE change the <local_interface_ip> at the end of the PostUp/Down commands to the output of the following command: ip a s | grep eth0 | tail -n 1 | awk '{print $2}' | sed 's/\/32//'.
  • [Peer]PublicKey We will be coming back to this very soon after setting up wireguard on the natted server.
  • [Peer]AllowedIPs The ip that we will be assigning to the natted server in the vpn.

So far all we’ve done is configure the interface but we haven’t brought it up yet, to do that enable and start the wireguard interface using the following systemd commands:

sudo systemctl enable wg-quick@wg0.service
sudo systemctl start wg-quick@wg0.service

Once thats done type in sudo wg show in the terminal and save the public key shown in there.

Peer Set-up

Once again install wireguard now on your local machine behind the cgnat, i’m using arch btw, but you can install it on pretty much any distro and even windows. Instructions here.
If you’re using arch like me, installing wireguard is as simple as typing this command:
sudo pacman -Syu wireguard-tools.
Once again create a file called /etc/wireguard/wg0.conf and paste in the following.

[Interface]
PrivateKey = <Server_private_key>
Address = 10.66.66.2/24
DNS = 8.8.8.8

[Peer]
PublicKey = <GCE_Public_key>
Endpoint = <GCE_Public_ip>:1194
AllowedIPs = 0.0.0.0/0

Explanation:

  • PrivateKey Generate another private key for your local Wireguard instance using wg genkey and replacing <Server_private_key> on the conf.
  • Address The address assigned to the local machine on the vpn.
  • DNS Since all traffic coming out of the local machine is being sent through the vpn you need to use a publicly accesible dns server if you weren’t before. In this example we’re setting it up to use googles dns server.
  • PublicKey Replace <GCE_Public_key> on the conf with the public key we saved earlier from the GCE.
  • Endpoint replace <GCE_Public_ip> on the conf with the public ip for the GCE you set up when creating the virtual machine on the google cloud console website.
  • AllowedIPs using 0.0.0.0/0 means that all traffic leaving the local machine will go through the vpn.

If you’re using docker and theres some containers you want to exclude from the vpn connection add something like the following:

PostUp = ip rule add from 172.20.0.120/29 table 200; ip route add default via 192.168.0.1 table 200
PostDown = ip rule delete from 172.20.0.120/29 table 200; ip route delete default via 192.168.0.1 table 200

Where every docker container between the ips 172.20.0.120 to 172.20.0.126 will be excluded from the vpn. this is also assuming your router’s ip is 192.168.0.1

Once all that is done, bring up the interface and get the public key from your local machine.

sudo systemctl enable wg-quick@wg0.service
sudo systemctl start wg-quick@wg0.service
sudo wg show

Copy the public key displayed with the last command and paste it on the configuration on the GCE replacing <peer_public_key>. Once that’s done restart the interface on both machines using sudo systemctl restart wg-quick@wg0.service and try to ping the GCE from your local machine using ping 10.66.66.1, if that works you’re set! connection has been established, now you can point your domain to the GCE public ip and it will redirect to your local machine sitting behind the cgnat.

Option b

Basically what we did is redirect all traffic from the GCE to the local machine and vice versa, so we could see external ips on the local machine logs. If you want to have more control over the traffic that gets sent through the vpn you’ll need to make a couple changes to the above configurations.

New Server Conf

[Interface]
Address = 10.66.66.1/24
ListenPort = 1194
PrivateKey = <Server_private_key>
PostUp = iptables -t nat -A PREROUTING -p tcp -i eth0 '!' --dport 22 -j DNAT --to-destination 10.66.66.2; iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source <local_interface_ip>; iptables -t nat -A POSTROUTING -o wg0 -j SNAT --to-source 10.66.66.1
PostDown = iptables -t nat -D PREROUTING -p tcp -i eth0 '!' --dport 22 -j DNAT --to-destination 10.66.66.2; iptables -t nat -D POSTROUTING -o eth0 -j SNAT --to-source <local_interface_ip>; iptables -t nat -D POSTROUTING -o wg0 -j SNAT --to-source 10.66.66.1

[Peer]
PublicKey = <peer_public_key>
AllowedIPs = 10.66.66.2/32

New Peer Conf

[Interface]
PrivateKey = <Server_private_key>
Address = 10.66.66.2/24
DNS = 8.8.8.8

[Peer]
PublicKey = <GCE_Public_key>
Endpoint = <GCE_Public_ip>:1194
AllowedIPs = 10.66.66.1/24

If you’re curious about the iptables commands and how i handle nat i will be doing a tutorial explaining these in-depth soon, after i’m done with this self-hosted guide series. Also, if you have any questions or problems leave a comment below i’ll try to help everyone. cheers!

Thank you for reading.