Bypass Network Lockdowns

Updated on

I manage the network infrastructure for a robotics club on campus. Unfortunately, the campus network has been heavily locked down for security. It takes years to get Information Services and Technology (IST) to agree to forward a port, and it’s even harder for student clubs.

Our club members need access to the robot through the open internet. This is especially important for new members who aren’t able to access the club’s room after hours.

Due to this, I’ve put considerable effort into bypassing the restrictions placed by the university. As it turns out, once you know what you’re doing, it’s pretty easy! All you need is outward access to the internet, from there you can easily get inward access to any port.

ZeroTier One

The easiest method, and probably recommended for most users, is using Software Driven WAN (SDWAN). This is similar to a VPN, but unlike Wireguard, ZeroTier One doesn’t1 require a centralized server.

Advantages:

  • Very simple setup
  • Very low latency through UDP hole-punching
  • Free (for up to 25 clients)

Drawbacks:

  • Requires installing separate software on all clients
  • Sometimes fails to connect for up to an hour… very hard to debug when it happens
  • Can only be used by clients on the VPN. For example, a public webserver won’t be able to use this
  • Limit of 25 clients (on the free version)

To use ZeroTier:

  1. Sign up for an account at zerotier.com
  2. Under “Networks” create a network and give it a name. Ensure Access Control is private.
  3. Install ZeroTier One on all clients you’d like to connect. You can add more later.
  4. On Linux, start and enable the ZeroTier one daemon with systemctl enable --now zerotier-one.service.
  5. Find your network ID in the online console, then join with sudo zerotier-cli join <network-id>.
  6. Back in the web console, click the checkmark next to the new client that joined, and give it a name.
  7. Now sudo zerotier-cli listnetworks should say OK PRIVATE for your network.
  8. Repeat steps 3-7 for every new client you add.

Now you should be able to use the IP listed in the web console for your device. This will be in the right column under “Managed IPs”. For example, if your server has a zerotier IP of 172.27.100.10 another client would be able to ping 172.27.100.10 or with an ssh daemon running ssh <user>@172.27.100.10.

[[1]]: ZeroTier One still requires a centralized server, but they have “root servers” which are available to the public for free, so no setup required on our part.

Reverse Proxy

Reverse proxies are a slightly more involved but generally better approach. They use a second computer to forward information to the internal network.

Advantages:

  • Can be accessed by the public
  • No additional software limits
  • No client limits

Drawbacks:

  • No UDP hole punching: might have a bit more latency
  • Requires a centralized server with a public IP (not necessarily static tho)

To setup a reverse proxy, you’ll need a computer with a public IP, that’s accessible behind the router. You could use your home computer, but it’ll need to always stay on. A more popular approach is grabbing a cheap cloud computer. I personally use the most basic droplets on Digital Ocean for $4/month. Be sure to choose the closest datacenter possible and ideally in the same country, to minimize latency.

I’ll be referring to the computer powering the reverse proxy as the “droplet”, but the steps can easily be applied on your own Linux machine.

Setting up a reverse proxy

  1. Build the rathole binary that’s compatible with your system. The droplet may require a different llvm target.
  2. Create a client-side rathole configuration on the server. The token can be any random string. Assuming the IP of the droplet is 172.27.10.10 and we’re using port 9001, the configuration might look like this:
[client]
remote_addr = "172.27.10.10:9001"
 
[client.services.ssh]
token = "yNx6KUuG4P-VRIkzr-NWysZSI6-;04IWQs4sG0;a"
local_addr = "127.0.0.1:22"
 
[client.services.webserver]
token = "C;KPmX-fYxAb;iGMwXEAl48woaYE;ey-CLapQJHs"
local_addr = "127.0.0.1:80"
 
[client.services.forgejo]
token = "-zUbD;Wy6v35KmO;GZLHs;51sws-;7o5bJ-TjSJs"
local_addr = "127.0.0.1:8080"
  1. Create a systemd service on the server called rathole_client.service. Edit the path of your rathole executable, and the argument to the config file above:
# /etc/systemd/system/rathole_client.service
[Unit]
Wants=network-online.target
After=network-online.target
 
[Service]
User=root
Group=root
Type=forking
Restart=on-failure
RestartSec=5s
KillMode=control-group
KillSignal=SIGTERM
ExecStart=/usr/local/bin/rathole /etc/rathole/client.toml
# You MUST edit the line above to point to your rathole binary
 
[Install]
WantedBy=multi-user.target
  1. On the droplet, create a server.toml. It might look something like this, matching the client.toml provided in step 2.
[server]
bind_addr = "0.0.0.0:9001"  # Must match port client is trying to contact
 
[server.services.ssh]
token = "yNx6KUuG4P-VRIkzr-NWysZSI6-;04IWQs4sG0;a"
bind_addr = "0.0.0.0:8022"  # Doesn't have to match client
 
[server.services.webserver]
token = "C;KPmX-fYxAb;iGMwXEAl48woaYE;ey-CLapQJHs"
bind_addr = "127.0.0.1:80"
 
[server.services.forgejo]
token = "-zUbD;Wy6v35KmO;GZLHs;51sws-;7o5bJ-TjSJs"
bind_addr = "127.0.0.1:443"
  1. On the droplet, you can simply start rathole manually in a tmux session with ./rathole server.toml.
  2. On the client run systemctl enable --now rathole_client.service

Check the output of both commands (systemctl status rathole_client.service) to make sure there aren’t any errors. Assuming it’s all good, you should be able to contract the server through the droplet. For example, if we wanted to ssh into the emiliko user on the server, the above config should allow that through:

ssh -p 8022 emiliko@172.27.10.10

This will be forwarded to port 22 on the server!

Reverse Proxy without a Static IP

If you’re planning to use a home computer, you’ll quickly find that most internet service providers do not offer static IPs for consumer plans. Luckily, there’s a very simple way around this: Domain Name Servers (DNS).

You will need a domain to achieve this. Domains should not cost more than $20/year. Here I’ll use the domain example.com as an example.

The idea is that a domain will point to a specific IP, but this IP is determined through a lookup to the DNS. This means that if we change the IP the DNS has every time our computer’s IP changes, we’ll appear to have a static IP!

First, put your nameservers on a good DNS provider. I use Cloudflare, it’s free and fast. You’ll need to find the DNS page. The URL will look something like:

https://dash.cloudflare.com/<long-project-id>/example.com/dns/records

There, you’ll want to add an “A Record”. The name will be the subdomain. So if my computer is called mycomputer and that’s in the name field, it’ll be accessible at mycomputer.example.com.

Now you need to identify your IP address. This is your PUBLIC IP address, not your LOCAL IP address. One easy way to do this is curl -q https://ifconfig.me/ip.

Make sure “Proxy Status” is OFF. Proxying the connection appears to make this whole idea break down very quickly, so don’t.

With that “A Record” set, try host mycomputer.example.com to see when the DNS updates. This can take up to 4 hour, but usually takes under a minute in practice. With this, you should be able to access your computer using the domain! Of course, make sure your router’s ports are forwarding to your computer.

We now need to make your computer update Cloudflare’s DNS, whenever the IP changes. I use the script below to do this. Fill in the HOST4, HOST6, TOKEN, ZONE_ID. The TOKEN is your Cloudflare application token:

#!/usr/bin/env bash
declare wan_ip_record wan_ip cf_records host_record cf_host_ip cf_rec_id
 
declare -r HOST4='mycomputer.example.com'
declare -r HOST6='mycomputer6.example.com'
declare -r TOKEN='CLOUDFLARE_TOKEN_HERE'
declare -r ZONE_ID='CLOUDFLARE_ZONEID_HERE'
 
utc_date() {
  date -u +'%Y-%m-%d_%H-%M-%S_UTC'
}
 
cf_update_ip() {
  local -r my_ip="$1"
  local -r cf_ip="$2"
  local -r cf_id="$3"
  local -r host="$4"
 
  if [[ -z "$my_ip" || "$my_ip" == null ]]; then
    echo "Failed to find local WAN ip: $my_ip" >&2
    return 1
  elif [[ -z "$cf_ip" || "$cf_ip" == null ]]; then
    echo "Failed to find content of A record for $host" >&2
    return 1
  elif [[ -z "$cf_id" || "$cf_id" == null ]]; then
    echo "Failed to find A record ID for $host" >&2
    return 1
  elif [[ "$my_ip" == "$cf_ip" ]]; then
    echo "Cloudflare for $host is up to date @ $(utc_date)" >&2
  else
    echo "Updating Cloudflare's for $host from $cf_ip to $my_ip" >&2
 
    patch_response="$(curl -s --request PATCH \
        --url "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records/${cf_id}" \
        --header 'Content-Type: application/json' \
        --header "Authorization: Bearer $TOKEN" \
        --data '{
        "comment": "'"${host} @ $(utc_date)"'",
        "content": "'"$my_ip"'",
        "name": "'"$host"'",
        "proxied": false,
        "ttl": 1
      }')"
 
    if [[ "$(echo "$patch_response" | jq -r '.success')" == true ]]; then
      echo "Update to $my_ip for $host succeeded @ $(utc_date)" >&2
    else
      echo "Failed to update $host. DUMP:"
      echo "$patch_response"
      return 1
    fi
  fi
}
 
#╔─────────────────────────────────────────────────────────────────────────────╗
#│ Gετ Λ rεcδrd δη Clδμdflαrε                                                  |
#╚─────────────────────────────────────────────────────────────────────────────╝
if ! cf_records="$(curl -s --request GET \
  --url https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/dns_records \
  --header 'Content-Type: application/json' \
  --header "Authorization: Bearer $TOKEN")"
then
  echo "Failed to retrive cloudflare zone records" >&2
  exit 1
fi
 
declare -r cf_ipv4_record="$(echo "$cf_records" |
  jq '.result[] | select(.name == "'"$HOST4"'")')"
declare -r cf_ipv4_ip="$(echo "$cf_ipv4_record" | jq --raw-output '.content')"
declare -r cf_ipv4_id="$(echo "$cf_ipv4_record" | jq --raw-output '.id')"
 
declare -r cf_ipv6_record="$(echo $cf_records |
  jq --raw-output '.result[] | select(.name == "'"$HOST6"'")')"
declare -r cf_ipv6_ip="$(echo "$cf_ipv6_record" | jq --raw-output '.content')"
declare -r cf_ipv6_id="$(echo "$cf_ipv6_record" | jq --raw-output '.id')"
 
#╔─────────────────────────────────────────────────────────────────────────────╗
#│ Sετ Λ rεcδrd τδ cμrrεητ WΛN                                                 |
#╚─────────────────────────────────────────────────────────────────────────────╝
if wan_ipv4="$(curl -s https://api4.ipify.org)"; then
  echo "ipv4 record: $wan_ipv4"
  cf_update_ip "$wan_ipv4" "$cf_ipv4_ip" "$cf_ipv4_id" "$HOST4"
else
  echo "Hosts timed out on ipv4. Not updating ipv4 A record" >&2
fi
 
if wan_ipv6="$(curl -s https://api6.ipify.org)"; then
  echo "ipv6 record: $wan_ipv6"
  cf_update_ip "$wan_ipv6" "$cf_ipv6_ip" "$cf_ipv6_id" "$HOST6"
else
  echo "Hosts timed out on ipv6. Not updating ipv6 AAAA record" >&2
fi

Now we need a systemd-timer to run this script. I run it once every 15 minutes. Please refer to the systemd-timers blog for more information, but breifly I use:

[Unit]
Wants=update_a_record.timer
Wants=network-online.target
After=network-online.target
 
[Service]
Type=oneshot
ExecStart=/usr/local/bin/set_a_records.sh

Timer:

[Unit]
Requires=update_a_record.service
 
[Timer]
Unit=update_a_record.service
OnCalendar=*-*-* *:00,15,30,45:00
RandomizedDelaySec=15min
 
[Install]
WantedBy=timers.target

Then start it with systemctl enable update_a_record.service. The name of the service will be different based on what you called the files.

Now to use this for reverse-proxy purposes, go back to your client.toml config and update the remote_addr:

[client]
remote_addr = "mycomputer.example.com:9001"

Multiple Services

Unless you have a lot of funds, you’ll likely only have one server running. However, you may want several “services” running on this one server. This is especially an issue when it comes to websites, as browsers request to port 80 or 443 (https) on the given IP.

Luckily, when a website is requested, he header of that request includes the domain name that request is going to. For example, I might have example1.com and example2.com pointing to the exact same IP address. However, the server will be able to see if the request is coming for example1.com or example2.com and choose to serve different content.

Nginx is the typical way to do this, but Caddy is much easier to get running. It almost provides extremely simple Let’s Encrypt integration for free https.

Here’s a very basic Caddyfile we use at arvp:

git.mami2.moe {
	reverse_proxy localhost:9123
}
 
woodpecker.mami2.moe {
	reverse_proxy localhost:9027
}
 
jupyter.mami2.moe {
	reverse_proxy localhost:9025
}
 
# This one keeps a log of connections
cvat.mami2.moe {
	reverse_proxy localhost:9026
	log {
		output file /root/cvat.mami2.moe.caddy.log {
			roll_size 1gb
			roll_keep 5
			roll_keep_for 720h
		}
	}
}
 
# This one uses basic password protection, provided by caddy
llama.mami2.moe {
	basicauth * {
		arvp $2y$03$BekY89f5/9s.oxtrGntlk23j4kl32jlk;23jl4j32l;23j4l32kY.
	}
	reverse_proxy localhost:9030
}