Remote SSH Using Real IP and Bypass Active VPN Connection

Goal: Remotely SSH into a server behind an active VPN connection using the real gateway IP and bypass connecting through the VPN tunnel.

Motivation

There is a server on my LAN which establishes a non-fixed VPN connection at boot. I often need to remote-terminal into that server from my iPhone. A simple port-forwarding from the gateway router to the machine fails to allow an SSH handshake because ACK packets from the login request are routed through the VPN tunnel (e.g. tun0 not eth0). Since the ACK packets return via a different route, they are treated as unsolicited packets and are dropped.

Solution

What works is to simply create a virtual interface, assign it a static IP (which has the highest rule specificity1), and route all egress packets originating from that IP through the default gateway.

Packet mangling and creative uses of lookup tables will probably fail because OpenVPN changes a lot of routing rules to force all traffic through the tun0 interface.

Given:

  • eth0 exists and is up
  • Gateway IP: 192.168.0.1
  • Machine IP: 192.168.0.106
  • DHCP range: 192.168.0.100~254
  • Virtual IP: 192.168.0.6 (outside DHCP range)

First, create a virtual interface.

Next, assign a rule that all packets from the virtual IP are to use a routing lookup table.

After that, create a single route in the lookup table that requires every packet to leave the machine through the gateway via the virtual interface.

The table number can be any number as long as it is not a reserved table number. On Ubuntu machines you can cat /etc/iproute2/rt_table to be sure.

As a bonus, protect the SSH port (e.g. 22) by dropping packets from the VPN tunnel.

Finally, set up the gateway router for port forwarding. I have the external port 22106 forwarding to port 22 on the machine with virtual adapter static IP 192.168.0.6.

Port forwarding on the gateway router
Port forwarding on the gateway router

If all goes well, you will see the new virtual adapter with the static IP when you run ip -c a (The -c colors the output).

Virtual adapter with static IP outside DHCP range
Virtual adapter with static IP outside DHCP range
Warning: You may be tempted to avoid all the virtual adapter instantiation with a high-specificity routing rule like ip rule add from 192.168.0.106 table 1234 and ip route add default via 192.168.0.1 dev eth0 table 1234. You will be able to SSH into the machine from the WAN on the real IP (with proper gateway port-forwarding), but you might not be able to SSH into the machine from the LAN.

Persistent Routes

On Ubuntu I created a script at /etc/network/if-up.d/eth0 which will run when interfaces are raised.

This script has atomic execution using mkdir in case several adapters are brought up simultaneously, and it also checks that the rules have not been added already. It will figure out the gateway IP, create the virtual adapter eth0:0, and assign it the IP of the gateway, but the last octet is user-supplied. For example, if the gateway IP is 192.168.0.1 and the supplied final octet is 6, then the virtual adapter will have an IP of 192.168.0.6. The script also performs logging to /var/log/eth0.log. Finally, it blocks SSH access from the VPN tunnel (because you don’t want to be back-hacked).

Colorized VPN SSH access script
Colorized VPN SSH access script

Here is the raw script:

Here is a sample log file output:

Sample log file output
Sample log file output
Now we can SSH into the machine from the LAN and the WAN. Though, there are some lesser-known aspects about the above script. For example, ${0##*/} returns the script file name, and if mkdir folder is an atomic operation preventing a race condition. By prefixing commands with the exe function, commands are logged and then executed to see what variable substitution took place.

Bonus: Automatically restart the OpenVPN daemon on tunnel failure

Here’s a useful systemd-system script I wrote to periodically ping a nameserver, and upon tunnel failure restart the OpenVPN client service.

References:

Notes:

  1. This means that specifying a specific IP address in the routing rules has the highest precedence, even higher than the OpenVPN rules.