Remote SSH Using Real IP and Bypass Active VPN Connection

EricJanuary 29, 2019August 25, 2019

Goal: Remotely SSH into a server behind an active VPN connection using the real gateway IP and bypass connecting through the VPN tunnel.

Motivation

There is a server on my LAN which establishes a non-fixed VPN connection at boot. I often need to remote-terminal into that server from my iPhone. A simple port-forwarding from the gateway router to the machine fails to allow an SSH handshake because ACK packets from the login request are routed through the VPN tunnel (e.g. tun0 not eth0). Since the ACK packets return via a different route, they are treated as unsolicited packets and are dropped.

Solution

What works is to simply create a virtual interface, assign it a static IP (which has the highest rule specificity¹), and route all egress packets originating from that IP through the default gateway.

Packet mangling and creative uses of lookup tables will probably fail because OpenVPN changes a lot of routing rules to force all traffic through the tun0 interface.

Given:

eth0 exists and is up
Gateway IP: 192.168.0.1
Machine IP: 192.168.0.106
DHCP range: 192.168.0.100~254
Virtual IP: 192.168.0.6 (outside DHCP range)

First, create a virtual interface.

ip addr add 192.168.0.6 dev eth0:0

1	ip addr add 192.168.0.6 dev eth0:0

Next, assign a rule that all packets from the virtual IP are to use a routing lookup table.

ip rule add from 192.168.0.6 table 1234

1	ip rule add from 192.168.0.6 table 1234

After that, create a single route in the lookup table that requires every packet to leave the machine through the gateway via the virtual interface.

ip route add default via 192.168.0.1 dev eth0:0 table 1234

1	ip route add default via 192.168.0.1 dev eth0:0 table 1234

The table number can be any number as long as it is not a reserved table number. On Ubuntu machines you can cat /etc/iproute2/rt_table to be sure.

As a bonus, protect the SSH port (e.g. 22) by dropping packets from the VPN tunnel.

iptables -A INPUT -i tun0 -p tcp -m tcp --dport 22 -j DROP

1	iptables -A INPUT -i tun0 -p tcp -m tcp --dport 22 -j DROP

Finally, set up the gateway router for port forwarding. I have the external port 22106 forwarding to port 22 on the machine with virtual adapter static IP 192.168.0.6.

If all goes well, you will see the new virtual adapter with the static IP when you run ip -c a (The -c colors the output).

Virtual adapter with static IP outside DHCP range

Warning: You may be tempted to avoid all the virtual adapter instantiation with a high-specificity routing rule like ip rule add from 192.168.0.106 table 1234 and ip route add default via 192.168.0.1 dev eth0 table 1234. You will be able to SSH into the machine from the WAN on the real IP (with proper gateway port-forwarding), but you might not be able to SSH into the machine from the LAN.

Persistent Routes

On Ubuntu I created a script at /etc/network/if-up.d/eth0 which will run when interfaces are raised.

This script has atomic execution using mkdir in case several adapters are brought up simultaneously, and it also checks that the rules have not been added already. It will figure out the gateway IP, create the virtual adapter eth0:0, and assign it the IP of the gateway, but the last octet is user-supplied. For example, if the gateway IP is 192.168.0.1 and the supplied final octet is 6, then the virtual adapter will have an IP of 192.168.0.6. The script also performs logging to /var/log/eth0.log. Finally, it blocks SSH access from the VPN tunnel (because you don’t want to be back-hacked).

Here is the raw script:

#!/bin/bash
# File: /etc/network/if-up.d/eth0
# chmod +x /etc/network/if-up.d/eth0
# Copyright Eric Draken, 2019

# Atomic lock
LOCK=/var/lock/.eth0.exclusivelock
if mkdir $LOCK > /dev/null 2>&1
then

    # User settings
    SSH_PORT=22
    DEV=eth0:0
    TABLE=201
    VIRT_LAST_OCTET=6
    LOG="/var/log/$(basename $BASH_SOURCE).log"

    # Get the gateway IP
    GATEWAY_IP=$(/sbin/ip route | awk '/default/ { print $3 }')

    # Create the virtual adapter IP from the /24 subnet and last octet
    VIRT_IP="$(echo $GATEWAY_IP | cut -d"." -f1-3).${VIRT_LAST_OCTET}"

    # Only run this script once
    if [ ! -z "$(/sbin/ip a | grep $VIRT_IP)" ]; then
        echo "$(date +"%m-%d-%y %T") skipping recreation of ${DEV}" >> $LOG
    else
        echo "$(date +"%m-%d-%y %T") ${DEV} up" >> $LOG

        # Log helper
        exe() { echo "\$ $@" >> $LOG ; "$@" ; }

        # Create a virtual adpater
        exe ip addr add $VIRT_IP dev $DEV

        # Add persist routes
        exe ip rule add from $VIRT_IP table $TABLE
        exe ip route add default via $GATEWAY_IP dev $DEV table $TABLE

        # Disable VPN ssh access
        exe iptables -A INPUT -i tun0 -p tcp -m tcp --dport $SSH_PORT -j DROP
    fi
fi

# Ensure the 'lock' is always released
rmdir $LOCK > /dev/null 2>&1

#!/bin/bash

# File: /etc/network/if-up.d/eth0

# chmod +x /etc/network/if-up.d/eth0

# Copyright Eric Draken, 2019

# Atomic lock

LOCK=/var/lock/.eth0.exclusivelock

if mkdir $LOCK > /dev/null 2>&1

then

# User settings

SSH_PORT=22

DEV=eth0:0

TABLE=201

VIRT_LAST_OCTET=6

LOG="/var/log/$(basename $BASH_SOURCE).log"

# Get the gateway IP

GATEWAY_IP=$(/sbin/ip route | awk '/default/ { print $3 }')

# Create the virtual adapter IP from the /24 subnet and last octet

VIRT_IP="$(echo $GATEWAY_IP | cut -d"." -f1-3).${VIRT_LAST_OCTET}"

# Only run this script once

if [ ! -z "$(/sbin/ip a | grep $VIRT_IP)" ]; then

echo "$(date +"%m-%d-%y %T") skipping recreation of ${DEV}" >> $LOG

else

echo "$(date +"%m-%d-%y %T") ${DEV} up" >> $LOG

# Log helper

exe() { echo "\$ $@" >> $LOG ; "$@" ; }

# Create a virtual adpater

exe ip addr add $VIRT_IP dev $DEV

# Add persist routes

exe ip rule add from $VIRT_IP table $TABLE

exe ip route add default via $GATEWAY_IP dev $DEV table $TABLE

# Disable VPN ssh access

exe iptables -A INPUT -i tun0 -p tcp -m tcp --dport $SSH_PORT -j DROP

# Ensure the 'lock' is always released

rmdir $LOCK > /dev/null 2>&1

Here is a sample log file output:

Now we can SSH into the machine from the LAN and the WAN. Though, there are some lesser-known aspects about the above script. For example, ${0##*/} returns the script file name, and if mkdir folder is an atomic operation preventing a race condition. By prefixing commands with the exe function, commands are logged and then executed to see what variable substitution took place.

Bonus: Automatically restart the OpenVPN daemon on tunnel failure

Here’s a useful systemd-system script I wrote to periodically ping a nameserver, and upon tunnel failure restart the OpenVPN client service.

#!/bin/bash
# Desc: Ping a nameserver and if failed, restart the VPN service
# File: nano /etc/openvpn/checkopenvpn.sh
# chmod +x /etc/openvpn/checkopenvpn.sh
# crontab -e
# Add this line to check every five minutes
#    */5 * * * * /etc/openvpn/checkopenvpn.sh >> /var/log/checkopenvpn.log 2>&1
# See cron logs
# grep -i cron /var/log/syslog
# Copyright Eric Draken, 2019

# CloudFlare nameserver
IP=1.1.1.1
RESULT=$(ping -c 8 -W 2 $IP | grep received | awk '{print $4}')
if [ $RESULT -eq 0 ]; then
    echo "OpenVPN tunnel failed"

    # Check if the OpenVPN daemon is running
    ps aux | grep [o]penvpn > /dev/null 2>&1
    if [ $? -eq 0 ]; then

        # Restart the VPN service if it is already started
        systemctl restart openvpn

        if [ $? -eq 0 ]; then
            echo "$(date) - OpenVPN restarted"
        else
            echo "$(date) - OpenVPN restart failed. See logs"
        fi
    fi
fi

# REF: https://forum.htpcguides.com/Thread-Monitor-linux-openvpn-daemon-and-restart-if-disconnected

#!/bin/bash

# Desc: Ping a nameserver and if failed, restart the VPN service

# File: nano /etc/openvpn/checkopenvpn.sh

# chmod +x /etc/openvpn/checkopenvpn.sh

# crontab -e

# Add this line to check every five minutes

# */5 * * * * /etc/openvpn/checkopenvpn.sh >> /var/log/checkopenvpn.log 2>&1

# See cron logs

# grep -i cron /var/log/syslog

# CloudFlare nameserver

IP=1.1.1.1

RESULT=$(ping -c 8 -W 2 $IP | grep received | awk '{print $4}')

if [ $RESULT -eq 0 ]; then

echo "OpenVPN tunnel failed"

# Check if the OpenVPN daemon is running

ps aux | grep [o]penvpn > /dev/null 2>&1

if [ $? -eq 0 ]; then

# Restart the VPN service if it is already started

systemctl restart openvpn

if [ $? -eq 0 ]; then

echo "$(date) - OpenVPN restarted"

else

echo "$(date) - OpenVPN restart failed. See logs"

# REF: https://forum.htpcguides.com/Thread-Monitor-linux-openvpn-daemon-and-restart-if-disconnected

References:

Unable to SSH from WAN into LAN device via real IP when OpenVPN is running

Notes:

This means that specifying a specific IP address in the routing rules has the highest precedence, even higher than the OpenVPN rules. ↩