Two uplinks, one box, and the reply going out the wrong interface

Box has two uplinks. Traffic arrives on eth1 from the second ISP. The reply consults the routing table, finds the default route, and leaves via eth0 — the first ISP — carrying a source address that belongs to the second ISP’s block.

The first ISP’s edge router sees a packet with a source address it doesn’t own, decides it’s spoofed, and drops it. From the client’s perspective, the SYN went out and nothing came back.

The routing table isn’t broken. It just answers a question you didn’t need answered.

Why the normal table can’t do this

ip route is a destination-based lookup. Given a destination, it returns a route. It has no way to express “reply on the interface the request arrived on,” because it never sees the request — it only sees where the packet is going.

What you need is a decision based on the source address. That’s policy routing, and Linux has had it built in since the 2.2 kernel. It’s ip rule, it’s four commands per uplink, and almost nobody has heard of it because ip route covers the single-uplink case that’s 95% of installations.

The setup

Two uplinks:

• eth0: 203.0.113.10/24, gateway 203.0.113.1
• eth1: 198.51.100.20/24, gateway 198.51.100.1

Name the tables so the rules are readable in six months:

echo "100 isp1" >> /etc/iproute2/rt_tables
echo "200 isp2" >> /etc/iproute2/rt_tables

Give each one a complete picture of its own uplink — the local subnet and a default route through its own gateway:

ip route add 203.0.113.0/24 dev eth0 src 203.0.113.10 table isp1
ip route add default via 203.0.113.1 dev eth0 table isp1

ip route add 198.51.100.0/24 dev eth1 src 198.51.100.20 table isp2
ip route add default via 198.51.100.1 dev eth1 table isp2

Then the rules — the part that does the actual work:

ip rule add from 203.0.113.10  table isp1 priority 100
ip rule add from 198.51.100.20 table isp2 priority 200

“A packet whose source address is 203.0.113.10 gets routed using table isp1.” Which means a reply from that address goes out eth0 with the right gateway, and a reply from the other address goes out eth1. Symmetric routing, restored.

The main table still handles everything else — locally-originated traffic with no explicit source, which follows whatever default you set there.

Reading what’s actually happening

ip rule show
0:	from all lookup local
100:	from 203.0.113.10 lookup isp1
200:	from 198.51.100.20 lookup isp2
32766:	from all lookup main
32767:	from all lookup default

Rules are evaluated in priority order, lowest first. Your rules at 100 and 200 come before main at 32766. Anything not matching a source rule falls through to main exactly as before.

The command that ends every argument about this:

ip route get 8.8.8.8 from 198.51.100.20
8.8.8.8 from 198.51.100.20 via 198.51.100.1 dev eth1 table isp2

ip route get runs the real lookup — rules, tables, everything — and prints which table won and which interface it chose. If it disagrees with what you expected, your rule is wrong, and you found out in a second instead of via a packet capture.

The two things that break it

rp_filter. Reverse path filtering makes the kernel drop an incoming packet if the route back to its source address doesn’t go out the interface it arrived on. In strict mode (1) that’s exactly backwards for a multi-homed box and it will silently eat your traffic:

sysctl -w net.ipv4.conf.all.rp_filter=2
sysctl -w net.ipv4.conf.eth0.rp_filter=2
sysctl -w net.ipv4.conf.eth1.rp_filter=2

Mode 2 is loose: the source must be reachable via some interface, not specifically the one it arrived on. Correct for asymmetric setups, and still keeps the basic anti-spoofing property. Note the all and per-interface values are combined with a max, so setting only one of them does nothing.

Persistence. Every command above evaporates on reboot. Where they go depends on your distro’s network layer — NetworkManager dispatcher scripts, systemd-networkd [RoutingPolicyRule] sections, /etc/network/interfaces post-up lines, or a netplan routing-policy block. Pick the one your box actually uses, and then reboot the box before you go home. A policy routing setup that only exists in the running kernel is a scheduled outage with the date left blank.

The other uses

Source-based rules are the classic case, but the rule selector takes more than from:

ip rule add fwmark 0x1 table isp2 priority 300     # mark in iptables/nft, route here
ip rule add to 10.50.0.0/16 table vpn priority 50  # this destination goes over the tunnel
ip rule add iif wg0 table vpn priority 60          # anything arriving on the VPN

The fwmark one is the useful general tool: mark packets in the firewall using any criteria the firewall can express — port, user, cgroup, connection state — and then route on the mark. That’s how you get “traffic from this container goes out the VPN and everything else doesn’t” without touching the container.

Which is a much better answer than the thing people usually build instead, which is a second box.