10 Feb 10 netfilter/iptables split access with multiple ISPs
Quite a while back, I posted article http://www.adamsinfo.com/extending-tc-and-iproute2-linux-routing-split-access-multiple-uplinks-multiple-isps-iptables-masquerading/
The article focuses on using the standard iproute2 tool to allow the box to attempt to balance traffic over multiple uplinks with multiple default routes. While relatively easy to set up, it has a few problems:
- Routes are cached, meaning that once the balancer has decided on a route to a certain IP for the first time, it will continue to use this route for a while.
- There is no real control over which packets end up over which route, other than some basic metrics such as source IP and destination IP.
- Certain long established TCP connections such as MSN or IRC die after the route cache expires and the packets begin being routed over the other connection. Logically, there should be a fix for this or theres a bug in my script, either way I gave up digging after a while, and just forced connections to given IPs over the same route each time.
I’ve recently decided to give this a go in netfilter purely. My environment is a router with a number of LAN devices, with eth0 being the LAN interface (192.168.1.0/24), while eth1 and eth2 are separate ISP links with public IPs.
Firstly my iproute2 script:
IF2=eth2
IP1=94.195.XXX.XXX #IP on conn1
IP2=94.194.XXX.XXX #IP on conn2
P1=94.195.XXX.XXX #Gateway on conn1
P2=94.194.XXX.XXX #Gateway on conn2
P1_NET=94.195.XXX.XXX #Network address conn1
P2_NET=94.194.XXX.XXX #Network address conn2
ip route add $P1_NET dev $IF1 src $IP1 table 1
ip route add default via $P1 table 1
ip route add $P2_NET dev $IF2 src $IP2 table 2
ip route add default via $P2 table 2
ip route add $P1_NET dev $IF1 src $IP1
ip route add $P2_NET dev $IF2 src $IP2
ip rule add from $IP1 table 1
ip rule add from $IP2 table 2
ip rule add fwmark 1 table 1
ip rule add fwmark 2 table 2
What we’ve done at this point is create two tables, and set up the routers for each table, then specify that any packets marked as ‘1′ will leave via table 1 (i.e. routed out over eth1), and any packets marked ‘2′ will leave via table 2 (i.e. routed out over eth2). We will then mark the packets with iptables.
Now, I’ve only done basic INPUT/FORWARD/OUTPUT chain firewalling with iptables so I’d very much appreciate any feedback on how this can be improved or fixed, specifically in regards to security. Here’s the IP tables script:
IPTABLES=iptables
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 3600 > /proc/sys/net/ipv4/route/secret_interval
$IPTABLES -F
$IPTABLES -X
$IPTABLES -t nat -F
$IPTABLES -t nat -X
$IPTABLES -t filter -F
$IPTABLES -t filter -X
$IPTABLES -t mangle -F
$IPTABLES -t mangle -X
${IPTABLES} -t mangle -N M1
${IPTABLES} -t mangle -A M1 -j MARK –set-mark 1
${IPTABLES} -t mangle -A M1 -j CONNMARK –save-mark
${IPTABLES} -t mangle -N M2
${IPTABLES} -t mangle -A M2 -j MARK –set-mark 2
${IPTABLES} -t mangle -A M2 -j CONNMARK –save-mark
${IPTABLES} -t nat -N OUT_ETH1
${IPTABLES} -t nat -A OUT_ETH1 -j SNAT –to-source 94.195.XXX.XXX
${IPTABLES} -t nat -N OUT_ETH2
${IPTABLES} -t nat -A OUT_ETH2 -j SNAT –to-source 94.194.XXX.XXX
${IPTABLES} -t nat -A POSTROUTING -o eth1 -j OUT_ETH1
${IPTABLES} -t nat -A POSTROUTING -o eth2 -j OUT_ETH2
${IPTABLES} -t nat -N IN_ETH1
${IPTABLES} -t nat -A IN_ETH1 -j MARK –set-mark 1
${IPTABLES} -t nat -A IN_ETH1 -j CONNMARK –save-mark
${IPTABLES} -t nat -N IN_ETH2
${IPTABLES} -t nat -A IN_ETH2 -j MARK –set-mark 2
${IPTABLES} -t nat -A IN_ETH2 -j CONNMARK –save-mark
${IPTABLES} -t nat -A PREROUTING -i eth1 -j IN_ETH1
${IPTABLES} -t nat -A PREROUTING -i eth2 -j IN_ETH2
#balance TCP
${IPTABLES} -t mangle -A PREROUTING -i eth0 -p tcp -m tcp -m state –state NEW -m statistic –mode nth –every 2 –packet 0 -j M1
${IPTABLES} -t mangle -A PREROUTING -i eth0 -p tcp -m tcp -m state –state NEW -m statistic –mode nth –every 2 –packet 1 -j M2
${IPTABLES} -t mangle -A PREROUTING -i eth0 -p tcp -m tcp -m state –state ESTABLISHED,RELATED -j CONNMARK –restore-mark
#balance ICMP
${IPTABLES} -t mangle -A PREROUTING -i eth0 -p icmp -m icmp -m state –state NEW -m statistic –mode nth –every 2 –packet 0 -j M1
${IPTABLES} -t mangle -A PREROUTING -i eth0 -p icmp -m icmp -m state –state NEW -m statistic –mode nth –every 2 –packet 1 -j M2
#balance UDP
${IPTABLES} -t mangle -A PREROUTING -i eth0 -p udp -m udp -m statistic –mode nth –every 2 –packet 0 -j M1
${IPTABLES} -t mangle -A PREROUTING -i eth0 -p udp -m udp -m statistic –mode nth –every 2 –packet 1 -j M2
#slingbox
${IPTABLES} -A PREROUTING -t nat -i eth1 -p tcp –dport 5001 -j DNAT –to 192.168.1.254:5001
${IPTABLES} -A PREROUTING -t nat -i eth2 -p tcp –dport 5001 -j DNAT –to 192.168.1.254:5001
ip route flush cache
exit
This script should do the following:
- We use SNAT to route traffic out of eth1 and eth2
- We mark any packets coming in to eth1 or eth2 with their respective marks #is this necessary?
- We set a counter total of ‘2′ and then count every packet. If the counter is at 0, packets are sent to our custom table M1 and if the counter is at 1, the packets are sent out to our custom table M2
- If the packet is a NEW TCP packet, then we mark the outgoing packet as in ‘3′ above, then any RELATED packets to the ESTABLISHED connection as automatically assigned the same mark. This has to happen, and will mean that one connection to a given host is only actually established over one external interface at a time.
- UDP and ICMP packets are just marked regardless. #is this correct?
- I then forward port 5001 to an internal LAN device ’slingbox’. I allow this over both connections so I can connect in via the other if one is down.
This basic logic for deciding which route to send a packet over works well so far with traffic balanced well. We can look to change this now to to match and decide based on packet contents, load on either line and a whole host of other statistics..
At this point, the box itself has no default gateway and will therefore not have any outbound internet access (although inbound will work through the routes we set up with iproute2). Is there a way to configure this with iptables in the OUTPUT or PREROUTING chain? I couldn’t get it working as with no default route set, I’m not sure that you can? Instead, I use the iproute2 balancing method:
This is not ideal but as it’s the router itself and I don’t really care about it’s internet access. The LAN is still routed using our iptables method above.
Some people may prefer outsourcing their server hosting but these relatively simple IP table scripts helped me sort things out in the office myself! Hopefully you’ll get some use from this.
Improvements, fixes and comments very much welcome.
Tags: balancer, connmark, conntrack, dnat, icmp, iproute2, iptables, load balancing, mark, netfilter, snat, split access, tcp, udp
Reader's Comments
Leave a Comment
You must be logged in to post a comment.
|
I'm Adam Palmer, and I'm a Freelance Embedded Linux/Embedded Hardware enthusiast and Security Consultant. I run 'APNIC Solutions Ltd' and spend most of my time managing server clusters, and doing some PHP/MySQL development, whilst what little spare time I have is dedicated to playing with gadgets. I'm always happy to consult on or manage any Linux related, Web Application or Hosting project. Please get in touch with me! |
The kernel connection tracking module is capable of tracking UDP streams. You need to use the state NEW configuration for the UDP traffic like the TCP traffic or your UDP will end up all over the place.
You are splitting both incoming (to IN_ETHx) and outgoing packets (to M1/M2) – which is not necessary. Follow http://dev.medozas.de/NF-Cookbook.txt item 5 for a unified solution that also does not need the weighted default gateway.