10 Feb 10 netfilter/iptables split access with multiple ISPs

Quite a while back, I posted article http://www.adamsinfo.com/extending-tc-and-iproute2-linux-routing-split-access-multiple-uplinks-multiple-isps-iptables-masquerading/

The article focuses on using the standard iproute2 tool to allow the box to attempt to balance traffic over multiple uplinks with multiple default routes. While relatively easy to set up, it has a few problems:

1. Routes are cached, meaning that once the balancer has decided on a route to a certain IP for the first time, it will continue to use this route for a while.
2. There is no real control over which packets end up over which route, other than some basic metrics such as source IP and destination IP.
3. Certain long established TCP connections such as MSN or IRC die after the route cache expires and the packets begin being routed over the other connection. Logically, there should be a fix for this or theres a bug in my script, either way I gave up digging after a while, and just forced connections to given IPs over the same route each time.

I’ve recently decided to give this a go in netfilter purely. My environment is a router with a number of LAN devices, with eth0 being the LAN interface (192.168.1.0/24), while eth1 and eth2 are separate ISP links with public IPs.
(more…)

Tags: balancer, connmark, conntrack, dnat, icmp, iproute2, iptables, load balancing, mark, netfilter, snat, split access, tcp, udp

16 Sep 09 Multithreaded Tunnel Proxy and OpenVPN experiment

Further to the Multithreaded TCP Tunnel Proxy that I wrote a while ago, I’ve picked up a low end UK VPS and installed OpenVPN on it, as well as my local machine. I set up the iproute2 split access load balancer and established the OpenVPN connection.

Now, each of the two DSL lines is established at 17mbit giving me a theoretical maximum of 2.125MB/sec. In actual fact to kernel.org I can get a steady 1.7-1.8MB/sec which is more than enough. From my 100mbit UK VPS, I can get 8-9MB/sec from kernel.org without issue. Establishing OpenVPN over a single connection and then pulling a file from kernel.org leaves me with only 1.3MB/sec which I’m not best pleased about. Pulling the file through a proxy running on the UK VPS downloads at 1.6MB/sec minimum, so it isn’t my new route that’s causing the slow down, it’s OpenVPN.  Either way, I didn’t bother testing for any improvement with pptpd because I need OpenVPN’s single TCP connection anyway for this experiment to work.

The positive outcome of the story, is that with iproute2 load balancing set up, and OpenVPN established through the multithreaded TCP proxy over both connections, and using -t4. My single 1.3MB/sec became 2.2MB/sec which is IMHO an incredibly successful outcome.

A problem to note, is that on more than one occasion, netstat/lsof showed 3 TCP connections established over one DSL, and 1 over the other DSL. I just restarted my tcp tunnel a few times until I had them equally balanced. If this was a big enough problem -t6, -t8 or -t10 might have showed interesting results, but the more threads the more delay and potential issue with misordered packets. -t4 with iptables forcing the TCP connections equally over the DSLs might also be worth investigating. Nevertheless, as the experiment goes, a pleasing outcome!

Tags: dsl, iproute2, load balancer, multithreaded tcp tunnel, openvpn, proxy, split access, tcp, tcp tunnel, vps

28 Jul 09 Multithreaded TCP Proxy Tunnel

Sometime back I posted a guide in the HOWTO section on load balancing across multiple ISP connections. This is great in certain instances but certainly doesn’t replace line bonding.

With torrent traffic for example, it works great. The various peer connections should be established equally over the various ISPs and you really can utilize the bandwidth of multiple uplinks. A single connection however can only ever be established over one ISP, and this is the biggest limitation for the following reasons:

1) A single HTTP, POP3, SMTP, FTP, etc connection for example will only be established over one ISP whilst the other remains idle.
2) Incoming connections can only come in over one ISP depending on the IP accessed.

I’m sure there are further situations that might suffer limitations due to the setup.

The best way around this that i could think of was creating a multithreaded tcp proxy. The software consists of a proxy client and proxy server. I would run the proxy client on the local end, and the server on some remote machine. The client and the server will create a number of TCP connections between them which should hopefully be split across the ISPs equally, and reassembled by the proxy server at the other end.

The limitations of this though are that the entire session will be limited to the speed of the slowest link, and any of the links failing will cause the tunnel to become corrupted and fail.

C code and an update to come shortly.

Tags: isps, multiple uplinks, Multithreaded TCP Proxy Tunnel, split access

14 Sep 08 Extending tc and iproute2. Linux routing split access multiple uplinks multiple isps iptables masquerading

It’s been a while since I last posted here, so I thought I’d post a followup article to http://www.adamsinfo.com/bandwith-limiting-with-linux-tc-and-iproute2/ which focused on bandwidth limiting in a datacenter environment using tc and iproute2.

I’ve taken the same script but tweaked IPs and bandwith values into my office. Previously I was on a 24mbit down 2.5mbit up DSL connection courtesy of www.bethere.co.uk The office is only about 800m from the closest exchange which is quite nice – I generally find I get 18+mbit down and 1.5+mbit up. Not only great bandwidth, but latency is also very small and responsiveness is great, especially as a regular [constant] SSH use. Recently, despite having no business justification whatsoever, I ordered the same again for the same office. This one clocks in at about 19mbit up and 1.7mbit down – even better! Some ISPs support line bonding – I dont believe that many in the UK do, and seeing as at the time of writing, bethere were the only ISP to support anywhere close to 24mbit, I wasn’t going to try and find another.
(more…)

Tags: Extending tc and iproute2, faster torrent, iproute2, iptables, Linux routing, masquerading, multiple isps, multiple uplinks, Routerboard load balancing, split access, tc, torrent

26 May 08 Split access SSH problems

So far everything works flawlessly, all applications, protocols and service with the split access setup, but I have a strange bug with SSH, and it only affects SSH clients built on openssl, such as dropbear and openssh. After authentication, it just hangs, and running in verbose mode shows that it seems to fail upon attempting to resize the tcp window. Only fails occasionally, when it receives no acknowledgement from the server.

After much digging, I only had minimal success getting to the bottom of the issue, but the following hack fixed the issue:
(more…)

Tags: cached routes, dropbear, iproute2, iptables, openssh, openssl, routes, split access, ssh, tcp window

26 May 08 Custom linux router, routing, split access and more..

So, now I have got this router set up and working, it was time to customize it somewhat.

In terms of networking, we have 3 10/100 ethernet ports and one wireless adapter, and this is set up as follows:

Internet comes in to eth1 and eth2, both of which have public IPs. eth0 which is another ethernet interface is bridged with ath0 which is my wireless interface (in master mode, with WPA-PSK/TKIP with hostap running) to form br-lan, whos IP is 192.168.100.1and is connected via a switch to feed the LAN PCs

I’m using a pretty simple netfilter masquerade script in use with this for NAT.
(more…)

Tags: bridge, ethernet, hostap, IP, load balance, multiple providers, router, split access, upnp, wireless, wpa