Bandwidth Limiting HOWTO with linux, tc and iproute2

I’ve recently optimized the scripts used for bandwidth management in one of our UK facilities and I thought I’d post a quick howto on it.

Discuss this page here

My setup here is a live feed entering eth0 on this linux router and leaving eth1 into a switch connected to a collection of other servers. This is set up as an unrestricted public router, routing between a /30 on eth0 and a /24 on eth1.
Note: We can’t in any way restrict the amount of traffic that eth0 receives from the outside, so instead we restrict how fast eth0 sends data out, the same applies the other way round. So, if we want to limit the amount of data that the local servers can send, we shape the router’s external interface (eth0). If we want to limit the amount of data that the local servers can receive, we shape the router’s internal interface (eth1)

With Debian Etch on 2.6.x, run:
apt-get install tc iproute2 bc

Then script as follows:

# Set some variables
#!/bin/bash
EXT_IFACE=”eth0″
INT_IFACE=”eth1″
TC=”tc”
UNITS=”kbit”
LINE=”10000″ #maximum ext link speed
LIMIT=”5000″ #maximum that we’ll allow
# Set some variables for individual “classes” that we’ll use to shape internal upload speed, i.e. shaping eth0
CLS1_RATE=”200″ # High Priority traffic class has 200kbit
CLS2_RATE=”300″ # Medium Priority class has 300kbit
CLS3_RATE=”4500″ # Bulk class has 4500kbit
# (We’ll set which ones can borrow from which later)
# Set some variables for individual “classes” that we’ll use to shape internal download speed, i.e. shaping eth1
INT_CLS1_RATE=”1000″ #Priority
INT_CLS2_RATE=”4000″ #Bulk
# Delete current qdiscs. i.e. clean up
${TC} qdisc del dev ${INT_IFACE} root
${TC} qdisc del dev ${EXT_IFACE} root
# Attach root qdiscs. We are using HTB here, and attaching this qdisc to both interfaces. We’ll label it “1:0″
${TC} qdisc add dev ${INT_IFACE} root handle 1:0 htb
${TC} qdisc add dev ${EXT_IFACE} root handle 1:0 htb
# Create root classes, with the maximum limits defined
# One for eth1
${TC} class add dev ${INT_IFACE} parent 1:0 classid 1:1 htb rate ${LIMIT}${UNITS} ceil ${LIMIT}${UNITS}
# One for eth0
${TC} class add dev ${EXT_IFACE} parent 1:0 classid 1:1 htb rate ${LIMIT}${UNITS} ceil ${LIMIT}${UNITS}
# Create child classes
# These are for our internal interface eth1
# Create a class labelled “1:2″ and give it the limit defined above
${TC} class add dev ${INT_IFACE} parent 1:1 classid 1:2 htb rate ${INT_CLS1_RATE}${UNITS} ceil ${LIMIT}${UNITS}
# Create a class labelled “1:3″ and give it the limit defined above
${TC} class add dev ${INT_IFACE} parent 1:1 classid 1:3 htb rate ${INT_CLS2_RATE}${UNITS} ceil ${INT_CLS2_RATE}${UNITS}
# EXT_IF (upload) now. We also set which classes can borrow and lend.
# This class is guaranteed 200kbit and can burst up to 5000kbit if available
${TC} class add dev ${EXT_IFACE} parent 1:1 classid 1:2 htb rate ${CLS1_RATE}${UNITS} ceil ${LIMIT}${UNITS}
# This class is guaranteed 300kbit and can burst up to 5000kbit-200kbit=4800kbit if available
${TC} class add dev ${EXT_IFACE} parent 1:1 classid 1:3 htb rate ${CLS2_RATE}${UNITS} ceil `echo ${LIMIT}-${CLS1_RATE}|bc`${UNITS}
# This class can is guaranteed 4500kbit and can’t burst past it (5000kbit-200kbit-300kbit=4500kbit).
# I.e. even if our bulk traffic goes crazy, the two classes above are still guaranteed availability.
${TC} class add dev ${EXT_IFACE} parent 1:1 classid 1:4 htb rate ${CLS3_RATE}${UNITS} ceil `echo ${LIMIT}-${CLS1_RATE}-${CLS2_RATE}|bc`${UNITS}
# Add pfifo. Read more about pfifo elsewhere, it’s outside the scope of this howto.
${TC} qdisc add dev ${INT_IFACE} parent 1:2 handle 12: pfifo limit 10
${TC} qdisc add dev ${INT_IFACE} parent 1:3 handle 13: pfifo limit 10
${TC} qdisc add dev ${EXT_IFACE} parent 1:2 handle 12: pfifo limit 10
${TC} qdisc add dev ${EXT_IFACE} parent 1:3 handle 13: pfifo limit 10
${TC} qdisc add dev ${EXT_IFACE} parent 1:4 handle 14: pfifo limit 10
### Done adding all the classes, now set up some rules! ###
# INT_IFACE
# Note the ‘dst’ direction. Traffic that goes OUT of our internal interface and to our servers is out server’s download speed, so SOME_IMPORTANT_IP is allocated to 1:2 class for download.
${TC} filter add dev ${INT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip dst SOME_IMPORTANT_IP/32 flowid 1:2
${TC} filter add dev ${INT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip dst SOME_OTHER_IMPORTANT_IP/32 flowid 1:2
#All other servers download speed goes to 1:3 – not as important as the above two
${TC} filter add dev ${INT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip dst 0.0.0.0/0 flowid 1:3
# EXT_IFACE
# Prioritize DNS requests
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src IMPORTANT_IP/32 match ip sport 53 0xffff flowid 1:2
# SSH is important
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src IMPORTANT_IP/32 match ip sport 22 0xffff flowid 1:2
# Our exim SMTP server is important too
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src IMPORTANT_MAILSERVER/32 match ip sport 25 0xffff flowid 1:3
# The bulk
${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src 0.0.0.0/0 flowid 1:4

I have tried not to get bogged down with too many irrelevant details here and would be happy to answer any questions or take any corrections. It’s pretty simple and it works well. Install bmon and you can confirm this yourself. The purpose of this is that I can take a 10mbit connection and limit the traffic to 5mbit ensuring that I don’t break the 95th percentile that I want to maintain at the datacenter. I can increase and decrease this at any time as traffic requires or permits respectively.

Discuss this page here

Tagged with: , , , , , , , , , , , , , , , , , , , , , , , , , ,
Posted in Internetworking & Routing, Linux, Technology
9 Comments » for Bandwidth Limiting HOWTO with linux, tc and iproute2
  1. mario says:

    Hi Adam,

    I have a similar requirement at the datacentre. The only difference is that I need to limit only the eth0 interface outbound speed and throttle it to between 3-5kbps. The eth1 interface needs to stay at gig speed for back end communication.

    Do you happen to have a one nic throttling version of this script handy?

    thanks
    -mario

  2. Adam Palmer says:

    Mario,

    I haven’t tested this. Give this a go:

    # As a note, it’s pretty easy to break your networking by throttling traffic to a stage where it is so congested, you can not log in to clear entries. Either test this out on a development server first, or if that isn’t possible, before you run your script, set something like: “sleep 10; tc qdisc del dev eth0 root &” which will sleep for 10 seconds and then delete your qdiscs. That way if you lock yourself out, it should automatically clear after 10 seconds.

    # Set some variables
    #!/bin/bash
    EXT_IFACE=”eth0″
    TC=”tc”
    UNITS=”kbit”
    LINE=”10000″ #maximum ext link speed
    LIMIT=”5″ #maximum that we’ll allow

    # Set some variables for individual “classes” that we’ll use to shape internal upload speed, i.e. shaping eth0
    CLS1_RATE=”5″ # High Priority traffic class has 200kbit

    # Delete current qdiscs. i.e. clean up
    ${TC} qdisc del dev ${EXT_IFACE} root

    # Attach root qdiscs. We are using HTB here, and attaching this qdisc to both interfaces. We’ll label it “1:0″
    ${TC} qdisc add dev ${EXT_IFACE} root handle 1:0 htb

    # Create root classes, with the maximum limits defined
    # One for eth0
    ${TC} class add dev ${EXT_IFACE} parent 1:0 classid 1:1 htb rate ${LIMIT}${UNITS} ceil ${LIMIT}${UNITS}

    # Create child classes

    # EXT_IF (upload) now. We also set which classes can borrow and lend.
    # This class is guaranteed 200kbit and can burst up to 5000kbit if available
    ${TC} class add dev ${EXT_IFACE} parent 1:1 classid 1:2 htb rate ${CLS1_RATE}${UNITS} ceil ${LIMIT}${UNITS}

    # Add pfifo. Read more about pfifo elsewhere, it’s outside the scope of this howto.
    ${TC} qdisc add dev ${EXT_IFACE} parent 1:2 handle 12: pfifo limit 10

    ### Done adding all the classes, now set up some rules! ###
    # EXT_IFACE

    ${TC} filter add dev ${EXT_IFACE} parent 1:0 protocol ip prio 1 u32 match ip src 0.0.0.0/0 flowid 1:2

  3. mario says:

    Thanks Adam, I’ll give it a try today…

    We have the luxury of “pulling a cowboy move” because we can access the blades remotely if we hose our networking.

    I’ll let you know how it works under stress.

    cheers
    -mario

  4. Fred says:

    Thanks Adam for posting this, it’ll be very useful (if it works).

  5. Adam Palmer says:

    Anytime. Do let me know if that second script doesn’t work as I haven’t tested it yet. If not, I’ll set up a test locally here and get back with something!

  6. Jose Colon says:

    Thanks for this valuable info given how cryptic tc can be to Linux outsiders! One note: I don’t know if that’s the real IP to your exim server but if it is, maybe you forgot to replace it with something like IMPORTANT_MAIL_SERVER. Just a heads up!

  7. Adam Palmer says:

    Thanks very much for this Jose – I’ve blanked out the exim server now! This took me a bit of fiddling and time to put together because TC is indeed pretty cryptic. Glad you found it helpful.

  8. Dan says:

    Hi Adam,

    I hope you can help me.. I have created a PopTop VPN server on a version of Linux – Debian Lenny.

    I have many ppp connections at one time and I need to bandwidth shape them. The problem lies with to solutions… Either Bandwidth shape each ppp (so, ppp0 – 50kbps) and secure each ppp use to there own tunnel, (therefore – vpnUSER001 – ppp0, vpnUSER002 – ppp1 and so on) I don’t know whether this is possible tho. or create each bandwidth shape for the ip within each ppp for every one so then they can get any ppp tunnel and have there own bandwidth shape.

    If you could help me that would be great. If you have any other possible solutions could you please tell me. If you know a solution please give me some code to work off as I have not been long working in the world of ”tc” and ”iproute2”.

    Thank you for you time!

    Dan

  9. Wilson says:

    I installed the script on a debian router but, it doesn’t seem to work. Users can still download and upload to the max speed. I don’t know what’s wrong. I see some packets go through the class when I run tc -s -d class ls dev eth1 but, don’t see any traffic control.

    Don’t know what’s wrong.

1 Pings/Trackbacks for "Bandwidth Limiting HOWTO with linux, tc and iproute2"
  1. [...] Talk, Linux HOWTOs & Discussion on APNIC Box – Linux on a Mikrotik 532a, Part 1Adam Palmer on Bandwidth Limiting HOWTO with linux, tc and iproute2Jose Colon on Bandwidth Limiting HOWTO with linux, tc and iproute2viraz on NetCat tutorial for Linux [...]

Leave a Reply