One of the more annoying things about Apple’s wireless routers is that there’s no way to shape the bandwidth. With two of us in the house, commonly using the Internet at the same time, and a limited 43 KB/s bandwidth, we wind up stepping on each other’s use fairly often. One bandwidth limiting tool is the Unix command trickle which allows you to control bandwidth on individual, command line programs. Something like:
trickle -u 20 -d 20 wget http://bigfiles.com/bigfile.mp3would limit the file download to 20 KB/second, about half our our bandwidth. Many commands like wget and rsync have bandwidth limiting built in, making trickle unnecessary for those programs.
These techniques don’t work when the programs don’t include limiting internally, and when you can’t run them from the command line. The program I use to download music from eMusic (eMusicJ) is an example. With my downloads refreshing in a couple days, I wanted to find a way to get my downloads in, without ruining the network for the next day and a half.
Since OS X is built on BSD, it comes with a super-sophisticated firewall, ipfw, that has traffic shaping built in. So here’s how I was able to consume only half of our bandwidth downloading music:
Start the download and use netstat -an to find the IP address of the download site (or do netstat -an before and after you’ve started the download to identify the new download IP Address):
$ netstat -an | less Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 0 10.0.1.198.54176 126.96.36.199.80 ESTABLISHED tcp4 0 0 10.0.1.198.54175 188.8.131.52.80 ESTABLISHED tcp4 0 0 10.0.1.198.54142 184.108.40.206.80 ESTABLISHED tcp4 0 0 10.0.1.198.54136 220.127.116.11.80 ESTABLISHED
Set up a pipe for data coming from that site:
$ sudo ipfw add pipe 10 ip from 18.104.22.168 to any 00100 pipe 10 ip from 22.214.171.124 to any
Configure the pipe to limit bandwidth:
$ sudo ipfw pipe 10 config bw 20KBytes/s queue 10KBytes
After you're done, delete the pipe:
$ sudo ipfw list 00100 pipe 10 ip from 126.96.36.199 to any 65535 allow ip from any to any $ sudo ipfw del 00100
Yesterday I found myself with 43 eMusic downloads available and my refresh date approaching quickly. I’ve got quite a few records in my queue, and choosing among them to exactly consume my available downloads is difficult to do by hand. So I wrote a program to do it.
It’s a Python script, so it’ll run any any platform. Click this link to download it: choose_albums.py
To use it, you’ll need to create a separate file that contains a list of the albums you’re interested in and the number of tracks on each album. Here’s the file I was working with yesterday, called queue:
clientele 14 rosebuds 9 okkervil river 11 stravinsky 19 saint-saens violin 3 8 mapmaker 12 of montreal 5 long blondes 14 glass #4 7 widor #5&9 9 bonnie billy 13
Each line contains an album name, a space, and the lines end with the number of tracks on the album.
To run the program, call it and pass the name of your file and the number of downloads you’ve got left:
$ ./choose_albums.py queue 43 glass #4, saint-saens violin 3, stravinsky, widor #5&9: 43
This is one (of many) ways to use up my 43 downloads. The script chooses albums randomly, so if you want to see all the possibilities, you’ll need to run it a lot. I wrote a very simple shell script to do that:
#! /bin/sh tracks=$@ for i in `seq 1 100`; do ./choose_albums.py queue $tracks; done | sort | uniq
You can download it here: doit.sh
Depending on how large your queue is, you may need to increase the number of times it runs the script. Because it’s random and not deterministic, it can take a lot of runs to find all possible options (in fact, with 25 albums in the queue and 90 tracks available, there are more than 40,000 possible combinations, so this script is best at choosing from a small set of options, unless a random choice is what you're after). You’ll also need to change the name of your queue file if it’s not called queue.
Here’s what I did yesterday:
$ ./doit.sh 43 | grep blonde | grep glass bonnie billy, glass #4, long blondes, rosebuds: 43 bonnie billy, glass #4, long blondes, widor #5&9: 43 clientele, glass #4, long blondes, saint-saens violin 3: 43 glass #4, long blondes, of montreal, rosebuds, saint-saens violin 3: 43 glass #4, long blondes, of montreal, saint-saens violin 3, widor #5&9: 43
The two grep commands were included because I knew I wanted to include the new Long Blondes album and Philip Glass’s Fourth Symphony in my selections. I wound up going with the second choice, adding Bonnie “Prince” Billy’s The Letting Go and Widor’s Fifth and Ninth Symphonies.
One final note: if after 1,000 tries, the script doesn’t find a set of choices that uses up all your downloads, it’ll report the last set of albums it found and the number of tracks used up. Be sure that the final number reported matches the number you passed in or you won’t be using all your downloads for the month. The script isn’t smart enough to find the “best” solution in this situation, so if this happens, you’ll need to run it a bunch of times to maximize the number you’re downloading (or better, add more items to the queue file and run it again).