Sent: Friday, 29 June 2007 8:55 p.m.
After attending PACNOG3, I had a moment of clarity and managed to fix my multihoming problem.
I thought you guys might be interested in the solution even though it's not something I would usually recommend.
My main problem was that I only had a T1 connection to my secondary upstream provider. However, even with as-path prepends, I was getting more inbound traffic than the T1 could provide and ran into congestion problems. This secondary link is purely used as a backup since it is much smaller than our primary path.
The fix was to advertise two /21 prefixes out of my /20 on my primary provider. This would force BGP path selection to only use the primary provider since it had the longest prefix match.
For redundancy, I still advertise my full /20 to both upstreams.
Now, this is actually bad CIDR practice, but I would deem it a necessary evil in my case.
I guess a disproportionate number of your users are viewing CNN :-), but one option for you is to look at netflow stats on your outbound routers and see where bulk of your traffic is heading to.
In the very brief look at had at the configs, I didn't see any way that you are consciously selecting your outbound path with local-preference or other outbound traffic engineering mechanisms.
Maybe a show int on the (secondary provider) interface in question will tell you if the traffic is inbound our outbound..