Networking 101: Understanding Internet Routing and Peering
Best of ENP: Internet routing involves a lot of players and a lot of pieces. Before we delve into Internet routing protocols, here are the broad strokes.
What exactly is the Internet? This article will explain the concepts required to understand BGP, our next Networking 101 topic. Shipping packets around the Internet requires the cooperation of separate organizations, so it isn't as straightforward as learning about an IGP routing mechanism.
Starting at the service provider level, let's use an example of a local ISP that isn't very big.
JoeBob ISP is going to need an ISP itself before it can ship a customer's traffic to the Internet. It should start by forming distinct relationships with two different tier 1 ISPs. These are the big players: Level 3, AT&T, Cogent, XO, etc. JoeBob ISP will "peer" with these ISPs, in a BGP sense. This means that they will form a BGP session, and exchange routes. The big ISP will send all of the Internet's routes to JoeBob, which amounts to just fewer than 200,000 routes at this time. We'll get into the details of how that works in the BGP article.
JoeBob ISP will pay a certain dollar amount per MB of traffic sent. It's connected to two ISPs, so it will likely have two different prices. BGP magic can be done to favor the cheaper link, in that case. The crux of the matter is that JoeBob ISP is getting routes from another ISP, and it can do what it wishes with them. Some traffic can be sent to either tier 1 ISPs, and some can be sent elsewhere.
Peering agreements aren't just for the ISP-to-ISP BGP sessions, they are also used to negotiate company to company traffic flows. Even the small ISP can sometimes hop onto a local exchange, and peer with other companies, schools or ISPs in the area. An exchange is a network connection point, and every city has at least one. All the major players in an area will connect to the exchange, and maintain their own routers within. If they decide to peer with a friend, they can simply have the exchange operators patch some fiber through to their friend's rack, and viola, free packets. There's normally a port charge to connect to exchanges, and I didn't mention anything about how you'd get traffic from your site to the exchange. That takes fiber (money) too.
Peering with others isn't a very common practice for small local ISPs, but this is exactly how larger ISPs (and even schools) operate. Sometimes an ISP will even provide its peers' routes to customers, and allow that traffic for free. The capability of the ISP to do this depends on the specifics of the peering agreements.
On the topic of peering, there are two types of exchanges: public and private.
A public exchange will allow most anyone to become a member. Public peering doesn't imply that you all of a sudden get free routes to every ISP that's on the exchange; rather it simply means that you can connect to the exchange, sometimes for free. One very popular exchange is the SIX, or Seattle Internet Exchange. If you are already a tenant of the Westin building in Seattle, you can pay them to patch you into the SIX closet. Once you get an IP from the SIX operator, you're on. After that, however, you must form peering arrangements with the other BGP speakers in the exchange.
The SIX and the PAIX are the most popular west-coast exchanges. A few people operate the SIX with a few Cisco switches in a rack. Essentially every major carrier and service provider now connects to the SIX, including Google, XO and AT&T broadband. The PAIX started life in Palo Alto, and is operated by Switch and Data now. It too is a public exchange, and it also runs private peering alongside its public offerings.
Then there's private peering.
Most public exchanges have 100Mb or 1Gb connections participants can connect to. If you're a very large ISP, you need (many) 10Gb links to handle all your traffic. The problem is that public exchanges don't normally have the funds to provide this type of service reliably. Private peering provides an SLA (service level agreement), and most of the time that comes with better hardware to handle the traffic. There's always ongoing public vs. private peering debates that are quite entertaining, but never really lead anywhere. If you're going to need custom solutions, like trunking two links together, then private peering is where you'll likely end up.
Peering is the opposite of transit. Transit providers, like tier 1 ISPs, give you all the Internet routes you pay for (normally all of them), and take care of everything else. A company's ISP is a transit provider, and so is the ISP that services a smaller ISP. ISPs that have peering along with their transit will generally send everything they can through the peer, and everything else onto the transit provider.
The concept of a default-free router means that you have a router that gets a full Internet routing table. If your transit provider doesn't send you a route for some network, and you don't have any other knowledge of it through your other peers, then it's simply unreachable. There is no "default router" that you simply send everything to in hopes that it will do the right thing.
When an ISP becomes large enough that tier 1 ISPs are sending a big portion of their traffic to said ISP, then peering agreements will normally happen. Peering is really motivated by two factors: money, and traffic efficiency. Often times two ISPs will notice that they talk between each other quite often, but their traffic has to travel half way around the world before it reaches the other. If the growing ISP is geographically disperse enough, they will be able to peer in many different exchanges, and pretty soon most of their traffic will no longer have to use a transit provider anymore. Anyone with enough muscle to get peering arranged with the big players, and their own infrastructure in all of these places, is probably a major service provider already.
That's a very quick rundown of how the Internet works. There are a lot more layers involved than we've implied, but this is the gist of how the Internet works. Come back next week to discover how the routing works between all of these separate entities. It's quite fascinating.