On Your Network: What the Heck is a TCAM?
Ever wondered why routers need reboots, or why your admins are shaking their heads when you ask for two sets of features they can't provide at the same time? Read on.
Let's talk about TCAM hardware, Cisco SDM, and try to answer that elusive question: "why do I have to reboot my router to enable certain features, which in turn disables others?"
First, CAM stands for Content Addressable Memory. A CAM is a special type of memory; some would say the opposite of RAM. With normal computer memory (RAM) the operating system provides an address, and receives the data stored at the supplied address. With a CAM, the operating system supplies the data, and the CAM returns a list of addresses where the data is stored, if it finds any. Furthermore, a CAM searches the entire memory in one operation, so it is considerably faster than RAM.
CAMs are very expensive, so they aren't normally found in PCs. Even router vendors will sometimes skimp, opting to instead implement advanced software-based searching algorithms. Most commonly CAMs and TCAMs are found in network processing devices, including Intel IXP cards and various routers or switches.
The most commonly implemented CAMs are called binary CAMs. They search only for ones and zeros; a simple operation. MAC address tables in switches commonly get stored inside binary CAMs. You can bet that any switch capable of forwarding Ethernet frames at line-speed gigabit is using CAMs for lookups. If they were using RAM, the operating system would have to remember the address where everything is stored. With CAMs, the operating system can find what it needs in a single operation. In this case it's the switchport that data should be sent out, based on the given MAC address, i.e. the essence of a MAC table. Some older Cisco switches running CatOS even opted to call this table the CAM table, thereby causing great confusion across the land.
Finally, a TCAM is a Ternary CAM. This allows the operating system to match a third state, "X." The X state is a "mask," meaning its value can be anything. This lends itself well to networking, since netmasks (define) operate this way. To calculate a subnet address we mask the bits we don't care about, and then apply the logical AND operation to the rest. Routers can store their entire routing table in these TCAMs, allowing for very quick lookups.
Hardware can sometimes seem to work like magic, but it isn't always transparent. When configuring routers most people will run into a situation where enabling a new feature will require that the Cisco SDM (Switching Database Manager) template be changed. This template is actually a method Cisco uses to assign specific application to specific TCAM resources.
Some routers will allow you to manually specify how much TCAM space you want to allocate to a specific feature. Others aren't so nice. They make you choose from a few restrictive templates, which allocate the resources automatically based on a few predetermined settings. For example, on the Cisco 3750, we recently wanted to enable policy-based routing (PBR) to implement a layer 3 jail. The basic idea with template-only routers is that you have to choose where you want most of the optimizations, and compromise on the rest.
For this platform, there are four templates: default, routing, PBR, and VLAN. Each of these tries to allow for a bit more resources allocated to the specified task. For policy routing, we'd have to choose "routing" or "PBR," which in turn limits the amount of unicast MAC addresses (define) that can be held in TCAMs. Likewise, selecting a VLAN (define) template will make PBR impossible, but allow for more VLAN database information to be held in TCAMs. There are always compromises when we need to use more advanced features. Keeping true to the spirit of router operating systems, there are also some mysterious side effects when a new template is chosen. On our specific router, if the PBR template is chosen, the router will become unable to support VPN routing/forwarding tables (VRF). The next unfortunate gotcha is that with the IOS version that supports IPv6, you cannot even enable PBR. There is no template to allow both policy routing and IPv6.
Here are some more TCAM allocation examples. Just because, for instance, 8K is allocated to routing tables, this doesn't mean that you can only have a routing table of that size. There's always the fallback of process switching. Process switching means that everything will be done by the processor instead of in hardware (TCAMs). Processor intervention is not desirable, mostly because it is much slower than hardware lookups. Also, the processor is supposed to be used for things like sending logs to a syslog server and controlling SSH sessions. If a router doing process switching gets really busy, it may be unable to service your console access attempts.
Hardware is finite, and we always need more. More expensive routers don't always suffer from the constant struggle for TCAMs because they have enough to support most features that currently exist. Unfortunately, most companies won't want to purchase the latest and greatest router with seemingly endless TCAMs unless they can justify the added cost by showing a need for them right now. So we're stuck having to adjust TCAM allocations.