One FIB, Three Homes: CEF on IOS XE Routers, Catalyst Switches and IOS XR
The Same Idea in Different Places
Cisco Express Forwarding is one design, but it is implemented in very different places depending on the box. On every platform the logic is the same: the routing protocols feed the RIB, the RIB feeds the Forwarding Information Base, and each FIB entry points to an adjacency holding the Layer 2 rewrite. What changes is where the FIB lives, who programs it and what happens when it is full. For a BGP engineer those differences are not trivia. They decide how many prefixes a device can carry, how a failure shows itself and which commands prove that forwarding matches routing.
Start with the platform most engineers learn first, a software-based IOS XE router such as a lab CSR or Catalyst 8000V. Here the FIB is a data structure in memory, built and used by the forwarding code. There is no special hardware to run out of, so the limit is memory. The full chain from BGP to the wire is visible with commands you already know.
R1#show ip route summary
IP routing table name is default (0x0)
IP routing table maximum-paths is 32
Route Source Networks Subnets Replicates Overhead Memory (bytes)
application 0 0 0 0 0
connected 0 4 0 384 1216
static 1 0 0 96 304
ospf 1 0 3 0 288 912
Intra-area: 3 Inter-area: 0 External-1: 0 External-2: 0
NSSA External-1: 0 NSSA External-2: 0
bgp 65001 12840 2311 0 1454496 4604160
External: 15151 Internal: 0 Local: 0
internal 231 2741624
Total 13072 2318 0 1455264 7347216
R1#show ip cef 203.0.113.0/24 detail
203.0.113.0/24, epoch 2
recursive via 10.1.12.2
attached to GigabitEthernet1
The routing table summary shows how many BGP routes the RIB holds and the memory they use. The CEF entry shows one of them programmed for forwarding. On a software router, if the RIB has the route and CEF shows the entry, forwarding works.
Hardware Tables on Catalyst 9000 Switches
That simple picture changes as soon as forwarding moves into hardware. A Catalyst 9000 switch forwards packets in an ASIC, not in software. The FIB that "#show ip cef" prints is still built in software, but each entry must then be programmed into the ASIC's own forwarding resources, which are fixed in size. Cisco describes the switch's IP route tables in exact match memory and TCAM as holding data that comes from the RIB and FIB. The software FIB and the hardware tables are two copies, and the second one can fill up.
This is where BGP becomes dangerous on a switch. Suppose a Catalyst 9300 acting as a small edge device is asked to accept a few hundred thousand prefixes from an upstream provider, because someone removed an inbound filter. BGP accepts them, the RIB installs them and CEF builds them. Then the hardware runs out of space.
SW1#show ip bgp summary
BGP router identifier 10.0.0.1, local AS number 65001
BGP table version is 412977, main routing table version 412977
412933 network entries using 102807316 bytes of memory
412933 path entries using 56158888 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
203.0.113.1 4 64500 418221 89 412977 0 0 00:41:12 412933
SW1#
%FED_L3_ERRMSG-3-RSRC_ERR: Switch 1 R0/0: fed: Failed to allocate hardware resource for fib entry due to hardware resource exhaustion
The BGP session is up and every prefix is accepted, so the control plane looks healthy. The log message tells the real story. Once the hardware is exhausted, routes that were already programmed keep forwarding, but new or changed prefixes cannot be written into the hardware and become unreachable. The failure is partial and depends on the order in which routes arrived, which makes it one of the hardest outages to diagnose if you do not know to look for this message.
The defence is design, not tuning. A switch should never be given more routes than its hardware supports. Filter inbound BGP to the prefixes you need, or accept a default route plus a small set of specifics. Use "#neighbor 203.0.113.1 maximum-prefix" so that an unexpected flood tears the session down instead of quietly exhausting the hardware. Check the platform's documented route scale and template settings before you put it at the edge. A router can usually absorb a full table in memory. A switch built for access or distribution often cannot.
SW1(config)#router bgp 65001
SW1(config-router)#address-family ipv4 unicast
SW1(config-router-af)#neighbor 203.0.113.1 maximum-prefix 20000 90
With this limit, the router logs a warning at 90 percent of 20,000 prefixes and closes the session if the limit is exceeded. A closed session is a visible, fast failure. Exhausted hardware is an invisible, slow one.
Distributed Forwarding on IOS XR
IOS XR takes the separation one step further, and it is the platform where the RIB and the FIB are most clearly different things. On XR, the RIB runs as its own process, separate from the routing protocols. The best routes are downloaded from the RIB to each line card's FIB by a distribution service called BCDL. Every line card holds its own FIB, so "#show cef" can give different answers depending on which location you ask about.
RP/0/RP0/CPU0:XR1#show cef 192.0.2.0/24
Wed Nov 11 19:05:33.802 UTC
192.0.2.0/24, version 58, source-destination sharing
Prefix Len 24
via 2.2.2.2/32, 0 dependencies, recursive
next hop 10.2.12.2/32 Gi0/0/0/0 via 10.2.12.0/24
valid remote adjacency
The structure is familiar: a BGP prefix recursing through a next hop, resolved down to an interface and an adjacency. "source-destination sharing" names the load sharing method, and "valid remote adjacency" says the adjacency belongs to an interface handled elsewhere in the system.
Because the route travels from the protocols to the RIB to every line card, XR also provides tools to prove the copies agree. Route Consistency Checking compares the RIB with the FIB, and Label Consistency Checking compares the label switching database with the FIB. They run on demand with "#show rcc ipv4 unicast", and the background scan is enabled with "#rcc ipv4 unicast enable". On a large XR router carrying full tables, these checks are the formal answer to the question every operator eventually asks: is the data plane really forwarding what the control plane decided?
XR also protects its RIB from a specific kind of instability. When two routes resolve through each other, a mutual recursion, the RIB quarantines the route that completes the loop, re-evaluates it periodically, and does not let BGP advertise a quarantined route. The point is to stop a recursion loop from reaching the FIB, where it would turn into a forwarding loop.
The three platforms lead to one consistent working method. On every Cisco device, BGP is only the first step. Read the RIB to see what was installed, read the FIB with "#show ip cef" or "#show cef" to see what is programmed, and on hardware or distributed platforms remember that a third copy exists below that, in the ASIC or on the line card, with its own limits. When a BGP design scales up, ask early which of those three copies will run out first. On a router it is usually memory, on a switch it is the hardware tables, and on a large XR system it is a question of keeping every line card in step. Plan for the smallest one, and BGP will never outgrow the box underneath it.