When the Routing Table Is Right but Traffic Is Wrong: Troubleshooting Forwarding With CEF
Why the Routing Table Is Not Enough
Every engineer eventually meets the ticket that makes no sense. The BGP route is there in "#show ip bgp". The routing table shows the right next hop in "#show ip route". The neighbour is up. And yet the customer's traffic either disappears or leaves through the wrong link. The reason is that both of those commands show the control plane, the router's intentions. Packets are switched by Cisco Express Forwarding, from the Forwarding Information Base (FIB) and the adjacency table, and that is where you have to look when intentions and reality disagree.
A short recap makes the rest of the lab easy to follow. CEF builds the FIB from the RIB, one entry for every route, with recursive next hops already resolved. Each FIB entry points to an adjacency, which holds the Layer 2 rewrite for a directly connected next hop. A packet costs one FIB lookup, one adjacency, one rewrite. Anything that breaks one of those three steps breaks forwarding, even when every control plane table looks fine.
The lab uses an edge router, R1, in AS 65001. It peers by eBGP with two upstream routers, R2 at 10.1.12.2 and R3 at 10.1.13.3, and by iBGP with R4, whose loopback is 4.4.4.4. OSPF carries the loopbacks. Four problems are built on this topology, and each one is found with the same small set of CEF commands.
The first command is the most useful one in this post. The command "#show ip cef exact-route" asks the router to run the full forwarding decision for one specific flow and tell you the outgoing interface and next hop it would use.
R1#show ip cef exact-route 10.10.1.50 198.51.100.20
10.10.1.50 -> 198.51.100.20 =>IP adj out of GigabitEthernet0/0, addr 10.1.12.2
This is the data plane's answer, including any load sharing hash. When a user says "my traffic goes through R2", this command tells you whether the router agrees before you look at a single routing table.
Four Faults, Found in the FIB
With the tools in hand, the first fault is a prefix that exists in BGP but is forwarded somewhere unexpected. The customer prefix 192.0.2.0/24 is learned by iBGP from R4 with next hop 4.4.4.4.
R1#show ip route 192.0.2.0
Routing entry for 192.0.2.0/24
Known via "bgp 65001", distance 200, metric 0
Tag 65100, type internal
Last update from 4.4.4.4 00:22:15 ago
Routing Descriptor Blocks:
* 4.4.4.4, from 4.4.4.4, 00:22:15 ago
Route metric is 0, traffic share count is 1
AS Hops 1
Route tag 65100
MPLS label: none
R1#show ip cef 192.0.2.0/24 detail
192.0.2.0/24, epoch 2
recursive via 4.4.4.4
nexthop 10.1.13.3 GigabitEthernet0/1
The RIB entry looks normal, but the FIB shows the recursion resolving 4.4.4.4 through R3 on GigabitEthernet0/1, an upstream link, not through the internal network. The cause is visible one level down.
R1#show ip route 4.4.4.4
Routing entry for 4.4.4.0/24
Known via "bgp 65001", distance 20, metric 0
Tag 64600, type external
Last update from 10.1.13.3 00:03:41 ago
Routing Descriptor Blocks:
* 10.1.13.3, from 10.1.13.3, 00:03:41 ago
Route metric is 0, traffic share count is 1
AS Hops 2
Route tag 64600
MPLS label: none
The OSPF /32 for 4.4.4.4 has disappeared, perhaps because of an IGP problem, and the next most specific match is a /24 that an upstream provider happens to announce by eBGP. The iBGP next hop now resolves through the Internet. The FIB showed it immediately, because it prints the resolved path, not the intended one. The fix is to repair the IGP, and the long term protection is a next hop policy such as "#bgp nexthop route-map" that only accepts host routes for next hop resolution.
That fault was about recursion. The second one is about the adjacency. A new server on the connected subnet 10.1.50.0/24 cannot be reached, although the subnet is in the routing table.
R1#show ip cef 10.1.50.25
10.1.50.0/24
attached to GigabitEthernet0/3
R1#show ip arp 10.1.50.25
Protocol Address Age (min) Hardware Addr Type Interface
Internet 10.1.50.25 0 Incomplete ARPA
The packet matches the connected subnet, which uses a glean adjacency: CEF knows the subnet is on GigabitEthernet0/3 but has no MAC rewrite for the host yet, so it asks ARP to resolve it. ARP stays incomplete, so no host adjacency is ever built. Routing is not the problem at all. The server is in the wrong VLAN, or its interface is down, or a firewall is answering for it. Once ARP resolves, a /32 attached entry appears in the FIB and forwarding starts.
R1#show ip cef 10.1.50.25
10.1.50.25/32
attached to GigabitEthernet0/3
The third fault looks like a black hole, and it is a deliberate one gone wrong. The engineer added a static route to Null0 for 10.10.0.0/16 so that BGP could announce the aggregate, as covered in the previous module. A week later, traffic to a new subnet 10.10.77.0/24 is dropped.
R1#show ip cef 10.10.77.10
10.10.0.0/16
attached to Null0
The FIB shows the longest match for 10.10.77.10 is the /16 to Null0, which uses a null adjacency and drops the packet. The /24 for the new subnet was never added to the IGP, so nothing more specific exists. The Null0 route did exactly its job, protecting the block from loops. The fix is to add the real /24 route, not to remove the Null0 route.
The fourth fault involves two links. BGP multipath is enabled towards R2 and R3, and the operations team reports that GigabitEthernet0/0 is nearly full while GigabitEthernet0/1 is almost idle.
R1#show ip cef 198.51.100.0/24
198.51.100.0/24
nexthop 10.1.12.2 GigabitEthernet0/0
nexthop 10.1.13.3 GigabitEthernet0/1
R1#show ip cef exact-route 10.10.1.50 198.51.100.20
10.10.1.50 -> 198.51.100.20 =>IP adj out of GigabitEthernet0/0, addr 10.1.12.2
R1#show ip cef exact-route 10.10.1.51 198.51.100.20
10.10.1.51 -> 198.51.100.20 =>IP adj out of GigabitEthernet0/1, addr 10.1.13.3
Both paths are installed, so BGP and the RIB are working. The FIB shares traffic per destination with a hash of addresses, and the exact-route checks show that different source addresses land on different links. The imbalance comes from the traffic itself: most of the load is one large flow between two hosts, and a per-destination hash always sends one flow down one path. That is expected behaviour, and the next post in this series explains how the hash works and what can and cannot be changed.
A Troubleshooting Method That Always Works
The four faults have little in common at the surface, yet each was found with the same sequence, and that sequence is worth turning into a habit.
Start with the flow, not the route. Use "#show ip cef exact-route" with the real source and destination to see what the data plane will do. If the answer is already correct, the problem is not on this router.
Then read the FIB entry for the destination with "#show ip cef" and the detail keyword. Check three things: which prefix actually matched, because longest match may surprise you, whether the next hop is recursive and where the recursion ends, and what kind of entry it is. An address means a normal adjacency. "attached" to a subnet means the router still depends on ARP for the host. "attached to Null0" means the packet is dropped on purpose. "receive" means the packet is for the router itself.
Next, follow the adjacency. Use "#show adjacency" for the interface, and "#show ip arp" for Ethernet neighbours. An incomplete ARP entry is a Layer 2 problem, never a BGP problem.
Only then go back to the control plane. Compare the FIB with "#show ip route" for the destination and for its next hop, and with "#show ip bgp" for the prefix. If the RIB and the FIB disagree with each other, that is a platform problem to escalate. If they agree but neither matches your intent, the fault is in routing policy or the IGP.
R1#show ip cef 192.0.2.0/24 detail
192.0.2.0/24, epoch 2
recursive via 4.4.4.4
nexthop 10.1.12.2 GigabitEthernet0/0
After the IGP repair from the first fault, this is the healthy picture: the iBGP prefix recursing through 4.4.4.4 and leaving towards the internal network. Get used to reading this output, because in every later module, from route reflection to MPLS VPNs, the final proof that BGP works is a correct FIB entry.