CEF Load Sharing: Why Two Equal BGP Paths Rarely Carry Equal Traffic
Equal Paths in the RIB, Shared in the FIB
Ask a new engineer what happens when a router has two equal paths to a destination, and the usual answer is "it alternates between them". On a Cisco router running CEF that is almost never true. The routing table decides which paths are equal. Cisco Express Forwarding decides which packet uses which path, and by default it does not alternate at all. Understanding how it chooses explains a whole family of tickets about unbalanced links, and it matters directly for BGP, because BGP multipath is the most common reason for an edge router to have several paths to the same prefix.
The first step happens in the control plane. BGP installs only one best path in the routing table by default. To give CEF anything to share, BGP must be told to install more, with "#maximum-paths" for eBGP or "#maximum-paths ibgp" for iBGP, and the paths must also pass BGP's own strict equality rules.
R1(config)#router bgp 65001
R1(config-router)#address-family ipv4 unicast
R1(config-router-af)#maximum-paths 2
R1#show ip route 198.51.100.0
Routing entry for 198.51.100.0/24
Known via "bgp 65001", distance 20, metric 0
Tag 65100, type external
Last update from 10.1.13.3 00:00:21 ago
Routing Descriptor Blocks:
10.1.13.3, from 10.1.13.3, 00:00:21 ago
Route metric is 0, traffic share count is 1
AS Hops 2
Route tag 65100
MPLS label: none
* 10.1.12.2, from 10.1.12.2, 00:00:21 ago
Route metric is 0, traffic share count is 1
AS Hops 2
Route tag 65100
MPLS label: none
R1#show ip cef 198.51.100.0/24
198.51.100.0/24
nexthop 10.1.12.2 GigabitEthernet0/0
nexthop 10.1.13.3 GigabitEthernet0/1
The RIB now holds two descriptor blocks, and the FIB holds two next hops for the prefix. The asterisk in the routing table only marks the path shown first in the output. It does not mean that path carries the traffic. From here on, the FIB makes every decision.
How CEF Chooses a Path
That decision is per destination by default, and the word "destination" is a little misleading. CEF does not look only at the destination address. It feeds the packet's source and destination addresses into a hash function, and the result picks one of the available paths. Every packet of the same source and destination pair produces the same hash, so it always takes the same path. That keeps the packets of a flow in order, which TCP and real-time applications depend on.
The hash result does not point at a path directly. It points at one of 16 hash buckets, and each bucket is assigned to a path. With two paths, eight buckets go to each. With four paths, four buckets each. With three paths, 16 cannot be divided evenly, so CEF uses 15 buckets, five per path, and leaves the last one unused. The bucket table is what turns a hash into a reasonably even split across many flows.
The hash function itself is selected with "#ip cef load-sharing algorithm". Modern IOS XE uses the universal algorithm by default. It hashes the source address, the destination address and a router-specific ID. That ID matters more than it seems. With the older "original" algorithm, every router in a chain used the same calculation, so a flow that hashed to "left" on the first router also hashed to "left" on the next one. In a network with several layers of equal-cost paths, half of the downstream links could end up carrying almost nothing. This effect is called polarisation, and the per-router ID in the universal algorithm is there to break it.
R1(config)#ip cef load-sharing algorithm universal 5A1F09C3
Setting the ID manually is rarely needed, but it is useful when you want two routers to make different choices on purpose.
Layer 4 ports are not part of the default hash. If most of the traffic is one host talking to another host, every packet has the same source and destination addresses and lands in the same bucket, whatever the application. The include-ports option adds TCP and UDP ports to the hash, so separate sessions between the same two hosts can spread across paths.
R1(config)#ip cef load-sharing algorithm include-ports source destination
This helps with backup servers, storage replication and other cases where a few hosts generate most of the load. The cost is that packets without port information, such as IP fragments after the first one, may hash differently from the first fragment of the same packet. Test it with your real traffic before you rely on it.
The way to see the decision for a specific flow is "#show ip cef exact-route". It runs the full hash with the addresses you give it, and optionally the ports, and prints the adjacency the packet would use.
R1#show ip cef exact-route 10.10.1.50 198.51.100.20
10.10.1.50 -> 198.51.100.20 =>IP adj out of GigabitEthernet0/0, addr 10.1.12.2
R1#show ip cef exact-route 10.10.1.51 198.51.100.20
10.10.1.51 -> 198.51.100.20 =>IP adj out of GigabitEthernet0/1, addr 10.1.13.3
Two sources, two different links. This is per-destination sharing working exactly as designed.
The Per-Packet Trap and Practical Design
Once engineers see that a single large flow always uses one link, many look for a way to force alternation, and Cisco has one. The interface command "#ip load-sharing per-packet" switches CEF to per-packet sharing on that interface, sending successive packets down different paths. It is almost always the wrong answer. Packets of the same TCP session arrive out of order at the far end, TCP treats the reordering as loss, and throughput falls instead of rising. Voice and video suffer too. Per-packet sharing is also not supported at all on some platforms, the ASR 1000 among them. Leave the default in place unless a specific design, such as two equal links feeding a device that reassembles traffic, truly needs it.
With the hashing rules clear, BGP multipath design becomes a matter of realistic expectations. Equal-cost sharing is statistical. It works well when there are many flows of similar size, such as a large population of Internet users, and badly when a few elephant flows dominate. An edge router facing two transit providers with thousands of customers behind it will usually see a fair split. A data centre interconnect carrying three replication sessions will not, however carefully BGP is tuned.
Unequal links need a different tool altogether. If one uplink is 10 Gbps and the other 1 Gbps, an even hash would overload the smaller one. BGP can account for this with the link bandwidth extended community and "#bgp dmzlink-bw", which makes CEF share traffic in proportion to the advertised bandwidth instead of evenly, and that feature has its own lesson later in the course.
A few habits tie the topic together. Remember that BGP installs one path until you configure multipath, so "load balancing does not work" often means there is only one path in the RIB. Read the FIB with "#show ip cef" to confirm how many paths CEF actually holds. Use "#show ip cef exact-route" with real addresses before blaming the hash. Keep the universal algorithm so that polarisation does not appear in multi-tier designs. And treat per-packet sharing as a last resort, not a fix for an unbalanced graph. A link that looks unfair on a monitoring dashboard is usually CEF keeping flows intact, which is exactly what it is supposed to do.