MSS, MTU and Path MTU Discovery: Why a BGP Session Comes Up and Then Dies

Segments, MTU and Why BGP Cares

There is a BGP failure that every service provider engineer meets sooner or later. The session comes up, reaches Established, exchanges a few prefixes, and then stops. A few minutes later it goes down with a hold timer expiry, comes back up, and repeats the cycle forever. The configuration is correct, the neighbour is reachable, and ping works. The cause is almost never BGP. It is the size of the TCP segments carrying BGP, and the path they cross.

Start with the terms. The MTU, maximum transmission unit, is the largest IP packet a link can carry, 1500 bytes on standard Ethernet. The MSS, maximum segment size, is the largest amount of data TCP puts in one segment. On a 1500-byte link it is 1460 bytes: the MTU minus 20 bytes of IP header and 20 bytes of TCP header. Both ends announce their MSS in the SYN packets during the handshake, and each side then sends segments no larger than the smaller value.

For most applications MSS is a detail. For BGP it decides speed. Small BGP messages, such as OPEN and KEEPALIVE, fit easily in any segment. UPDATE messages carrying a full table do not. They are packed into segments as large as TCP allows, so a bigger MSS means fewer packets, fewer acknowledgements and faster convergence. A full Internet table of about 1.08 million IPv4 prefixes takes noticeably longer to send in 536-byte segments than in 1460-byte ones.

R1#show ip bgp neighbors 10.1.12.2 | include max data segment|Option Flags|path-mtu
  Transport(tcp) path-mtu-discovery is enabled
Option Flags: nagle, path mtu capable
Datagrams (max data segment is 1460 bytes):

This R1 peers with a directly connected eBGP neighbour on Ethernet, and TCP uses the full 1460 bytes. The "path mtu capable" flag shows that path MTU discovery is active on this session.

Path MTU Discovery and the Black Hole

Directly connected neighbours are the easy case, because the router knows the MTU of its own interface. iBGP sessions between loopbacks, and multihop eBGP, cross other links whose MTU the router cannot see. One of them may be a GRE tunnel, an MPLS link with several labels, or a provider circuit with a lower MTU. TCP needs a way to find the smallest MTU on the path, and that is Path MTU Discovery (PMTUD).

PMTUD works by sending packets with the Don't Fragment bit set. If a router on the path cannot forward a packet because it is too large, it drops it and returns an ICMP "fragmentation needed" message that includes the MTU it can carry. The sender lowers its segment size and tries again. On IOS XE, BGP sessions use PMTUD by default. Other TCP sessions on the router do not, because the global "#ip tcp path-mtu-discovery" is off by default.

When PMTUD is turned off for a session, IOS falls back to a conservative rule. For a destination that is not directly connected, TCP uses an MSS of 536 bytes, the safe minimum every IP path must carry. Directly connected neighbours still get 1460. The session is safe but slow.

Now the failure. Imagine R1 and R4 peering between loopbacks, with a GRE tunnel of MTU 1476 somewhere in the path, and a firewall that drops all ICMP. The TCP handshake works, because SYN packets are tiny. The OPEN and KEEPALIVE messages work, because they are small. The session reaches Established. Then R1 starts sending UPDATE messages in full 1460-byte segments, which become 1500-byte packets with the Don't Fragment bit set. The tunnel router cannot forward them and sends back an ICMP message, which the firewall drops. R1 never learns about the smaller MTU, so it keeps retransmitting the same large segment.

R1#show ip bgp neighbors 4.4.4.4 | include BGP state|Retrans|max data segment|snd
  BGP state = Established, up for 00:02:37
Retrans            19         19      0x2B44A10
sndwnd:  16384  scale:      0  maxrcvwnd:  16384
Datagrams (max data segment is 1460 bytes):
Sent: 41 (retransmit: 19, fastretransmit: 0, partialack: 0, Second Congestion: 0), with data: 22, total data bytes: 25140

The retransmit counter keeps climbing, and the same segment is sent again and again. Meanwhile the small KEEPALIVE messages stuck behind the large segment in the TCP stream cannot be delivered either, because TCP delivers data in order. R4 receives nothing new, its hold timer expires, and it tears the session down. Then the whole cycle starts again. This is the PMTUD black hole, and it is the classic cause of a BGP session that flaps every three minutes with "hold time expired" as the reason.

R4#
%BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 4/0 (hold time expired) 0 bytes
%BGP-5-ADJCHANGE: neighbor 1.1.1.1 Down BGP Notification sent
%BGP_SESSION-5-ADJCHANGE: neighbor 1.1.1.1 IPv4 Unicast topology base removed from session  BGP Notification sent

Fixing It, and the CCIE Details

With the cause clear, there are four fixes, and the best one depends on who controls the path.

The first and best fix is to let the ICMP messages through. PMTUD only works if "fragmentation needed" messages reach the sender, so firewalls and ACLs on the path should permit them. Blocking all ICMP for security breaks TCP in exactly this way.

The second fix is to raise the MTU of the path so that 1500-byte packets fit. In an MPLS core this is the normal design: core links carry a larger MTU so that labelled customer packets and loopback-sourced iBGP never hit the limit.

The third fix is to make TCP send smaller segments from the start. The global "#ip tcp mss" sets the MSS used by TCP sessions that the router itself originates, BGP included. On the tunnel, "#ip tcp adjust-mss" rewrites the MSS in SYN packets passing through, which protects any session crossing the tunnel.

R1(config)#ip tcp mss 1360
R1#show ip bgp neighbors 4.4.4.4 | include BGP state|max data segment|retransmit
  BGP state = Established, up for 00:12:05
Datagrams (max data segment is 1360 bytes):
Sent: 412 (retransmit: 0, fastretransmit: 0, partialack: 0, Second Congestion: 0), with data: 389, total data bytes: 1641322

After the change the session negotiates 1360 bytes, retransmissions stop and the full table flows. The fourth fix, turning PMTUD off for the neighbour with "#neighbor 4.4.4.4 transport path-mtu-discovery disable", also stops the black hole by dropping to 536 bytes, at the cost of slower updates.

Two details often surprise engineers at CCIE level. The first concerns MD5 authentication. The MD5 option adds 18 bytes to every TCP header, padded to 20, yet "#show ip bgp neighbors" still prints "max data segment is 1460 bytes" on an Ethernet session with MD5. The option is not included in that figure, and the Option Flags line simply adds md5. Keep the extra 20 bytes in mind when you calculate how close a session is to the path MTU.

R1#show ip bgp neighbors 10.1.12.2 | include max data segment|Option Flags
Option Flags: nagle, path mtu capable, md5
Datagrams (max data segment is 1460 bytes):

The second concerns IOS XR. There, TCP path MTU discovery and selective acknowledgement are off by default, and the MSS for TCP sessions is set with "#tcp mss", with a range of 68 to 10000. A mixed IOS XE and IOS XR network can therefore use different segment sizes on the two ends of the same design, and only reading the session on each router shows which.

The takeaway is a rule worth remembering for every later module. When a BGP session flaps with hold time expired, check the retransmit counter and the segment size before you check a single BGP setting. A session that comes up and then dies under load is very often a session whose packets are too big for the path.