Designing Fast Converging BGP Networks

by Ivan Pepelnjak

Border Gateway Protocol (BGP) was always considered a mastodonic routing protocol: huge, complex, hard to understand and configure, and very slow to converge. When Cisco decided to use it to implement layer-3 Virtual Private Networks (VPNs) due to its enormous route carrying capabilities, the slow convergence that did not bother us when we were implementing Internet with BGP, all of a sudden became a liability. Initially we were told to tweak numerous BGP parameters in Cisco IOS, but finally the Cisco engineers decided to fix the BGP code, resulting in a routing protocol with decent convergence times (assuming you configure it properly). In this article, you’ll see how you can optimize BGP convergence in your network without overloading the routers running it.

BGP Convergence Components

BGP is a distance vector routing protocol designed to perform well in environments with a huge number of routes (today there are almost 250,000 routes in the global Internet, and the number is rising). The convergence times were not considered important in initial BGP implementations and were commonly sacrificed to reduce the CPU utilization BGP imposed on the router.

Technical detail

The CPU utilization is less important on today’s high-end platforms with dedicated control- and data (forwarding) planes, as BGP uses only the control-plane CPU.

To understand the reasons for the BGP’s slow convergence, let’s analyze a typical scenario, where a router loses its BGP neighbor:

Even though the IP routing table might indicate the neighbor is no longer reachable, BGP will not trigger the convergence process until the TCP session with the neighbor times out or BGP session holddown timer expires (the default value is 3 minutes) or TCP disconnects the session due to TCP acknowledgement timeouts.

BGP limits the generation of outbound advertisements to its neighbors. By default, it sends the updates to the routers in the same autonomous system every five seconds and the external peers are notified only every 30 seconds.

BGP relies on another interior routing protocol (IGP) to calculate the best paths across your network. However, if the BGP neighbor is lost and IGP reports a change in the next-hop reachability, BGP might not react for a long time. For decades, IOS implementation of BGP verified the presence of the BGP next-hop every 60 seconds.

This article addresses the neighbor loss detection issues, as you can solve majority of BGP convergence problems by reducing the time it takes a BGP router to detect its BGP peer has disappeared. In a follow-up article, I’ll describe the next-hop-related improvements and other tuning mechanisms implemented in Cisco IOS.

Fast External Neighbor Loss Detection

IOS implementation of BGP got the first convergence boost years ago in IOS release 10.0 with the introduction of the fast external fall-over: external BGP (EBGP) session between directly connected EBGP neighbors is disconnected (resulting in route elimination and alternate route selection) as soon as the connected subnet through which the session is established is lost. You can see the results of this functionality in the logging printout taken from a BGP router (Listing 1): very soon after the interface goes down, the EBGP neighbor is lost as well.

Note

The internal BGP (IBGP) or multihop EBGP sessions are not affected by this functionality, they still rely on BGP hold timer to detect neighbor loss.

Listing 1

EBGP session is dropped on interface flap

22:46:28.354: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/2, changed state to down

22:46:29.402: %BGP-5-ADJCHANGE: neighbor 192.168.2.1 Down Interface flap

The fast external fall-over functionality is enabled by default; you can disable it globally with the no bgp fast-external-fallover router configuration command (for example, if flaky external links continuously disrupt EBGP sessions). After the no bgp fast-external-fallover has been configured, the interface flap no longer causes immediate loss of EBGP session, as illustrated in Listing 2. The EBGP sessions are disconnected only after the BGP hold timer expires (note the 20 second difference between the interface flap and BGP hold timer expiration in Listing 2).

Technical detail

If needed, you can fine-tune the BGP timers with the neighbor address timers keepalive holdtime router configuration command; it’s best if you use BGP session template to ensure consistent behavior toward all neighbors.

Listing 2

Interface flap no longer causes an EBGP session loss

23:43:05.702: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial1/2, changed state to down

23:45:26.650: %BGP-5-ADJCHANGE: neighbor 192.168.2.1 Down BGP Notification sent

23:45:26.654: %BGP-3-NOTIFICATION: sent to neighbor 192.168.2.1 4/0 (hold time expired) 0 bytes

Internal Neighbor Loss Detection

The detection of IBGP neighbor loss has always been a major pain, as you had to rely on tweaking BGP timers. Similar to regular BGP update packets, BGP keepalive messages are exchanged across the reliable TCP connection between the BGP peers; reduced keepalive and holdtime values thus always result in increased CPU utilization.

IOS release 12.0S and 12.3T addressed this deficiency with the BGP Support for Fast Peering Deactivation. The idea is quite simple: as soon as the IP address of the BGP peer disappears from the IP routing table, the BGP session with the peer is deactivated, resulting in immediate convergence. The fast peering deactivation is configured with the neighbor fall-over router configuration command.

For example, fast peering deactivation of the IBGP sessions on the POP router in the sample network displayed in Figure 1 is configured with two neighbor command highlighted in Listing 3.

Figure 1

Sample network

Listing 3

Fast peering deactivation configured on the POP router

router bgp 65000

neighbor 10.0.0.21 remote-as 65000

neighbor 10.0.0.21 update-source Loopback0

neighbor 10.0.0.21 fall-over

neighbor 10.0.0.22 remote-as 65000

neighbor 10.0.0.22 update-source Loopback0

neighbor 10.0.0.22 fall-over

Alternatively, you could use the BGP peer session templates, resulting in a very concise IBGP configuration displayed in Listing 4.

Listing 4

BGP peer session template significantly reduces BGP configuration

router bgp 65000

template peer-session IBGP

remote-as 65000

update-source Loopback0

fall-over

neighbor 10.0.0.21 inherit peer-session IBGP

neighbor 10.0.0.22 inherit peer-session IBGP

With the fast session deactivation configured on the POP router, the IBGP session is terminated as soon as the IP route toward the IBGP peer is lost (Listing 5).

Debugging hint

When an extended IP access-list is used to debug the IP routing table, you can match the IP address as well as the subnet mask. The access-list 199 in Listing 5 is used to match only the host routes within the 10.0.0.0/8 address space.

Listing 5

IBGP session is terminated as soon as the IP route to the IBGP peer is lost

POP#show debug

IP routing:

IP routing debugging is on for access list 199

POP#show access-list 199

Extended IP access list 199

10 permit ip 10.0.0.0 0.255.255.255 host 255.255.255.255 (90 matches)

POP#

13:25:52.126: RT: del 10.0.0.22/32 via 10.0.2.1, ospf metric [110/129]

13:25:52.126: RT: delete subnet route to 10.0.0.22/32

13:25:52.126: RT: NET-RED 10.0.0.22/32

13:25:52.126: RT: Try lookup less specific 10.0.0.22/32, default 1

13:25:52.126: RT: Failed found subnet on less specific

13:25:52.126: RT: return NULL

13:25:52.130: %BGP-5-ADJCHANGE: neighbor 10.0.0.22 Down Route to peer lost

Obviously, you should use the fast session deactivation only if you’re running a fast converging interior routing protocol (IGP), for example OSPF, IS-IS or EIGRP. Your IGP should be able to find an alternate route to the BGP peer (if one exists) immediately. If there is even a slight interruption between the moment the original route to the BGP peer is lost until an alternate route is inserted in the IP routing table, the BGP session would already be disconnected (there is no hold-down or delay in BGP session deactivation).

Complex Scenarios

The fast session deactivation works extremely well if you don’t use summary routes or default routes in your network; the summary or default IGP routes almost always break it. For example, if you decide to use partial BGP routing in your network and advertise OSPF default routes from X1 and X2 (the modified IP routing table on the POP router is shown in Listing 6), the path toward the IBGP peer on the POP router would never be lost, as there would always be a less-specific route toward the IBGP peer’s IP address (the default route). This behavior is very evident in the debugging printouts in Listing 7; the IGBP session is terminated long after the host route toward the IBGP peer disappears (after the expiration of the BGP hold timer).

Design tip

You could bypass this limitation by announcing the default route in BGP, as BGP routes are rejected as possible paths toward BGP peers.

Listing 6

The POP router receives two default routes through OSPF

POP#show ip route 0.0.0.0

Routing entry for 0.0.0.0/0, supernet

Known via "ospf 1", distance 110, metric 1, candidate default path

Tag 1, type extern 2, forward metric 128

Last update from 10.0.2.1 00:01:00 ago

Routing Descriptor Blocks:

10.0.2.1, from 10.0.0.22, 00:01:00 ago

Route metric is 1, traffic share count is 1

Route tag 1

* 10.0.1.1, from 10.0.0.21, 00:06:05 ago

Route metric is 1, traffic share count is 1

Route tag 1

Listing 7

IBGP peer is lost only after the BGP hold timer expires

14:08:37.723: RT: del 10.0.0.22/32 via 10.0.2.1, ospf metric [110/129]

14:08:37.723: RT: add 10.0.0.22/32 via 10.0.1.1, ospf metric [110/139]

14:08:37.723: RT: NET-RED 10.0.0.22/32

14:08:37.739: RT: del 10.0.0.22/32 via 10.0.1.1, ospf metric [110/139]

14:08:37.739: RT: delete subnet route to 10.0.0.22/32

14:08:37.739: RT: NET-RED 10.0.0.22/32

14:08:37.739: RT: Try lookup less specific 10.0.0.22/32, default 1

14:08:37.739: RT: Failed found subnet on less specific

14:08:37.739: RT: return default 0.0.0.0/0

14:10:47.899: %BGP-5-ADJCHANGE: neighbor 10.0.0.22 Down BGP Notification sent

14:10:47.903: %BGP-3-NOTIFICATION: sent to neighbor 10.0.0.22 4/0 (hold time expired) 0 bytes

To address the undesired side effects of the summary or default IGP routes in the network, the Selective Address Tracking for BGP Fast Session Deactivation introduced in IOS release 12.4(4)T enhanced the neighbor fall-over command with a route-map option that allows you to select which IP routes can be used to reach BGP peers. In our sample network, all IBGP peers are within the IP address space 10.0.0.0/8 and have to be reached via a host route (the OSPF-advertised route toward the IGBP peer’s loopback address). We can thus define a route map (see Listing 8) that selects only the OSPF host routes within the desired address range (a prefix-list is used within the route map to match the addresses and subnet masks of the IP routes).

Listing 8

A route-map matching loopback addresses in AS 65000

ip prefix-list Loopbacks description Match BGP router loopbacks

ip prefix-list Loopbacks seq 10 permit 10.0.0.0/8 ge 32

route-map Loopbacks permit 10

match ip address prefix-list Loopbacks

match source-protocol ospf 1

When the route map is applied to IBGP neighbors with the neighbor fall-over route-map router configuration command (Listing 9), the default route is no longer considered as a potential path toward the IBGP peer; the BGP session is terminated as soon as the host route toward the BGP peer’s loopback address disappears (Listing 10).

Note

The configuration in Listing 9 clearly demonstrates the scalability offered by the BGP peer templates; we’ve changed only the peer session template parameters and the change was applied to all IBGP peers.

Listing 9

BGP fast session deactivation with a route-map

router bgp 65000

template peer-session IBGP

remote-as 65000

update-source Loopback0

fall-over route-map Loopbacks

exit-peer-session

Listing 10

Caption

14:36:11.024: RT: del 10.0.0.22/32 via 10.0.2.1, ospf metric [110/129]

14:36:11.028: RT: add 10.0.0.22/32 via 10.0.1.1, ospf metric [110/139]

14:36:11.028: RT: NET-RED 10.0.0.22/32

14:36:11.072: RT: del 10.0.0.22/32 via 10.0.1.1, ospf metric [110/139]

14:36:11.072: RT: delete subnet route to 10.0.0.22/32

14:36:11.072: RT: NET-RED 10.0.0.22/32

14:36:11.072: RT: Try lookup less specific 10.0.0.22/32, default 1

14:36:11.072: RT: Failed found subnet on less specific

14:36:11.072: RT: return default 0.0.0.0/0

14:36:11.072: %BGP-5-ADJCHANGE: neighbor 10.0.0.22 Down Route to peer lost

EBGP Fast Session Deactivation

BGP fast session deactivation works on all BGP sessions. You can use it to quickly detect failures of EBGP sessions established between loopback interfaces of EBGP peers (a common scenario used to implement simple EBGP load sharing) or to detect EBGP neighbor loss when you had to disable fast external fall-over.

Note

Fast external fall-over is a global setting, whereas the fast session deactivation is configured per-neighbor. You can thus disable fast external fall-over with the no bgp fast-external-fallover router configuration command and retain the quick response to interface failures for a selected subset of EBGP neighbors (configured with the neighbor fall-over router configuration command).

The EBGP fast session deactivation is identical to the IBGP case; the configuration command to use is neighbor fall-over. You can even reflect the rule that the BGP peer has to be directly connected with a route-map that matches only connected subnets. The relevant parts of the X1’s configuration are included in Listing 11.

Listing 11

Fast EBGP session deactivation configured on X1

router bgp 65000

neighbor 192.168.1.5 remote-as 65100

neighbor 192.168.1.5 fall-over route-map Connected

route-map Connected permit 10

match source-protocol connected

Summary

BGP has been considered a slowly converging protocol due to its distance-vector nature, slow propagation of routing updates and enormous amounts of time it usually takes to detect a BGP neighbor loss. Cisco has addressed some of these concerns years ago when the fast fall-over of directly connected EBGP session has been implemented. However, the detection of peer failures in IBGP or multi-hop EBGP cases was still based solely on BGP hold time, which has a default value of 3 minutes. It was always possible to fine-tune BGP keepalive messages, but the drastic reduction in keepalive- and hold timer resulted in increased CPU utilization on routers with a significant number of BGP peers.

IOS release 12.3T introduced support for fast peering session deactivation which terminates a BGP session as soon as the IP address of the BGP peer becomes unreachable. You should use this feature only in fast converging networks (ideally based on a single routing protocol), as any potential alternate route should replace the primary route toward a BGP peer immediately after the primary path failure has been detected.

The fast peering session deactivation does not work correctly in environments where IGP carries summary or default routes (BGP summary routes are not considered as a viable path toward BGP peers). The BGP selective address tracking introduced in IOS release 12.4(4)T solves this problem, as it allows you to select the potential routes toward the BGP peer with a route map.

The fast peering session deactivation can be used on IBGP or EBGP sessions, where it effectively replaces the older fail-over mechanism, allowing you to configure different fail-over behavior for each BGP neighbor (the old mechanism was enabled or disabled globally).