Archive
Using BFD to Track WAN Status and Change HSRP Priority
It’s been five years since I started this blog! Time flies and a lot has happened since. Thanks for being along for the ride. What better way to celebrate than a blog post?
This post is going to be short and to the point.
Many of us run HSRP or VRRP. It is quite common to run it in a topology where you have dual routers and dual exits to the WAN and you don’t want to black hole your traffic.
One traditional way of achieving this is by tracking the interface that goes towards the WAN. There are a couple of drawbacks to this approach though:
- You may not get link down on failure (connecting to switch)
- You may experience an error that does not produce link down event
The next option is to use IP SLA that sends ICMP Echo towards the next-hop of the WAN or some destination further into the network. Ehanced Object Tracking (EOT) can then be used to create a track object that decrements the priority of the HSRP active router when the ICMP Echo probe fails. This works better but there are still some drawbacks to this approach:
- Frequency can’t be set to lower than one second
- There is no multiplier, one failed ping is enough which can lead to false positives
- False positives will lead to the state changing more than necessary
- The above can be solved by using a delay for the tracking object
- Using IP SLA is likely more CPU intensive than using BFD
Unfortunately there is no way to directly configure HSRP to check the status of BFD running on your WAN interface. That does not mean we can’t solve the task at hand though. BFD is supported over static routes. What if we insert a dummy route into the RIB when BFD is running successfully over the WAN link and track that this route is installed into the RIB. If it is not installed it must mean that BFD has failed and that the HSRP priority of the active router should be decremented.
The configuration is quite simple. In my lab I have an ISP router with the following config:
interface GigabitEthernet1.100
encapsulation dot1Q 100
ip address 10.0.0.1 255.255.255.252
bfd interval 500 min_rx 500 multiplier 3!
router bgp 1
bgp log-neighbor-changes
network 1.1.1.1 mask 255.255.255.255
neighbor 10.0.0.2 remote-as 2
neighbor 10.0.0.2 fall-over bfd
I’m using BGP in this case to have BFD packets sent over the link. There needs to be a protocol registered with BFD for the packets to be sent. It would be more likely for the ISP to configure a static route using BFD as well. If you are already running BGP, this configuration may be overkill since you could track routes coming from BGP.
This is then the configuration of the active HSRP router:
track 1 ip route 169.254.0.0 255.255.0.0 reachability
!
interface GigabitEthernet1
no ip address
negotiation auto
!
interface GigabitEthernet1.100
encapsulation dot1Q 100
ip address 10.0.0.2 255.255.255.252
bfd interval 500 min_rx 500 multiplier 3
!
interface GigabitEthernet1.200
encapsulation dot1Q 200
ip address 10.0.10.2 255.255.255.0
standby 1 ip 10.0.10.1
standby 1 priority 110
standby 1 preempt
standby 1 track 1 decrement 11
!
router bgp 2
bgp log-neighbor-changes
neighbor 10.0.0.1 remote-as 1
neighbor 10.0.0.1 fall-over bfd
!
ip route static bfd GigabitEthernet1.100 10.0.0.1
ip route 169.254.0.0 255.255.0.0 GigabitEthernet1.100 10.0.0.1
To trigger the BFD packets being sent over the WAN link we first have a static route pointing out the egress interface and the next-hop.
ip route static bfd GigabitEthernet1.100 10.0.0.1
Then we put a standard IP route statement which will insert the dummy route. It is important to point out the egress interface though for single-hop BFD.
ip route 169.254.0.0 255.255.0.0 GigabitEthernet1.100 10.0.0.1
EOT is used to track if the route is installed into the RIB or not.
track 1 ip route 169.254.0.0 255.255.0.0 reachability
The HSRP priority is decremented if the route is not in the RIB which will make the standby router become active.
standby 1 track 1 decrement 11
We can then test if it works by shutting down the interface on the ISP router.
ISP1(config)#int gi1
ISP1(config-if)#shut
ISP1(config-if)#
The change in the RIB is detected by the HSRP active router:
*Jul 29 15:57:23.278: %TRACK-6-STATE: 1 ip route 169.254.0.0/16 reachability Up -> Down
*Jul 29 15:57:24.872: %HSRP-5-STATECHANGE: GigabitEthernet1.200 Grp 1 state Active -> Speak
*Jul 29 15:57:36.582: %HSRP-5-STATECHANGE: GigabitEthernet1.200 Grp 1 state Speak -> Standby
The standby router takes over:
*Jul 29 15:57:24.872: %HSRP-5-STATECHANGE: GigabitEthernet1.200 Grp 1 state Standby -> Active
What are the advantages of this setup compared to IP SLA?
- Lightweight protocol designed to test reachability
- Can send packets faster than one second in between each
- Can define what multiplier to use
The drawback may be that you have to get your ISP to run BFD and that they need to put a static route in on their side as well. This can be a real route or a dummy route though.
Hopefully this post was somewhat useful and you’ll stay with me for another five years. Thanks for reading!
HSRP AWARE PIM
In environments that require redundancy towards clients, HSRP will normally be running. HSRP is a proven protocol and it works but how do we handle when we have clients that need multicast? What triggers multicast to converge when the Active Router (AR) goes down? The following topology is used:
One thing to notice here is that R3 is the PIM DR even though R2 is the HSRP AR. The network has been setup with OSPF, PIM and R1 is the RP. Both R2 and R3 will receive IGMP reports but only R3 will send PIM Join, due to it being the PIM DR. R3 builds the (*,G) towards the RP:
R3#sh ip mroute 239.0.0.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 239.0.0.1), 02:54:15/00:02:20, RP 1.1.1.1, flags: SJC
Incoming interface: Ethernet0/0, RPF nbr 13.13.13.1
Outgoing interface list:
Ethernet0/2, Forward/Sparse, 00:25:59/00:02:20
We then ping 239.0.0.1 from the multicast source to build the (S,G):
S1#ping 239.0.0.1 re 3 Type escape sequence to abort. Sending 3, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds: Reply to request 0 from 10.0.0.10, 35 ms Reply to request 1 from 10.0.0.10, 1 ms Reply to request 2 from 10.0.0.10, 2 ms
The (S,G) has been built:
R3#sh ip mroute 239.0.0.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 239.0.0.1), 02:57:14/stopped, RP 1.1.1.1, flags: SJC
Incoming interface: Ethernet0/0, RPF nbr 13.13.13.1
Outgoing interface list:
Ethernet0/2, Forward/Sparse, 00:28:58/00:02:50
(41.41.41.10, 239.0.0.1), 00:02:03/00:00:56, flags: JT
Incoming interface: Ethernet0/0, RPF nbr 13.13.13.1
Outgoing interface list:
Ethernet0/2, Forward/Sparse, 00:02:03/00:02:50
The unicast and multicast topology is not currently congruent, this may or may not be important. What happens when R3 fails?
R3(config)#int e0/2 R3(config-if)#sh R3(config-if)#
No replies to the pings coming in until PIM on R2 detects that R3 is gone and takes over the DR role, this will take between 60 to 90 seconds with the default timers in use.
S1#ping 239.0.0.1 re 100 ti 1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 239.0.0.1, timeout is 1 seconds: Reply to request 0 from 10.0.0.10, 18 ms Reply to request 1 from 10.0.0.10, 2 ms.................................................................... ....... Reply to request 77 from 10.0.0.10, 10 ms Reply to request 78 from 10.0.0.10, 1 ms Reply to request 79 from 10.0.0.10, 1 ms Reply to request 80 from 10.0.0.10, 1 ms
We can increase the DR priority on R2 to make it become the DR.
R2(config-if)#ip pim dr-priority 50 *Feb 13 12:42:45.900: %PIM-5-DRCHG: DR change from neighbor 10.0.0.3 to 10.0.0.2 on interface Ethernet0/2
HSRP aware PIM is a feature that started appearing in IOS 15.3(1)T and makes the HSRP AR become the PIM DR. It will also send PIM messages from the virtual IP which is useful in situations where you have a router with a static route towards an Virtual IP (VIP). This is how Cisco describes the feature:
HSRP Aware PIM enables multicast traffic to be forwarded through the HSRP active router (AR), allowing PIM to leverage HSRP redundancy, avoid potential duplicate traffic, and enable failover, depending on the HSRP states in the device. The PIM designated router (DR) runs on the same gateway as the HSRP AR and maintains mroute states.
In my topology, I am running HSRP towards the clients, so even though this feature sounds as a perfect fit it will not help me in converging my multicast. Let’s configure this feature on R2:
R2(config-if)#ip pim redundancy HSRP1 hsrp dr-priority 100 R2(config-if)# *Feb 13 12:48:20.024: %PIM-5-DRCHG: DR change from neighbor 10.0.0.3 to 10.0.0.2 on interface Ethernet0/2
R2 is now the PIM DR, R3 will now see two PIM neighbors on interface E0/2:
R3#sh ip pim nei e0/2
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
P - Proxy Capable, S - State Refresh Capable, G - GenID Capable
Neighbor Interface Uptime/Expires Ver DR
Address Prio/Mode
10.0.0.1 Ethernet0/2 00:00:51/00:01:23 v2 0 / S P G
10.0.0.2 Ethernet0/2 00:07:24/00:01:23 v2 100/ DR S P G
R2 now has the (S,G) and we can see that it was the Assert winner because R3 was previously sending multicasts to the LAN segment.
R2#sh ip mroute 239.0.0.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 239.0.0.1), 00:20:31/stopped, RP 1.1.1.1, flags: SJC
Incoming interface: Ethernet0/0, RPF nbr 12.12.12.1
Outgoing interface list:
Ethernet0/2, Forward/Sparse, 00:16:21/00:02:35
(41.41.41.10, 239.0.0.1), 00:00:19/00:02:40, flags: JT
Incoming interface: Ethernet0/0, RPF nbr 12.12.12.1
Outgoing interface list:
Ethernet0/2, Forward/Sparse, 00:00:19/00:02:40, A
What happens when R2’s LAN interface goes down? Will R3 become the DR? And how fast will it converge?
R2(config)#int e0/2 R2(config-if)#sh
HSRP changes to active on R3 but the PIM DR role does not converge until the PIM query interval has expired (3x hellos).
*Feb 13 12:51:44.204: HSRP: Et0/2 Grp 1 Redundancy "hsrp-Et0/2-1" state Standby -> Active
R3#sh ip pim nei e0/2
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
P - Proxy Capable, S - State Refresh Capable, G - GenID Capable
Neighbor Interface Uptime/Expires Ver DR
Address Prio/Mode
10.0.0.1 Ethernet0/2 00:04:05/00:00:36 v2 0 / S P G
10.0.0.2 Ethernet0/2 00:10:39/00:00:36 v2 100/ DR S P G
R3#
*Feb 13 12:53:02.013: %PIM-5-NBRCHG: neighbor 10.0.0.2 DOWN on interface Ethernet0/2 DR
*Feb 13 12:53:02.013: %PIM-5-DRCHG: DR change from neighbor 10.0.0.2 to 10.0.0.3 on interface Ethernet0/2
*Feb 13 12:53:02.013: %PIM-5-NBRCHG: neighbor 10.0.0.1 DOWN on interface Ethernet0/2 non DR
We lose a lot of packets while waiting for PIM to converge:
S1#ping 239.0.0.1 re 100 time 1 Type escape sequence to abort. Sending 100, 100-byte ICMP Echos to 239.0.0.1, timeout is 1 seconds: Reply to request 0 from 10.0.0.10, 5 ms Reply to request 0 from 10.0.0.10, 14 ms................................................................... Reply to request 68 from 10.0.0.10, 10 ms Reply to request 69 from 10.0.0.10, 2 ms Reply to request 70 from 10.0.0.10, 1 ms
HSRP aware PIM didn’t really help us here… So when is it useful? If we use the following topology instead:
The router R5 has been added and the receiver sits between R5 instead. R5 does not run routing with R2 and R3, only static routes pointing at the RP and the multicast source:
R5(config)#ip route 1.1.1.1 255.255.255.255 10.0.0.1 R5(config)#ip route 41.41.41.0 255.255.255.0 10.0.0.1
Without HSRP aware PIM, the RPF check would fail because PIM would peer with the physical address but R5 sees three neighbors on the segment, where one is the VIP:
R5#sh ip pim nei
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
P - Proxy Capable, S - State Refresh Capable, G - GenID Capable
Neighbor Interface Uptime/Expires Ver DR
Address Prio/Mode
10.0.0.2 Ethernet0/0 00:03:00/00:01:41 v2 100/ DR S P G
10.0.0.1 Ethernet0/0 00:03:00/00:01:41 v2 0 / S P G
10.0.0.3 Ethernet0/0 00:03:00/00:01:41 v2 1 / S P G
R2 will be the one forwarding multicast during normal conditions due to it being the PIM DR via HSRP state of active router:
R2#sh ip mroute 239.0.0.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 239.0.0.1), 00:02:12/00:02:39, RP 1.1.1.1, flags: S
Incoming interface: Ethernet0/0, RPF nbr 12.12.12.1
Outgoing interface list:
Ethernet0/2, Forward/Sparse, 00:02:12/00:02:39
Let’s try a ping from the source:
S1#ping 239.0.0.1 re 3 Type escape sequence to abort. Sending 3, 100-byte ICMP Echos to 239.0.0.1, timeout is 2 seconds: Reply to request 0 from 20.0.0.10, 1 ms Reply to request 1 from 20.0.0.10, 2 ms Reply to request 2 from 20.0.0.10, 2 ms
The ping works and R2 has the (S,G):
R2#sh ip mroute 239.0.0.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
L - Local, P - Pruned, R - RP-bit set, F - Register flag,
T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
U - URD, I - Received Source Specific Host Report,
Z - Multicast Tunnel, z - MDT-data group sender,
Y - Joined MDT-data group, y - Sending to MDT-data group,
G - Received BGP C-Mroute, g - Sent BGP C-Mroute,
N - Received BGP Shared-Tree Prune, n - BGP C-Mroute suppressed,
Q - Received BGP S-A Route, q - Sent BGP S-A Route,
V - RD & Vector, v - Vector, p - PIM Joins on route
Outgoing interface flags: H - Hardware switched, A - Assert winner, p - PIM Join
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 239.0.0.1), 00:04:18/00:03:29, RP 1.1.1.1, flags: S
Incoming interface: Ethernet0/0, RPF nbr 12.12.12.1
Outgoing interface list:
Ethernet0/2, Forward/Sparse, 00:04:18/00:03:29
(41.41.41.10, 239.0.0.1), 00:01:35/00:01:24, flags: T
Incoming interface: Ethernet0/0, RPF nbr 12.12.12.1
Outgoing interface list:
Ethernet0/2, Forward/Sparse, 00:01:35/00:03:29
What happens when R2 fails?
R2#conf t Enter configuration commands, one per line. End with CNTL/Z. R2(config)#int e0/2 R2(config-if)#sh R2(config-if)#
S1#ping 239.0.0.1 re 200 ti 1 Type escape sequence to abort. Sending 200, 100-byte ICMP Echos to 239.0.0.1, timeout is 1 seconds: Reply to request 0 from 20.0.0.10, 9 ms Reply to request 1 from 20.0.0.10, 2 ms Reply to request 1 from 20.0.0.10, 11 ms.................................................................... ...................................................................... ............................................................
The pings time out because when the PIM Join from R5 comes in, R3 does not realize that it should process the Join.
*Feb 13 13:20:13.236: PIM(0): Received v2 Join/Prune on Ethernet0/2 from 10.0.0.5, not to us *Feb 13 13:20:32.183: PIM(0): Generation ID changed from neighbor 10.0.0.2
As it turns out, the PIM redundancy command must be configured on the secondary router as well for it to process PIM Joins to the VIP.
R3(config-if)#ip pim redundancy HSRP1 hsrp dr-priority 10
After this has configured, the incoming Join will be processed. R3 triggers R5 to send a new Join because the GenID is set in the PIM hello to a new value.
*Feb 13 13:59:19.333: PIM(0): Matched redundancy group VIP 10.0.0.1 on Ethernet0/2 Active, processing the Join/Prune, to us
*Feb 13 13:40:34.043: PIM(0): Generation ID changed from neighbor 10.0.0.1
After configuring this, the PIM DR role converges as fast as HSRP allows. I’m using BFD in this scenario.
The key concept for understanding HSRP aware PIM here is that:
- Initially configuring PIM redundancy on the AR will make it the DR
- PIM redundancy must be configured on the secondary router as well, otherwise it will not process PIM Joins to the VIP
- The PIM DR role does not converge until PIM hellos have timed out, the secondary router will process the Joins though so the multicast will converge
This feature is not very well documented so I hope you have learned a bit from this post how this feature really works. This feature does not work when you have receiver on a HSRP LAN, because the DR role is NOT moved until PIM adjacency expires.
Daniels networking blog 