UK CAN WAN Circuit Failure Senarios
Author: Ann Jezak
Scenario 1: Welwyn and Wapping COGENT is down + ExpressRoute in Welwyn down + ER primary in Wapping down, leaving ONLY Wapping secondary ER circuit up. This means that:
UK on-premise routes are being advertised out to all remote countries
UK is seeing all country Azure routes
UK is seeing AUS on-premise routes (VOCUS ER in AUS is different)
ER Global Reach (secondary only) for the UK is accepting CA and US on-premise routes
ER Global Reach for CA <-> UK (secondary) is not accepting UK on-premise routes
ER Global Reach for US <-> UK (secondary) is not accepting UK on-premise routes, although CA is advertising RFC1918 so the US would send RFC1918 to CA
UK on-prem <-> CA on-prem and UK on-prem <-> US on-prem has stopped working
Under these conditions, we would need to manually make the following changes to allow UK on-prem <-> CA on-prem and UK on-prem <-> US on-prem traffic to flow:
Modify CYXTERA CL-ASR-1001X-1 (10.28.200.8) BGP to permit UK on-premise subnets inbound:
route-map FROM-AZURE-RMAP permit 200
match ip address prefix-list UK-ONPREM-PL
exit
route-map BGP-TO-OSPF permit 200
match ip address prefix-list match ip address prefix-list UK-ONPREM-PL
set metric 10
set metric-type type-1
Modify US1-WAN-4451-2 (10.40.222.4) BGP to permit UK on-premise subnets inbound
route-map FROM-AZURE-RMAP permit 200
match ip address prefix-list UK-ONPREM-PL
exit
route-map BGP-TO-OSPF-RMAP permit 200
match ip address prefix-list UK-ONPREM-PL
set metric 100
set metric-type type-1
Scenario 2: Welwyn and Wapping COGENT is down + ExpressRoute in Wapping down + ER primary in Welwyn down, leaving ONLY Welwyn secondary ER circuit up. This means that:
UK on-premise routes are being advertised out to all remote countries
UK is seeing all country Azure routes
UK is seeing AUS on-premise routes (VOCUS ER in AUS is different)
WEL-4451-WAN2 is COGENT and ER primary. WEL-4451-WAN1 is COLT and ER secondary.
ER Global Reach (secondary only) for the UK is not accepting CA and US on-premise routes (WEL-4451-WAN1 using FROM-AZURE-RMAP)
ER Global Reach for CA <-> UK (secondary) is not accepting UK on-premise routes (CL-ASR-1001X-1 Te0/1/0.101 + MCC-8300-WAN1 Te0/0/4.700)
ER Global Reach for US <-> UK (secondary) is not accepting UK on-premise routes (US1-WAN-4451-2 Gi0/0/3 + MARKLEY_4451_WAN1 Gi0/0/1)
UK on-prem <-> CA on-prem and UK on-prem <-> US on-prem has stopped working
Under these conditions, we would need to manually make the following changes to allow UK on-prem <-> CA on-prem and UK on-prem <-> US on-prem traffic to flow:
Modify CYXTERA CL-ASR-1001X-1 (10.28.200.8) BGP to permit UK on-premise subnets inbound:
route-map FROM-AZURE-RMAP permit 200
match ip address prefix-list UK-ONPREM-PL
exit
route-map BGP-TO-OSPF permit 200
match ip address prefix-list match ip address prefix-list UK-ONPREM-PL
set metric 10
set metric-type type-1
Modify US1-WAN-4451-2 (10.40.222.4) BGP to permit UK on-premise subnets inbound
route-map FROM-AZURE-RMAP permit 200
match ip address prefix-list UK-ONPREM-PL
exit
route-map BGP-TO-OSPF-RMAP permit 200
match ip address prefix-list UK-ONPREM-PL
set metric 100
set metric-type type-1
Modify WEL-4451-WAN1 (10.5.200.3) to permit CA and US on-premise subnets inbound
route-map FROM-AZURE-RMAP permit 200
match ip address prefix-list CA-ONPREM-PL CA-ONPREM-PUBLIC-PL US-ONPREM-PL US-ONPREM-MARKLEY-PL US-ONPREM-PUBLIC-PL
Scenario 3: NY2 and Markley COGENT is down + ExpressRoute in Markley down + ER primary in NY2 is down, leaving ONLY NY2 secondary ER circuit up. This means that:
US on-premise routes are being advertised out to all remote countries
US is seeing all country Azure routes
US is seeing AUS on-premise routes (VOCUS ER in AUS is different)
US1-WAN-4451-1 is COGENT and ER primary. US1-WAN-4451-2 is CenturyLink and ER secondary.
ER Global Reach (secondary only) for the US is not accepting CA and UK on-premise routes (US1-WAN-4451-2 Gi0/0/3 using FROM-AZURE-RMAP) and not redistributing from BGP into OSPF even if it received the on-premise routes
ER Global Reach for CA <-> US (secondary) is accepting US on-premise routes in MCC only (CL-ASR-1001X-1 Te0/1/0.101 and MCC-8300-WAN1 Te0/0/4.700) and MARKLEY_4451_WAN1 is also redistributing CA and UK on-prem routes into OSPF from BGP
ER Global Reach for US <-> UK (secondary) is accepting US on-premise routes in Wapping only (WEL-4451-WAN1 Gi0/3/0.100 + LON-WAP-4451-1 Gi0/0/2.100 ) and LON-WAP-4451-1 is also redistributing the US and CA on-premise routes into OSPF
UK on-prem <-> CA on-prem and UK on-prem <-> US on-prem is still working
Scenario 4: NY2 and Markley COGENT is down + ExpressRoute in NY2 down + ER primary in Markley is down, leaving ONLY Markley secondary ER circuit up. This means that:
Markley uses a single WAN router with COGENT and both Azure ER links, so the Markley router (MARKLEY_4451_WAN1 @ 10.12.201.3) advertises and receives all routes over both ER circuits and redistributing CA and UK on-prem routes into OSPF from BGP, so no issue at this location with ER Global Reach primary and secondary
Scenario 5: CYXTERA and MCC COGENT is down + ExpressRoute in CYXTERA down + ER primary in MCC is down, leaving ONLY MCC secondary ER circuit up. This means that:
MCC uses a single WAN router with COGENT and both Azure ER links, so the MCC router (MCC-8300-WAN1 @ 172.31.10.124 ) advertises and receives all routes over both ER circuits, so no issue at this location with ER Global Reach primary and secondary. However, in MCC, none of the remote on-premise subnets are redistributed into OSPF, so modifications required would be:
On MCC-8300-WAN1:
route-map BGP-TO-OSPF-RMAP permit 200
match ip address prefix-list US-ONPREM-PL US-ONPREM-MARKLEY-PL US-ONPREM-PUBLIC-PL UK-ONPREM-PL
set metric 100
set metric-type type-2
Scenario 6: CYXTERA and MCC COGENT is down + ExpressRoute in MCC down + ER primary in CYXTERA is down, leaving ONLY CYXTERA secondary ER circuit up. This means that:
CA on-premise routes are being advertised out to all remote countries
CA is seeing all country Azure routes
CA is seeing AUS on-premise routes (VOCUS ER in AUS is different)
CL-ASR-1001X-WAN2 is COGENT and ER primary. CL-ASR-1001X-1 is Telus and ER secondary.
ER Global Reach (secondary only) for the UK is not accepting CA on-premise routes at Welwyn (WEL-4451-WAN1 using FROM-AZURE-RMAP) but is accepting CA and US on-premise routes at Wapping (LON-WAP-4451-1 Gi0/0/2.100). LON-WAP-4451-1 is already redistributing CA and US on-premise routes from BGP into OSPF
ER Global Reach for CA <-> UK (secondary) is not accepting CA on-premise routes at CYXTERA (CL-ASR-1001X-1 Te0/1/0.101) but is accepting them at MCC (MCC-8300-WAN1 Te0/0/4.700). However MCC-8300-WAN1 is not redistributing the US and UK on-premise routes into OSPF
ER Global Reach for US <-> UK (secondary) is not accepting CA on-premise routes in NY2 (US1-WAN-4451-2 Gi0/0/3) but is accepting UK and CA on-premise routes in Markley (MARKLEY_4451_WAN1 Gi0/0/1) and MARKLEY_4451_WAN1 is also redistributing those UK and CA on-premise routes into OSPF
UK on-prem <-> CA on-prem and UK on-prem <-> US on-prem is still working
Under these conditions, we would need to manually make the following changes to allow UK on-prem <-> CA on-prem and UK on-prem <-> US on-prem traffic to flow:
Modify MCC-8300-WAN1 (172.31.10.124) to redistribute US and UK on-premise into OSPF:
route-map BGP-TO-OSPF-RMAP permit 200
match ip address prefix-list US-ONPREM-PL US-ONPREM-MARKLEY-PL US-ONPREM-PUBLIC-PL UK-ONPREM-PL
set metric 100
set metric-type type-2
Modify US1-WAN-4451-2 (10.40.222.4) BGP to permit UK on-premise subnets inbound
route-map FROM-AZURE-RMAP permit 200
match ip address prefix-list UK-ONPREM-PL CA-ONPREM-PL CA-ONPREM-PUBLIC-PL