Disaster recovery across regions

A primary and a recovery metro with redundant cloud on-ramps, routing that fails over by itself, and a test procedure.

Scenario#

Production runs in Ashburn with AWS us-east-1, and the recovery site is Dallas with AWS us-east-2. The recovery objectives are a recovery time of 15 minutes for the network and replication that keeps the data loss below 5 minutes. The network must fail over without a manual change, and the recovery path must be tested every quarter.

Failure domains#

The components of the design follow from the failures it has to survive. Each row in the following table protects against one more failure and adds the cost in its last column.

Failure	Mitigation	Adds
One virtual connection or one provider device	Primary and secondary connection in a redundancy group, to two provider locations	One connection
One Fabric port or Fabric device	Two ports on different devices	One port, one cross connect
One Cloud Router	Two routers in the metro	One router fee
One IBX	Ports in two IBXs of the metro	A second cage or a metro connect
One metro or cloud region	A second metro with its own on-ramps, as below	The whole second site

Architecture#

Each metro has its own Cloud Router and its own on-ramp to the nearest AWS region. A backbone connection joins the routers. BGP path attributes make Ashburn primary.

Replication between the two AWS regions stays on the AWS network. The Fabric backbone carries site-to-site traffic and is the recovery path when one metro's on-ramp is lost.

Step 1: test the placement against a metro loss#

The optimizer's resilience objective measures the share of the design that survives the loss of the metro whose loss removes the most.

Java

import com.eqixiac.equinix.core.enums.MetroCode;
import com.eqixiac.equinix.design.optimizer.enums.OptimizationStrategy;
import com.eqixiac.equinix.design.optimizer.enums.RedundancyTier;
import com.eqixiac.equinix.design.optimizer.enums.WorkloadType;
import com.eqixiac.equinix.design.optimizer.model.OptimizationResult;
import com.eqixiac.equinix.design.spi.Objective;
import com.eqixiac.equinix.fabric.model.implementation.cloud.CloudProviderType;

OptimizationResult result = fabric.optimizeMetros()
    .name("dr-pair")
    .addSite("Reston HQ").nearestMetro(MetroCode.DC).headcount(800).done()
    .addSite("Dallas office").nearestMetro(MetroCode.DA).headcount(200).done()
    .requireProvider(CloudProviderType.AWS).done()
    .addWorkload("erp").type(WorkloadType.TRANSACTIONAL).bandwidthMbps(1_000).dependsOn(CloudProviderType.AWS).done()
    .addWorkload("erp-dr").type(WorkloadType.DISASTER_RECOVERY).bandwidthMbps(500).dependsOn(CloudProviderType.AWS).done()
    .constraints().redundancy(RedundancyTier.MULTI_METRO).requireMetro("DC").maxMetros(2).done()
    .strategy(OptimizationStrategy.REDUNDANCY_FIRST)
    .optimize();

System.out.println(result.getRecommended().portfolio().metros());
System.out.println(result.getRecommended().values().get(Objective.RESILIENCE).explanation());

The explanation states the surviving share of sites, clouds and workloads after the worst metro loss, and the same for the worst region loss. RedundancyTier.MULTI_REGION forces two Equinix regions, for example AMER and EMEA. Two metros in one country are MULTI_METRO.

For services that depend on peering, the blast radius report gives the corresponding figure for exchange presence.

Step 2: redundant on-ramps in each metro#

A redundancy group holds a primary and a secondary connection. For AWS, the two hosted connections should end on different Direct Connect locations or devices, which AWS assigns from the service profile and the metro.

Java

import com.eqixiac.equinix.fabric.enums.ConnectionType;
import com.eqixiac.equinix.fabric.enums.RedundancyPriority;
import com.eqixiac.equinix.fabric.model.CloudRouter;
import com.eqixiac.equinix.fabric.model.Connection;
import com.eqixiac.equinix.fabric.model.implementation.cloud.AwsDirectConnectAdapter;

CloudRouter fcrDc = fabric.cloudRouters().getByUuid(routerUuid);

List<Connection> onRamps = new ArrayList<>();
for (RedundancyPriority priority : RedundancyPriority.values()) {
    onRamps.add(fabric.connections().define(ConnectionType.IP_VC)
        .name("fcr-dc-to-aws-" + priority.name().toLowerCase())
        .bandwidth(1000)
        .redundancy("fcr-dc-aws", priority)
        .aSideAccessPointCloudRouter(fcrDc)
        .zSideCloudProvider(AwsDirectConnectAdapter.of("123456789012", "us-east-1", awsProfileUuid, "DC"))
        .notification("noc@example.com")
        .create());
}

Repeat for Dallas with us-east-2 and metro "DA". Each of the four connections gets its own /30, a DIRECT and a BGP protocol.

Step 3: backbone between the routers#

Java

CloudRouter fcrDa = fabric.cloudRouters().getByUuid("uuid-of-fcr-da");

Connection backbone = fabric.connections().define(ConnectionType.IP_VC)
    .name("fcr-dc-to-fcr-da")
    .bandwidth(1000)
    .aSideAccessPointCloudRouter(fcrDc)
    .zSideAccessPointCloudRouter(fcrDa)
    .notification("noc@example.com")
    .dryRun()                                  // remove to create
    .create();

The backbone has to carry the traffic that crosses it during a failure, which exceeds the traffic of normal operation. If Ashburn loses its on-ramp, all traffic between Ashburn and AWS crosses the backbone to Dallas.

Step 4: make failover a routing outcome#

The design keeps the recovery path advertised at all times with less preferred BGP attributes, so that failover needs no manual route change.

Java

import com.eqixiac.equinix.fabric.enums.RoutingProtocolType;

// Primary on-ramp: default attributes, BFD for fast detection
fabric.routingProtocols().define()
    .ofType(RoutingProtocolType.BGP)
    .withCustomerAsn(64512L)
    .withBGPIpv4("10.100.1.2", "10.100.1.1", true)
    .withBFD(true, 300)
    .create(onRamps.get(0));

// Secondary on-ramp: AS-path prepend outbound and a higher MED, so AWS and the router prefer the primary
fabric.routingProtocols().define()
    .ofType(RoutingProtocolType.BGP)
    .withCustomerAsn(64512L)
    .withBGPIpv4("10.100.2.2", "10.100.2.1", true)
    .withBGPIpv4Options(3L, 200L, 200L, null)     // outbound prepend count, inbound MED, outbound MED, max routes
    .withBFD(true, 300)
    .create(onRamps.get(1));

With BFD at a 300 ms interval and the usual multiplier of 3, a failed session is detected in about a second, and traffic moves to the next-best path. Without BFD the session is declared down when the BGP hold timer expires, which typically takes 90 seconds or more.

The preference has to be consistent in both directions. AWS chooses its outbound path from the AS-path length and the local preference communities it receives. If the design prefers the primary for inbound traffic and the secondary for outbound traffic, a stateful device on either path sees one direction of a flow and drops it.

Step 5: verify before the test#

Java

import com.eqixiac.equinix.fabric.model.RouteTableEntry;

Map<String, List<String>> nextHopsByPrefix = new TreeMap<>();
for (RouteTableEntry route : fabric.cloudRouters().searchRoutes(routerUuid).loadAll()) {
    nextHopsByPrefix.computeIfAbsent(route.getPrefix(), k -> new ArrayList<>())
        .add(route.getNextHop() + " (" + route.getState() + ")");
}
nextHopsByPrefix.forEach((prefix, hops) -> {
    if (hops.size() < 2) {
        System.out.println("NO BACKUP PATH: " + prefix + " via " + hops);
    }
});

A prefix with one next hop has no recovery path, so this check should run on a schedule and after every change.

Step 6: quarterly failover test#

A test removes the primary path under controlled conditions and shows whether the recovery path carries the traffic.

Announce the test window, and confirm with the check from step 5 that every production prefix has two next hops.
Record the route table and the received routes on each connection as the baseline.
Reset the BGP session on the primary on-ramp, which interrupts traffic on that connection.
Measure the time until traffic flows over the secondary, from your application probes.
Read the route table again and confirm the secondary path is active.
Wait for the primary session to re-establish, and confirm traffic returns.

Java

import com.eqixiac.equinix.fabric.enums.BGPActionType;

String primaryConnection = onRamps.get(0).getUuid();
String bgpUuid = fabric.routingProtocols().list(primaryConnection).stream()
    .filter(p -> p.getType() == RoutingProtocolType.BGP)
    .findFirst().orElseThrow().getUuid();

fabric.routingProtocols().createBgpAction(primaryConnection, bgpUuid, BGPActionType.RESET_BGPIPV4);

A session reset tests the routing failover of one connection. A test of a metro loss also has to shut the cage-side sessions in Ashburn and fail the applications over, which the network API cannot do.

Record the measured failover time against the 15-minute objective each quarter. A failover time that grows from one test to the next indicates that the configuration has changed, for example by a prefix that was added on the primary path only.

Cost#

The recovery site adds a second router, two more on-ramp connections with their provider ports, and the backbone, which roughly doubles the Fabric cost. Run the TCO comparison with and without the second metro to obtain the monthly cost of the recovery site, and compare it with the cost of the outage it prevents.