Multi-metro enterprise WAN

Six metros in three regions joined over a Fabric IP-WAN network, with a Cloud Router and cloud on-ramps in each.

Scenario#

An enterprise has regional hubs in Silicon Valley, Ashburn, London, Amsterdam, Singapore and Sydney. Each hub needs private access to two clouds, and the hubs need a private backbone between them. The MPLS contract that carries this today ends in six months.

Architecture#

One Cloud Router per metro. Every router attaches to one global IP-WAN network, so routes propagate between metros without a mesh of router-to-router connections.

Element	Count
Cloud Routers	6
IP-WAN network	1
Router-to-network connections	6
Router-to-cloud connections	12
Routing protocols	24 on the cloud connections

Backbone choice#

Option	Connections for 6 metros	Fits
IP-WAN network, as here	6	Any-to-any routing with the least to manage; Fabric routes between the attached routers
Full mesh of router-to-router connections	15	Explicit per-pair bandwidth and policy
Hub and spoke	5	Lowest cost; the hub is a single point of failure and adds latency between spokes

Step 1: choose the metros from data#

If the hub locations are not fixed, the optimizer can choose them.

Java

import com.eqixiac.equinix.core.enums.MetroCode;
import com.eqixiac.equinix.design.optimizer.enums.OptimizationStrategy;
import com.eqixiac.equinix.design.optimizer.enums.RedundancyTier;
import com.eqixiac.equinix.design.optimizer.model.OptimizationResult;
import com.eqixiac.equinix.fabric.model.implementation.cloud.CloudProviderType;

OptimizationResult result = fabric.optimizeMetros()
    .name("enterprise-wan")
    .addSite("San Jose").nearestMetro(MetroCode.SV).headcount(900).done()
    .addSite("Reston").nearestMetro(MetroCode.DC).headcount(1200).done()
    .addSite("London").nearestMetro(MetroCode.LD).headcount(700).done()
    .addSite("Singapore").nearestMetro(MetroCode.SG).headcount(400).done()
    .requireProvider(CloudProviderType.AWS).done()
    .requireProvider(CloudProviderType.AZURE).done()
    .preferProvider(CloudProviderType.GOOGLE_CLOUD).done()
    .constraints().redundancy(RedundancyTier.MULTI_REGION).maxLatencyMs(60).maxMetros(6).done()
    .strategy(OptimizationStrategy.LATENCY_FIRST)
    .optimize();

System.out.println(result.getRecommended().portfolio().metros());
System.out.println(result.getLatencyMatrix().toTableString());

The frontier contains the smaller designs next to the recommended one. It shows, for example, how many milliseconds of latency a fifth or sixth metro saves for one more router fee.

Step 2: network and routers#

Java

import com.eqixiac.equinix.fabric.enums.GatewayPackageCode;
import com.eqixiac.equinix.fabric.enums.NetworkScope;
import com.eqixiac.equinix.fabric.enums.NetworkType;
import com.eqixiac.equinix.fabric.enums.NotificationType;
import com.eqixiac.equinix.fabric.model.CloudRouter;
import com.eqixiac.equinix.fabric.model.Network;

Network wan = fabric.networks().define(NetworkType.IPWAN)
    .name("enterprise-ipwan")
    .scope(NetworkScope.GLOBAL)
    .notification(NotificationType.ALL, "noc@example.com")
    .create();

List<String> metros = List.of("SV", "DC", "LD", "AM", "SG", "SY");
Map<String, CloudRouter> routers = new LinkedHashMap<>();
for (String metro : metros) {
    routers.put(metro, fabric.cloudRouters().define()
        .name("fcr-" + metro.toLowerCase())
        .inMetro(metro)
        .withPackage(GatewayPackageCode.STANDARD)
        .accountNumber(123456L)
        .projectId(projectId)
        .notification(NotificationType.ALL, List.of("noc@example.com"))
        .create());
}

Both builders accept dryRun(), so all seven requests can be validated before the first is created.

Step 3: attach each router to the network#

Java

import com.eqixiac.equinix.fabric.enums.ConnectionType;
import com.eqixiac.equinix.fabric.model.Connection;

for (var entry : routers.entrySet()) {
    Connection attachment = fabric.connections().define(ConnectionType.IPWAN_VC)
        .name("ipwan-" + entry.getKey().toLowerCase())
        .bandwidth(10_000)
        .aSideAccessPointCloudRouter(entry.getValue())
        .zSideAccessPointNetwork(wan.getUuid())
        .notification("noc@example.com")
        .create();
}

Fabric determines which connection type it accepts between a Cloud Router and an IP-WAN network. A dry run of the first attachment confirms the type for your account before the six connections are created.

Step 4: cloud on-ramps per metro#

Java

import com.eqixiac.equinix.fabric.enums.PeeringType;
import com.eqixiac.equinix.fabric.model.implementation.cloud.AwsDirectConnectAdapter;
import com.eqixiac.equinix.fabric.model.implementation.cloud.AzureExpressRouteAdapter;
import com.eqixiac.equinix.fabric.model.implementation.cloud.CloudProviderConnectionAdapter;
import com.eqixiac.equinix.fabric.model.implementation.cloud.GoogleCloudInterconnectAdapter;
import com.eqixiac.equinix.fabric.model.implementation.cloud.OracleFastConnectAdapter;

Map<String, List<CloudProviderConnectionAdapter<?>>> onRamps = Map.of(
    "SV", List.of(AwsDirectConnectAdapter.of("123456789012", "us-west-1", awsProfileUuid, "SV"),
                  GoogleCloudInterconnectAdapter.of("pairing-key-sv", "us-west1", gcpProfileUuid, "SV")),
    "DC", List.of(AwsDirectConnectAdapter.of("123456789012", "us-east-1", awsProfileUuid, "DC"),
                  AzureExpressRouteAdapter.of("service-key-dc", azureProfileUuid, PeeringType.PRIVATE, "DC")),
    "LD", List.of(AzureExpressRouteAdapter.of("service-key-ld", azureProfileUuid, PeeringType.PRIVATE, "LD"),
                  GoogleCloudInterconnectAdapter.of("pairing-key-ld", "europe-west2", gcpProfileUuid, "LD")),
    "AM", List.of(AwsDirectConnectAdapter.of("123456789012", "eu-west-1", awsProfileUuid, "AM"),
                  AzureExpressRouteAdapter.of("service-key-am", azureProfileUuid, PeeringType.PRIVATE, "AM")),
    "SG", List.of(AwsDirectConnectAdapter.of("123456789012", "ap-southeast-1", awsProfileUuid, "SG"),
                  GoogleCloudInterconnectAdapter.of("pairing-key-sg", "asia-southeast1", gcpProfileUuid, "SG")),
    "SY", List.of(AzureExpressRouteAdapter.of("service-key-sy", azureProfileUuid, PeeringType.PRIVATE, "SY"),
                  OracleFastConnectAdapter.of("ocid1.virtualcircuit.oc1.example", "ap-sydney-1", oracleProfileUuid, "SY")));

List<Connection> cloudConnections = new ArrayList<>();
for (String metro : metros) {
    for (CloudProviderConnectionAdapter<?> adapter : onRamps.get(metro)) {
        cloudConnections.add(fabric.connections().define(ConnectionType.IP_VC)
            .name("fcr-" + metro.toLowerCase() + "-to-" + adapter.getProviderType().shortCode())
            .bandwidth(5000)
            .aSideAccessPointCloudRouter(routers.get(metro))
            .zSideCloudProvider(adapter)
            .notification("noc@example.com")
            .create());
    }
}

The metro in each adapter is the metro of that router. A provider region can be reached from several metros, which the metro list of the service profile names.

Step 5: routing protocols#

Java

import com.eqixiac.equinix.fabric.enums.RoutingProtocolType;

int subnet = 0;
for (Connection connection : cloudConnections) {
    String base = "10.200." + (subnet++) + ".";
    fabric.routingProtocols().define()
        .ofType(RoutingProtocolType.DIRECT).withDirectIpv4(base + "1/30").create(connection);
    fabric.routingProtocols().define()
        .ofType(RoutingProtocolType.BGP)
        .withCustomerAsn(64512L)                    // the provider-side ASN of this session
        .withBGPIpv4(base + "2", base + "1", true)
        .withBFD(true, 300)
        .create(connection);
}

Take each provider-side ASN and address from the provider object, as described in Hybrid multicloud.

Failure handling during the build#

Eighteen connections and six routers are 24 billable creates. If the seventeenth create fails, sixteen resources exist and are billed. There are three ways to handle such a failure.

Keep every returned object in a list and delete the objects in reverse order when a create throws. Connection deletes are asynchronous, so the code has to wait for each connection delete before it deletes the router.
Use the Deployment Wizard: execute with ABORT_AND_ROLLBACK does the reverse-order cleanup and reports orphans. The wizard plans router-to-router backbone links and no IP-WAN network, so its backbone differs from the one on this page.
Use reconcile with a checkpoint store, which resumes after a crash and creates the resources that are still missing.

Operations#

Attach every connection and router to a stream and alert on BGP state and bandwidth.
Run plan.diff() or terraform plan on a schedule to catch changes made in the portal.
Run estate analysis every quarter to find connections that were sized for a migration and carry less traffic afterwards.

Trade-offs#

A STANDARD router in six metros is six router fees. A metro with one small cloud connection can be served from a neighbouring metro at higher latency, and the optimizer's frontier contains that smaller design.
Every router on the IP-WAN network can reach every other. Separating business units requires separate networks or route filters.
With one router per metro, the loss of the router removes all on-ramps of that metro. A second router in a metro removes that dependency at the cost of a second router fee.