Multi-cloud networking CE-HA demo

This deployment demonstrates active/active Customer Edge high availability in Azure, AWS, and On-Premise KVM using interface binding, registration, BGP/ECMP, TGW Connect, HA, upgrades, and lifecycle automation in a single Terraform plan:

1. Rest of World (ROW): Serves mcn-ce-ha.f5-sales-demo.com via CEs in eastus with global public RE advertisement.
2. AWS Customer Edge Extension: Serves aws.mcn-ce-ha.f5-sales-demo.com via 3 EC2 CEs (m5.2xlarge) in AWS VPC (10.150.0.0/16) in ap-northeast-1 with dual NICs and eBGP/ECMP active/active routing.
3. On-Premise KVM Extension: One CE (onprem-ce-01, 10.100.0.11) peers with Terraform-owned FRR (10.100.0.2, ASN 65515) on the local libvirt network (10.100.0.0/24); the NAT gateway is 10.100.0.1 and the workload is 10.100.0.100. One BGP peer imports 198.51.100.0/24; this is not a KVM ECMP demonstration.

Azure US uses two FRR relays between the CEs and Route Server, plus Standard Internal Load Balancers with separate application and Site Console frontends. The AWS path uses Transit Gateway Connect with six GRE peers and twelve BGP sessions; an internal NLB fronts three routed site listeners. See the SMSv2 guide for the accepted AWS topology.

Evidence is platform- and date-scoped. AWS v8 acceptance verified one-CE failover and three serial upgrades. Azure observations through 2026-08-08 remain historical; final provider-v8 acceptance did not mutate or re-prove Azure end-to-end.

What it deploys

Observed 2026-08-05: example addresses and names below came from the earlier deployment; use current Terraform outputs for a live presentation. The intended three-platform hybrid architecture is:

  Azure US:      3 CEs ── 2 FRRs ── Route Server ── BGP VIP
                    └──── internal ILB (.10 app, .11 console)
  AWS:           3 CEs ── 6 Connect peers / 12 BGP sessions ── NLB
  KVM LAN:       1 CE (NAT SLO + physical-LAN SLI) ── custom inside VIP

Piece	What it is
F5 Distributed Cloud tenant	var.expected_xc_tenant — the only place the tenant is named
Namespaces	CE sites in system; origin pool and load balancer in var.xc_app_namespace
Rest of World CEs	var.ce_count single-node Secure Mesh v2 sites in var.location (eastus)
AWS CEs	Three independent, single-node SecureMesh v2 sites in var.aws_location (ap-northeast-1), each with MAC-bound SLO and SLI interfaces
AWS workload	Amazon Linux SSM client with no ingress rules in a dedicated TGW-attached VPC (10.151.0.0/16)
On-Premise KVM CE	One Customer Edge (onprem-ce-01) with NAT SLO and an SLI on the pre-existing physical LAN bridge
AWS Virtual Site	mcn-ce-ha-aws-vsite, selecting sites mcn-ce-ha-aws-ap-northeast-1-01 through 03
Route exchange	Two FRR relays and a Route Server in each Azure region; AWS Transit Gateway Connect; the KVM FRR router on the NAT SLO network
Peerings	eBGP peerings per CE, CE ASN var.ce_asn (64512)
Advertised VIPs	Separate regional BGP and inside ILB application VIPs; internal AWS NLB fronts three BGP-routed site listeners; KVM advertises its custom inside VIP on the physical LAN
Load balancers	var.lb_domain (f5-sales-demo.com) for ROW; var.aws_lb_domain (aws.mcn-ce-ha.f5-sales-demo.com) for AWS
Test clients	Client VMs in each regional VNet, AWS VPC, and local test environment for driving traffic at the VIPs

Nothing above is a literal you should copy. Every value is readable from the deployment, and each page reads it rather than naming it:

cd terraform
terraform output -raw resource_group_name
terraform output -json xc_site_names

The VIP must sit outside every VNet CIDR

terraform/main.tf carries a check block that asserts the VIP is outside both the hub and spoke CIDRs, and refuses the plan otherwise. Its stated reason is that a VIP inside a VNet CIDR loses to the VNet’s own system route despite the BGP route being more specific.

Treat the guard as the contract: if you re-point this demo at different addressing, the VIP has to move out of the VNet ranges with it.

Renaming is a rebuild, not a rename

Because every name derives from var.component, changing it renames the whole deployment — but site_prefix is not a cosmetic change. It becomes the ClusterName in each CE’s cloud-init, custom_data is replace-only, and cloud-init runs only on a CE’s first boot. So changing it replaces every CE VM and its XC site, and the VIP is then briefly lossy while ECMP reconverges.

That is not theoretical: adopting these derived names on the running demo replaced three CE VMs, three sites, three BGP objects, the load balancer, the origin pool, the Azure Route Server, the Bastion and the test client — 23 to add, 3 to change, 26 to destroy. Pin site_prefix to the current value when you need existing sites to stay put.

Where to go next

SMSv2 deployment guideVerified lifecycle, AWS Transit Gateway Connect, Azure Route Server, and troubleshooting.

Deploy itPrerequisites, the two values you must supply, why the first apply is two-phase, and how to tear it down.

HA Options ComparisonIn-depth inflection point guide comparing dynamic BGP/ECMP routing against Azure Internal Load Balancer (ILB) ingress.

The TerraformThe configuration itself, read from the files that deploy it rather than retyped.

Prove it healthyFour checks in order, and the normal-looking results that read as an outage and are not.

Present itWhat this demo can show, what it cannot yet show, and the one command that makes ECMP visible.

The Canadian regional deployment is maintained independently in Canada Topology.