Multi-cloud networking CE-HA demo

This deployment demonstrates active/active Customer Edge high availability in Azure, AWS, and On-Premise KVM using platform-specific routing and ingress, expanded to support parallel regional network paths, an AWS extension, and an on-premise KVM extension in a single Terraform plan:

1. Rest of World (ROW): Serves mcn-ce-ha.f5-sales-demo.com via CEs in eastus with global public RE advertisement.
2. Canada Regional Path: Serves mcn-ce-ha.f5-sales-demo.ca strictly within Canada via CEs in canadacentral, a Canadian Regional Edge Virtual Site targeting Toronto and Montreal PoPs (ves.io/city in (toronto, montreal)), and an Azure Internal Load Balancer (enable_canada_ilb).
3. AWS Customer Edge Extension: Serves aws.mcn-ce-ha.f5-sales-demo.com via 3 EC2 CEs (m5.2xlarge) in AWS VPC (10.150.0.0/16) in ap-northeast-1 with dual NICs and eBGP/ECMP active/active routing.
4. On-Premise KVM Extension: One CE (onprem-ce-01, 10.100.0.11) peers with Terraform-owned FRR (10.100.0.2, ASN 65515) on the local libvirt network (10.100.0.0/24); the NAT gateway is 10.100.0.1 and the workload is 10.100.0.100. One BGP peer imports 198.51.100.0/24; this is not a KVM ECMP demonstration.

Azure US and Canada use Standard Internal Load Balancers to verify supported CE health and Site Console traffic. Azure Route Server remains deferred because F5 has not published the writable eBGP multihop request control it requires. The AWS path uses Transit Gateway Connect with six GRE peers and twelve BGP sessions; an internal NLB fronts three routed site listeners. See the SMSv2 guide for the accepted AWS topology.

Evidence is platform- and date-scoped. AWS v8 acceptance verified one-CE failover and three serial upgrades. Azure observations through 2026-08-08 remain historical; final provider-v8 acceptance did not mutate or re-prove Azure end-to-end.

What it deploys

Observed 2026-08-05. The three-path hybrid architecture:

                     Rest of World (ROW)              Canada Regional Path               On-Premise KVM Extension
                 mcn-ce-ha.f5-sales-demo.com       mcn-ce-ha.f5-sales-demo.ca           onprem.mcn-ce-ha.example.com
                           |                                 |                                        |
                 Global Public REs               Canada RE Virtual Site                       On-Prem KVM Virtual Site
                         |                     (Toronto & Montreal REs)                    (KVM Customer Edge Sites)
                         |                                   |                                        |
                Azure Route Server                Canada Azure Route Server                 FRR ToR BGP Router (Container)
                   ASN 65515                         ASN 65515                                ASN 65515 (10.100.0.1)
                       |                                     |                                        |
           +-----------+-----------+             +-----------+-----------+                    +-----------+-----------+
           |           |           |             |           |           |                    |           |           |
        CE-01       CE-02       CE-03         CE-CA-01    CE-CA-02    CE-CA-03             CE-KV-01    CE-KV-02    CE-KV-03
       (eastus)    (eastus)    (eastus)     (canadacentral) (canadacentral) (canadacentral)   (onprem-01) (onprem-02) (onprem-03)
           |           |           |             |           |           |                    |           |           |
           +---- VIP <ROW_VIP> ----+             +---- VIP <CA_VIP> -------+                    +---- VIP <KVM_VIP> --------+

Piece	What it is
F5 Distributed Cloud tenant	var.expected_xc_tenant — the only place the tenant is named
Namespaces	CE sites in system; origin pool and load balancer in var.xc_app_namespace
Rest of World CEs	var.ce_count single-node Secure Mesh v2 sites in var.location (eastus)
Canadian CEs	var.ca_ce_count single-node Secure Mesh v2 sites in var.ca_location (canadacentral)
AWS CEs	Three independent, single-node SecureMesh v2 sites in var.aws_location (ap-northeast-1), each with MAC-bound SLO and SLI interfaces
AWS workload	Amazon Linux SSM client with no ingress rules in a dedicated TGW-attached VPC (10.151.0.0/16)
On-Premise KVM CEs	3 KVM Customer Edge nodes (onprem-ce-01, 02, 03) on local libvirt network ce-bgp-net (10.100.0.0/24)
Canadian Virtual Sites	xcsh_virtual_site.canada_re (Toronto & Montreal REs) and xcsh_virtual_site.canada_ce (Canadian CEs)
AWS Virtual Site	mcn-ce-ha-aws-vsite, selecting sites mcn-ce-ha-aws-ap-northeast-1-01 through 03
Route exchange	Azure Route Servers in eastus and canadacentral (ASN var.rs_asn), AWS Transit Gateway Connect (ASN var.aws_tgw_asn), FRR ToR BGP Router for KVM (ASN 65515, 10.100.0.2)
Peerings	eBGP peerings per CE, CE ASN var.ce_asn (64512)
Advertised VIPs	var.vip (10.250.0.10) for ROW; var.ca_vip (10.250.1.10) for Canada; internal AWS NLB var.aws_vip (10.151.1.10) fronts three BGP-routed SMSv2 site listeners. KVM does not advertise a VIP in this topology.
Load balancers	var.lb_domain (f5-sales-demo.com) for ROW; var.ca_lb_domain (f5-sales-demo.ca) for Canada; var.aws_lb_domain (aws.mcn-ce-ha.f5-sales-demo.com) for AWS
Test clients	Client VMs in each regional VNet, AWS VPC, and local test environment for driving traffic at the VIPs

Nothing above is a literal you should copy. Every value is readable from the deployment, and each page reads it rather than naming it:

cd terraform
terraform output -raw resource_group_name
terraform output -json xc_site_names

The VIP must sit outside every VNet CIDR

terraform/main.tf carries a check block that asserts the VIP is outside both the hub and spoke CIDRs, and refuses the plan otherwise. Its stated reason is that a VIP inside a VNet CIDR loses to the VNet’s own system route despite the BGP route being more specific.

Treat the guard as the contract: if you re-point this demo at different addressing, the VIP has to move out of the VNet ranges with it.

Renaming is a rebuild, not a rename

Because every name derives from var.component, changing it renames the whole deployment — but site_prefix is not a cosmetic change. It becomes the ClusterName in each CE’s cloud-init, custom_data is replace-only, and cloud-init runs only on a CE’s first boot. So changing it replaces every CE VM and its XC site, and the VIP is then briefly lossy while ECMP reconverges.

That is not theoretical: adopting these derived names on the running demo replaced three CE VMs, three sites, three BGP objects, the load balancer, the origin pool, the Azure Route Server, the Bastion and the test client — 23 to add, 3 to change, 26 to destroy. Pin site_prefix to the current value when you need existing sites to stay put.

Where to go next

SMSv2 deployment guideVerified lifecycle, AWS Transit Gateway Connect, Azure Route Server, and troubleshooting.

Deploy itPrerequisites, the two values you must supply, why the first apply is two-phase, and how to tear it down.

HA Options ComparisonIn-depth inflection point guide comparing dynamic BGP/ECMP routing against Azure Internal Load Balancer (ILB) ingress.

The TerraformThe configuration itself, read from the files that deploy it rather than retyped.

Prove it healthyFour checks in order, and the normal-looking results that read as an outage and are not.

Present itWhat this demo can show, what it cannot yet show, and the one command that makes ECMP visible.