Multi-cloud networking CE-HA demo

This deployment demonstrates active/active Customer Edge high availability in Azure, AWS, and On-Premise KVM with no cloud load balancer in front of the CEs, expanded to support parallel regional network paths, an AWS extension, and an on-premise KVM extension in a single Terraform plan:

1. Rest of World (ROW): Serves mcn-ce-ha.f5-sales-demo.com via CEs in eastus with global public RE advertisement.
2. Canada Regional Path: Serves mcn-ce-ha.f5-sales-demo.ca strictly within Canada via CEs in canadacentral and a Canadian Regional Edge Virtual Site targeting Toronto and Montreal PoPs (ves.io/city in (toronto, montreal)), supporting both Azure Route Server (eBGP) and Azure Internal Load Balancer (ILB) architectures (enable_canada_ilb).
3. AWS Customer Edge Extension: Serves aws.mcn-ce-ha.f5-sales-demo.com via 3 EC2 CEs (m5.2xlarge) in AWS VPC (10.150.0.0/16) in us-east-2 with dual NICs and eBGP/ECMP active/active routing.
4. On-Premise KVM Extension: Serves on-premise workloads via 3 KVM CEs (onprem-ce-01, 02, 03) on local libvirt network ce-bgp-net (10.100.0.0/24) peering eBGP (CE ASN 64512) with a containerized FRR ToR BGP Router (10.100.0.1, ASN 65515) configured with maximum-paths 4 ECMP.

Each set of CEs originates its host route by eBGP to a regional Azure Route Server, which installs all of them as equal-cost next hops in the VNet. The routing fabric, rather than a load balancer, is where traffic distribution and failed-node removal happen.

Everything in these pages was run against the live deployment, and the output shown is the output it produced. What that sentence does not cover — failover, and how traffic divides across the next hops — is called out explicitly rather than left to be assumed.

What it deploys

Observed 2026-08-05. The three-path hybrid architecture:

                     Rest of World (ROW)              Canada Regional Path               On-Premise KVM Extension
                 mcn-ce-ha.f5-sales-demo.com       mcn-ce-ha.f5-sales-demo.ca           onprem.mcn-ce-ha.example.com
                           |                                 |                                        |
                 Global Public REs               Canada RE Virtual Site                       On-Prem KVM Virtual Site
                         |                     (Toronto & Montreal REs)                    (KVM Customer Edge Sites)
                         |                                   |                                        |
                Azure Route Server                Canada Azure Route Server                 FRR ToR BGP Router (Container)
                   ASN 65515                         ASN 65515                                ASN 65515 (10.100.0.1)
                       |                                     |                                        |
           +-----------+-----------+             +-----------+-----------+                    +-----------+-----------+
           |           |           |             |           |           |                    |           |           |
        CE-01       CE-02       CE-03         CE-CA-01    CE-CA-02    CE-CA-03             CE-KV-01    CE-KV-02    CE-KV-03
       (eastus)    (eastus)    (eastus)     (canadacentral) (canadacentral) (canadacentral)   (onprem-01) (onprem-02) (onprem-03)
           |           |           |             |           |           |                    |           |           |
           +---- VIP <ROW_VIP> ----+             +---- VIP <CA_VIP> -------+                    +---- VIP <KVM_VIP> --------+

Piece	What it is
F5 Distributed Cloud tenant	var.expected_xc_tenant — the only place the tenant is named
Namespaces	CE sites in system; origin pool and load balancer in var.xc_app_namespace
Rest of World CEs	var.ce_count single-node Secure Mesh v2 sites in var.location (eastus)
Canadian CEs	var.ca_ce_count single-node Secure Mesh v2 sites in var.ca_location (canadacentral)
AWS CEs	var.aws_ce_count EC2 Customer Edge nodes in var.aws_location (us-east-2) VPC (10.150.0.0/16)
On-Premise KVM CEs	3 KVM Customer Edge nodes (onprem-ce-01, 02, 03) on local libvirt network ce-bgp-net (10.100.0.0/24)
Canadian Virtual Sites	xcsh_virtual_site.canada_re (Toronto & Montreal REs) and xcsh_virtual_site.canada_ce (Canadian CEs)
AWS Virtual Site	xcsh_virtual_site.aws (AWS Customer Edge Site aws-site)
Route exchange	Azure Route Servers in eastus and canadacentral (ASN var.rs_asn), AWS VPC Router (ASN 65515), FRR ToR BGP Router for KVM (ASN 65515, 10.100.0.1)
Peerings	eBGP peerings per CE, CE ASN var.ce_asn (64512)
Advertised VIPs	var.vip (10.250.0.10) for ROW; var.ca_vip (10.250.1.10) for Canada; var.aws_vip (10.150.0.10) for AWS; 10.100.0.10 for On-Prem KVM
Load balancers	var.lb_domain (f5-sales-demo.com) for ROW; var.ca_lb_domain (f5-sales-demo.ca) for Canada; var.aws_lb_domain (aws.mcn-ce-ha.f5-sales-demo.com) for AWS
Test clients	Client VMs in each regional VNet, AWS VPC, and local test environment for driving traffic at the VIPs

Nothing above is a literal you should copy. Every value is readable from the deployment, and each page reads it rather than naming it:

cd terraform
terraform output -raw resource_group_name
terraform output -json xc_site_names

The VIP must sit outside every VNet CIDR

terraform/main.tf carries a check block that asserts the VIP is outside both the hub and spoke CIDRs, and refuses the plan otherwise. Its stated reason is that a VIP inside a VNet CIDR loses to the VNet’s own system route despite the BGP route being more specific.

Treat the guard as the contract: if you re-point this demo at different addressing, the VIP has to move out of the VNet ranges with it.

Renaming is a rebuild, not a rename

Because every name derives from var.component, changing it renames the whole deployment — but site_prefix is not a cosmetic change. It becomes the ClusterName in each CE’s cloud-init, custom_data is replace-only, and cloud-init runs only on a CE’s first boot. So changing it replaces every CE VM and its XC site, and the VIP is then briefly lossy while ECMP reconverges.

That is not theoretical: adopting these derived names on the running demo replaced three CE VMs, three sites, three BGP objects, the load balancer, the origin pool, the Azure Route Server, the Bastion and the test client — 23 to add, 3 to change, 26 to destroy. Pin site_prefix to the current value when you need existing sites to stay put.

Where to go next

Deploy itPrerequisites, the two values you must supply, why the first apply is two-phase, and how to tear it down.

HA Options ComparisonIn-depth inflection point guide comparing dynamic BGP/ECMP routing against Azure Internal Load Balancer (ILB) ingress.

The TerraformThe configuration itself, read from the files that deploy it rather than retyped.

Prove it healthyFour checks in order, and the normal-looking results that read as an outage and are not.

Present itWhat this demo can show, what it cannot yet show, and the one command that makes ECMP visible.