Engineering Specification

This document serves as the human-readable Engineering Specification for the Active/Active eBGP/ECMP F5 XC Customer Edge (Secure Mesh v2) architecture on Microsoft Azure. It functions as both an architectural specification and an execution prompt for xcsh -p.

---

Evidence scope

The SMSv2 guide is the operator reference for current AWS Transit Gateway Connect behavior: three independent sites, six physical interfaces, six Connect peers, and twelve BGP sessions. PR #1103 proves AWS failover and serial upgrades. Azure observations below retain their dates; the final provider-v8 acceptance did not mutate or re-prove Azure end-to-end.

1. System Overview & Objective

Observed 2026-08-05. Architectural specification:

The deployment provisions an Active/Active eBGP/ECMP F5 Distributed Cloud (XC) Customer Edge (CE) Secure Mesh v2 topology on Microsoft Azure with an Amazon Web Services (AWS) extension, a parallel Canadian regional extension, and an On-Premise KVM extension.

• Routing Fabric: CEs originate host routes (/32) via eBGP (ASN 64512) to an Azure Route Server, AWS Transit Gateway Connect, or containerized ToR BGP router (ASN 65515). The VNets, VPCs, and local libvirt networks use native ECMP for active/active traffic distribution.
• Four Network Paths:
  1. Rest of World (ROW): Serves mcn-ce-ha.f5-sales-demo.com via 3 CEs in eastus (10.0.0.0/16 VNet) with global Public Regional Edge advertisement.
  2. Canada Extension: Serves mcn-ce-ha.f5-sales-demo.ca via 3 CEs in canadacentral (10.200.0.0/16 VNet) advertised strictly through Canadian Regional Edges (Toronto and Montreal) and Canadian CEs.
  3. AWS Customer Edge Extension: Serves aws.mcn-ce-ha.f5-sales-demo.com via 3 EC2 CEs (m5.2xlarge) in AWS VPC (10.150.0.0/16 in ap-northeast-1) with dual NICs (public SLO / private SLI) and eBGP/ECMP routing.
  4. On-Premise KVM Extension: One CE (onprem-ce-01, 10.100.0.11) peers with FRR (10.100.0.2, ASN 65515) on the local libvirt network (10.100.0.0/24). The NAT gateway is 10.100.0.1, the workload is 10.100.0.100, and one BGP peer imports 198.51.100.0/24.

---

2. Provider Requirements & Tenant Guardrails

Provider Specifications (versions.tf, providers.tf)

• Terraform CLI: >= 1.16.1
• Providers:
  • f5-sales-demo/xcsh (= 11.0.2): api_url is dynamically derived from var.expected_xc_tenant (https://${var.expected_xc_tenant}.console.ves.volterra.io).
  • hashicorp/azurerm (~> 4.0): Declares features {} and subscription_id = var.subscription_id.
  • hashicorp/aws (~> 5.0): Configures AWS region var.aws_location (ap-northeast-1).
  • hashicorp/azuread (~> 3.0): Resolves deployer identity for resource naming and tagging.
  • hashicorp/external (~> 2.3): Executes environment tenant guard script.
  • hashicorp/random (~> 3.0): Generates per-CE Site Console admin passwords.
  • dmacvicar/libvirt (~> 0.8.0 / v0.8.3): Manages local KVM domains, volumes, seed ISOs, and network bridges for on-premise Customer Edge nodes.
• Backend: Partial Azure Blob storage backend (backend "azurerm" {} with backend.hcl.example).

Tenant Guard & CIDR Safety Checks (data.tf, main.tf, scripts/xc-env-tenant.sh)

• Tenant Guard: Executable script scripts/xc-env-tenant.sh (chmod +x) outputs JSON {"tenant":"<label>","api_url_set":"<bool>"} from XCSH_API_URL. A lifecycle { postcondition } block on data.external.xc_env_tenant asserts the ambient tenant matches var.expected_xc_tenant (f5-sales-demo), failing the plan if pointed at the wrong tenant.
• CIDR Safety Checks: check "vip_outside_vnet_cidrs" and check "ca_vip_outside_vnet_cidrs" assert that VIPs (10.250.0.10 and 10.250.1.10) sit strictly outside all hub and spoke VNet CIDRs to prevent VNet system routes from overriding BGP host routes.

---

3. Appliance Bootstrapping (cloud-init/ce-node.yaml)

Each Azure CE virtual machine renders this Azure-specific #cloud-config template on first boot. AWS uses the separate site-bound JWT and certified-image flow, which preserves /etc/vpm/config.yaml and injects /etc/vpm/user_data:

• VPM Configuration: Writes /etc/vpm/config.yaml (0600, root:root) containing:
  • Vpm: ClusterType: ce, ClusterName: ${cluster_name}, Token: ${token}
  • MauriceEndpoint: https://register.ves.volterra.io
  • MauricePrivateEndpoint: https://register-tls.ves.volterra.io
  • CertifiedHardwareEndpoint: https://vesio.blob.core.windows.net/releases/certified-hardware/azure.yml
  • Kubernetes: EtcdUseTLS: true, Server: vip, CloudProvider: disabled
• SSH Access: Writes operator public SSH key to /var/home/admin/.ssh/authorized_keys (0600, admin:admin) for Site Console CLI access.

---

4. Modular Infrastructure Topologies (modules/)

Topology Expansion (modules/ce-topology)

Pure-computation module expanding ce_count (1..3) into per-CE node maps:

• hostname: <hostname_prefix>-0<n>
• site_name: <site_prefix>-<region_short>0<n>
• slo_ip: cidrhost(mgmt_subnet_prefix, 4 + i)
• az: Availability zones 1, 2, 3
• interface_name: ves-io-securemesh-site-v2-<site_prefix>-<region_short>0<n>-network-<hostname_prefix>-0<n>-eth0-0

Azure Hub Infrastructure (modules/azure-hub)

• Resource Group: rg-<component>-<deployer> where <deployer> resolves from Azure AD via 4-tier fallback logic in locals.tf.
• VNet & Subnets: Hub VNet (hub-vnet) with snet-hub-management, snet-hub-external, snet-hub-internal, RouteServerSubnet (/27, no NSG), and optional AzureBastionSubnet (/26).
• Azure Route Server: <component>-rs with Standard Public IP and ASN 65515.
• Azure Bastion: <component>-bastion (Standard SKU, tunneling_enabled = true, ip_connect_enabled = true).

Customer Edge Nodes (modules/ce-node)

• User-Assigned Identity per CE node.
• 3 NICs per VM (mgmt/SLO with private_ip_address_allocation = "Static", external Dynamic, internal Dynamic) with ip_forwarding_enabled = true and accelerated_networking_enabled = false.
• azurerm_linux_virtual_machine: Admin username azureuser (Azure blocks reserved name admin). Uses marketplace image volterraedgeservices/volterra-node/volterra-node with required plan block, 64 GB StandardSSD_LRS OS disk, and custom_data = base64encode(cloud_init).

F5 XC Site & BGP Binding (modules/xc-site)

• XC Site: Resource xcsh_securemesh_site_v2 with explicit eth0 interface in azure.not_managed.node_list.interface_list with ethernet_interface and network_option.site_local_network = {} bound to mgmt NIC MAC.
• Node Rebuild Lifecycle Coupling: Contains terraform_data.ce_vm parking the Azure VM instance ID with lifecycle { replace_triggered_by = [terraform_data.ce_vm] } inside xcsh_securemesh_site_v2. Rebuilding the VM destroys the site object, releasing its stale registration index and preventing UniqueSecondaryIndexViolation on re-boot.
• Registration Approval: Data source data.xcsh_site_registration resolves runtime r-<uuid> registration name, approved automatically by xcsh_registration_approval with name = data.xcsh_site_registration.this.name and namespace = "system".
• eBGP Peering: Resource xcsh_bgp configures CE ASN 64512 to Azure Route Server ASN 65515 on port 179 bound to interface var.interface_name in system namespace, using where.site.ref (network_type = "VIRTUAL_NETWORK_SITE_LOCAL", disable_internet_vip = {}), bgp_parameters with asn = var.ce_asn and local_address = {}, and dynamic peers (external block with asn = var.rs_asn, address = var.rs_peer_ips[peers.value], port = 179, interface reference to var.interface_name in system namespace, disable_v6 = {}, passive_mode_disabled = {}, bfd_disabled = {}).

Route Server Peering & Test Client

• modules/azure-route-server-bgp: Resource azurerm_route_server_bgp_connection connecting Route Server to CE management private IP.
• modules/client-vm: Test client Ubuntu VM in snet-hub-internal with NSG, public IP, admin_username = "azureuser", and private_ip_address_allocation = "Dynamic".

On-Premise KVM Infrastructure (kvm.tf, onprem_kvm.tf, providers_libvirt.tf)

• libvirt Provider: provider "libvirt" targeting local QEMU hypervisor (qemu:///system).
• Network Bridge: libvirt_network.ce_bgp_net creates a generation-named NAT bridge from the declared MAC/IP reservation map, with DHCP and DNS enabled.
• Base Image & Storage: libvirt_volume.base_cloud imports only the F5-issued KVM appliance URL from xcsh_site_image; libvirt_volume.ce_disk provisions one 100 GiB root overlay disk (onprem-ce-01). The pinned v11.0.2 provider resolves the image by the uniquely owned Site UID, checks its exact response mapping, HTTPS URL and MD5, and rechecks ownership before libvirt mutation.
• Cloud-Init ISO Seed: libvirt_cloudinit_disk.ce_cloudinit injects the site-bound SMSv2 token into /etc/vpm/user_data for the generated ${local.site_prefix}-kvm site.
• KVM Domain: libvirt_domain.ce_node defines one 8 vCPU / 32 GiB RAM CE with autostart enabled, attached to ce-bgp-net at 10.100.0.11.
• F5 XC Site & eBGP: xcsh_securemesh_site_v2.onprem_kvm creates the non-AppStack Secure Mesh Site v2 site, and xcsh_bgp.onprem_ebgp peers CE ASN 64512 with the Terraform-owned FRR router (10.100.0.2, ASN 65515) over eth0; 10.100.0.1 remains the libvirt NAT gateway.

---

5. Application Tier & Regional Extension (main.tf)

Rest of World App Tier

• Azure Registration Token: Tenant-scoped xcsh_token.ce (mcn-ce-registration in system namespace, no type argument).
• Namespace Lookup: Read-only lookup data.xcsh_namespace.mcn (var.xc_app_namespace defaulting to multi-cloud-networking, namespace = "").
• Origin Pool: xcsh_origin_pool.this targeting var.origin_ip:var.origin_port with ROUND_ROBIN and DISTRIBUTED selection.
• HTTP Load Balancer: xcsh_http_loadbalancer.this (mcn-ce-ha-f5se) serving var.lb_domain with advertise_custom advertising VIP 10.250.0.10 on SITE_NETWORK_OUTSIDE per CE site, with explicit depends_on = [module.xc_site].

Canada Regional Extension (enable_canada = true)

• Parallel Hub VNet (10.200.0.0/16 in canadacentral), 3 Canadian CE VMs, Canada Route Server, Canadian eBGP connections, Client VM.
• Canadian Virtual Sites:
  • xcsh_virtual_site.canada_re: Targets Regional Edge cities toronto and montreal (ves.io/city in (toronto, montreal)).
  • xcsh_virtual_site.canada_ce: Targets Canadian CE site names (ves.io/siteName).
• Canadian Origin Pool & Load Balancer: xcsh_origin_pool.canada and xcsh_http_loadbalancer.canada serving mcn-ce-ha.f5-sales-demo.ca advertising VIP 10.250.1.10/32.

---

6. Deployment Artifacts & Outputs (outputs.tf)

Exports operational metrics required for verification:

• Node maps, private/public IPs, Route Server IDs and peer IPs, BGP peer addresses.
• Domain names, advertised VIPs, client NIC names.
• Sensitive ce_registration_token and per-CE Site Console admin passwords (site_console_admin_passwords).