Engineering Specification

This document serves as the human-readable Engineering Specification for the Active/Active eBGP/ECMP F5 XC Customer Edge (Secure Mesh v2) architecture on Microsoft Azure. It functions as both an architectural specification and an execution prompt for xcsh -p.

---

1. System Overview & Objective

Observed 2026-08-05. Architectural specification:

The deployment provisions an Active/Active eBGP/ECMP F5 Distributed Cloud (XC) Customer Edge (CE) Secure Mesh v2 topology on Microsoft Azure with an Amazon Web Services (AWS) extension, a parallel Canadian regional extension, and an On-Premise KVM extension.

• Routing Fabric: CEs originate host routes (/32) via eBGP (ASN 64512) to an Azure Route Server, AWS VPC Router, or containerized ToR BGP router (ASN 65515). The VNets, VPCs, and local libvirt networks use native ECMP for active/active traffic distribution.
• Four Network Paths:
  1. Rest of World (ROW): Serves mcn-ce-ha.f5-sales-demo.com via 3 CEs in eastus (10.0.0.0/16 VNet) with global Public Regional Edge advertisement.
  2. Canada Extension: Serves mcn-ce-ha.f5-sales-demo.ca via 3 CEs in canadacentral (10.200.0.0/16 VNet) advertised strictly through Canadian Regional Edges (Toronto and Montreal) and Canadian CEs.
  3. AWS Customer Edge Extension: Serves aws.mcn-ce-ha.f5-sales-demo.com via 3 EC2 CEs (m5.2xlarge) in AWS VPC (10.150.0.0/16 in us-east-2) with dual NICs (public SLO / private SLI) and eBGP/ECMP routing.
  4. On-Premise KVM Extension: Serves on-premise workloads via 3 KVM CEs (onprem-ce-01, 02, 03) on local libvirt network ce-bgp-net (10.100.0.0/24) peering eBGP (CE ASN 64512) with a containerized FRR ToR BGP Router (10.100.0.1, ASN 65515) configured with maximum-paths 4 ECMP.

---

2. Provider Requirements & Tenant Guardrails

Provider Specifications (versions.tf, providers.tf)

• Terraform CLI: >= 1.10.0
• Providers:
  • f5-sales-demo/xcsh (>= 3.81.1): api_url is dynamically derived from var.expected_xc_tenant (https://${var.expected_xc_tenant}.console.ves.volterra.io).
  • hashicorp/azurerm (~> 4.0): Declares features {} and subscription_id = var.subscription_id.
  • hashicorp/aws (~> 5.0): Configures AWS region var.aws_location (us-east-2).
  • hashicorp/azuread (~> 3.0): Resolves deployer identity for resource naming and tagging.
  • hashicorp/external (~> 2.3): Executes environment tenant guard script.
  • hashicorp/random (~> 3.0): Generates per-CE Site Console admin passwords.
  • dmacvicar/libvirt (~> 0.8.0 / v0.8.3): Manages local KVM domains, volumes, seed ISOs, and network bridges for on-premise Customer Edge nodes.
• Backend: Partial Azure Blob storage backend (backend "azurerm" {} with backend.hcl.example).

Tenant Guard & CIDR Safety Checks (data.tf, main.tf, scripts/xc-env-tenant.sh)

• Tenant Guard: Executable script scripts/xc-env-tenant.sh (chmod +x) outputs JSON {"tenant":"<label>","api_url_set":"<bool>"} from XCSH_API_URL. A lifecycle { postcondition } block on data.external.xc_env_tenant asserts the ambient tenant matches var.expected_xc_tenant (f5-sales-demo), failing the plan if pointed at the wrong tenant.
• CIDR Safety Checks: check "vip_outside_vnet_cidrs" and check "ca_vip_outside_vnet_cidrs" assert that VIPs (10.250.0.10 and 10.250.1.10) sit strictly outside all hub and spoke VNet CIDRs to prevent VNet system routes from overriding BGP host routes.

---

3. Appliance Bootstrapping (cloud-init/ce-node.yaml)

Each CE virtual machine renders a #cloud-config template on first boot:

• VPM Configuration: Writes /etc/vpm/config.yaml (0600, root:root) containing:
  • Vpm: ClusterType: ce, ClusterName: ${cluster_name}, Token: ${token}
  • MauriceEndpoint: https://register.ves.volterra.io
  • MauricePrivateEndpoint: https://register-tls.ves.volterra.io
  • CertifiedHardwareEndpoint: https://vesio.blob.core.windows.net/releases/certified-hardware/azure.yml
  • Kubernetes: EtcdUseTLS: true, Server: vip, CloudProvider: disabled
• SSH Access: Writes operator public SSH key to /var/home/admin/.ssh/authorized_keys (0600, admin:admin) for Site Console CLI access.

---

4. Modular Infrastructure Topologies (modules/)

Topology Expansion (modules/ce-topology)

Pure-computation module expanding ce_count (1..3) into per-CE node maps:

• hostname: <hostname_prefix>-0<n>
• site_name: <site_prefix>-<region_short>0<n>
• slo_ip: cidrhost(mgmt_subnet_prefix, 4 + i)
• az: Availability zones 1, 2, 3
• interface_name: ves-io-securemesh-site-v2-<site_prefix>-<region_short>0<n>-network-<hostname_prefix>-0<n>-eth0-0

Azure Hub Infrastructure (modules/azure-hub)

• Resource Group: rg-<component>-<deployer> where <deployer> resolves from Azure AD via 4-tier fallback logic in locals.tf.
• VNet & Subnets: Hub VNet (hub-vnet) with snet-hub-management, snet-hub-external, snet-hub-internal, RouteServerSubnet (/27, no NSG), and optional AzureBastionSubnet (/26).
• Azure Route Server: <component>-rs with Standard Public IP and ASN 65515.
• Azure Bastion: <component>-bastion (Standard SKU, tunneling_enabled = true, ip_connect_enabled = true).

Customer Edge Nodes (modules/ce-node)

• User-Assigned Identity per CE node.
• 3 NICs per VM (mgmt/SLO with private_ip_address_allocation = "Static", external Dynamic, internal Dynamic) with ip_forwarding_enabled = true and accelerated_networking_enabled = false.
• azurerm_linux_virtual_machine: Admin username azureuser (Azure blocks reserved name admin). Uses marketplace image volterraedgeservices/volterra-node/volterra-node with required plan block, 64 GB StandardSSD_LRS OS disk, and custom_data = base64encode(cloud_init).

F5 XC Site & BGP Binding (modules/xc-site)

• XC Site: Resource xcsh_securemesh_site_v2 with explicit eth0 interface in azure.not_managed.node_list.interface_list with ethernet_interface and network_option.site_local_network {} bound to mgmt NIC MAC.
• Node Rebuild Lifecycle Coupling: Contains terraform_data.ce_vm parking the Azure VM instance ID with lifecycle { replace_triggered_by = [terraform_data.ce_vm] } inside xcsh_securemesh_site_v2. Rebuilding the VM destroys the site object, releasing its stale registration index and preventing UniqueSecondaryIndexViolation on re-boot.
• Registration Approval: Data source data.xcsh_site_registration resolves runtime r-<uuid> registration name, approved automatically by xcsh_registration_approval with name = data.xcsh_site_registration.this.name and namespace = "system".
• eBGP Peering: Resource xcsh_bgp configures CE ASN 64512 to Azure Route Server ASN 65515 on port 179 bound to interface var.interface_name in system namespace, using where.site.ref (network_type = "VIRTUAL_NETWORK_SITE_LOCAL", disable_internet_vip {}), bgp_parameters with asn = var.ce_asn and local_address {}, and dynamic peers (external block with asn = var.rs_asn, address = var.rs_peer_ips[peers.value], port = 179, interface reference to var.interface_name in system namespace, disable_v6 {}, passive_mode_disabled {}, bfd_disabled {}).

Route Server Peering & Test Client

• modules/azure-route-server-bgp: Resource azurerm_route_server_bgp_connection connecting Route Server to CE management private IP.
• modules/client-vm: Test client Ubuntu VM in snet-hub-internal with NSG, public IP, admin_username = "azureuser", and private_ip_address_allocation = "Dynamic".

On-Premise KVM Infrastructure (kvm.tf, onprem_kvm.tf, providers_libvirt.tf)

• libvirt Provider: provider "libvirt" targeting local QEMU hypervisor (qemu:///system).
• Network Bridge: libvirt_network.ce_bgp_net creating NAT-mode bridge virbr-ce-bgp on subnet 10.100.0.0/24 with DHCP and DNS enabled.
• Base Image & Storage: libvirt_volume.base_cloud importing base cloud OS image, and libvirt_volume.ce_disk provisioning 20 GB root overlay disks per CE (onprem-ce-01, 02, 03).
• Cloud-Init ISO Seed: libvirt_cloudinit_disk.ce_cloudinit injecting /etc/vpm/config.yaml with ClusterName: onprem-kvm-site and ClusterType: ce.
• KVM Domains: libvirt_domain.ce_node defining 2 vCPU / 2048 MB RAM VMs with autostart enabled, attached to ce-bgp-net.
• F5 XC Site & eBGP: xcsh_securemesh_site_v2.onprem_kvm (onprem-kvm-site) and xcsh_bgp.onprem_ebgp peering CE ASN 64512 to containerized FRR ToR BGP Router (10.100.0.1, ASN 65515) on port 179 over eth0.

---

5. Application Tier & Regional Extension (main.tf)

Rest of World App Tier

• Registration Token: Tenant-scoped xcsh_token.ce (mcn-ce-registration in system namespace, no type argument).
• Namespace Lookup: Read-only lookup data.xcsh_namespace.mcn (var.xc_app_namespace defaulting to multi-cloud-networking, namespace = "").
• Origin Pool: xcsh_origin_pool.this targeting var.origin_ip:var.origin_port with ROUND_ROBIN and DISTRIBUTED selection.
• HTTP Load Balancer: xcsh_http_loadbalancer.this (mcn-ce-ha-f5se) serving var.lb_domain with advertise_custom advertising VIP 10.250.0.10 on SITE_NETWORK_OUTSIDE per CE site, with explicit depends_on = [module.xc_site].

Canada Regional Extension (enable_canada = true)

• Parallel Hub VNet (10.200.0.0/16 in canadacentral), 3 Canadian CE VMs, Canada Route Server, Canadian eBGP connections, Client VM.
• Canadian Virtual Sites:
  • xcsh_virtual_site.canada_re: Targets Regional Edge cities toronto and montreal (ves.io/city in (toronto, montreal)).
  • xcsh_virtual_site.canada_ce: Targets Canadian CE site names (ves.io/siteName).
• Canadian Origin Pool & Load Balancer: xcsh_origin_pool.canada and xcsh_http_loadbalancer.canada serving mcn-ce-ha.f5-sales-demo.ca advertising VIP 10.250.1.10/32.

---

6. Deployment Artifacts & Outputs (outputs.tf)

Exports operational metrics required for verification:

• Node maps, private/public IPs, Route Server IDs and peer IPs, BGP peer addresses.
• Domain names, advertised VIPs, client NIC names.
• Sensitive ce_registration_token and per-CE Site Console admin passwords (site_console_admin_passwords).