The Terraform

Every file below is the file that deploys, imported at build time rather than pasted. A copy that can drift is a copy that will, so a check in CI fails the moment what is published here differs from what is in terraform/.

Why the configuration is imported rather than shown inline

Pasted into a code fence, this configuration would be part of the page source, and the page source is translated into twelve languages. The comments inside it would be translated with it, and each locale would publish a different, wrong configuration. Imported, it never enters the page source at all.

Root module

terraform/backend.tf

Azure storage backend; its configuration is supplied at init time and not committed.

terraform/backend.tf

terraform {
  # Azure Blob Storage remote state, configured as a PARTIAL backend: no
  # environment-specific values are hardcoded here. Supply them at init time.
  #
  #   Local: terraform init -backend-config=backend.hcl   (copy backend.hcl.example; gitignored)
  #   CI:    terraform init -backend=false                (no state; config-validity + plan tests only)
  #
  # Auth is the storage account access key via the ARM_ACCESS_KEY environment
  # variable (never committed).
  backend "azurerm" {}
}

terraform/data.tf

Read-only lookups, including the Azure AD identity used to derive the deployer.

terraform/data.tf

# Deployer identity resolution (read-only, azuread). Only evaluated during a
# real root plan/apply — the plan-level tests target child modules and/or mock
# the azuread provider, so these are never read without credentials.
data "azuread_client_config" "current" {}

data "azuread_user" "current" {
  count     = var.deployer == "" ? 1 : 0
  object_id = data.azuread_client_config.current.object_id
}

terraform/kvm.tf

Declarative KVM & libvirt infrastructure managed natively via dmacvicar/libvirt (v0.8.3). Provisions local network bridge ce-bgp-net (10.100.0.0/24), base cloud storage volumes, per-CE overlay disks, cloud-init seed ISOs containing /etc/vpm/config.yaml bootstrap configs, and KVM virtual machine domains (onprem-ce-01, 02, 03).

terraform/kvm.tf

# KVM / libvirt Network for On-Prem Customer Edge nodes
resource "libvirt_network" "ce_bgp_net" {
  name      = "ce-bgp-net"
  mode      = "nat"
  domain    = "ce.local"
  addresses = ["10.100.0.0/24"]

  bridge = "virbr-ce-bgp"

  autostart = true

  dhcp {
    enabled = true
  }

  dns {
    enabled = true
  }
}

# Base cloud OS image volume in libvirt
resource "libvirt_volume" "base_cloud" {
  name   = "base-cloud-noble.qcow2"
  pool   = "default"
  source = "${path.module}/../kvm/images/base-cloud.qcow2"
  format = "qcow2"
}

# Per-CE root overlay disks
resource "libvirt_volume" "ce_disk" {
  for_each       = toset(["01", "02", "03"])
  name           = "onprem-ce-${each.key}-disk.qcow2"
  pool           = "default"
  base_volume_id = libvirt_volume.base_cloud.id
  size           = 21474836480
  format         = "qcow2"
}

# Cloud-Init ISO seed disks per CE node
resource "libvirt_cloudinit_disk" "ce_cloudinit" {
  for_each  = toset(["01", "02", "03"])
  name      = "onprem-ce-${each.key}-cloudinit.iso"
  pool      = "default"
  user_data = <<-EOF
    #cloud-config
    hostname: onprem-ce-${each.key}
    write_files:
      - path: /etc/vpm/config.yaml
        permissions: '0600'
        owner: root:root
        content: |
          Vpm:
            ClusterType: ce
            ClusterName: onprem-kvm-site
            Token: ${xcsh_token.ce.id}
            MauriceEndpoint: https://register.ves.volterra.io
            MauricePrivateEndpoint: https://register-tls.ves.volterra.io
            CertifiedHardwareEndpoint: https://vesio.blob.core.windows.net/releases/certified-hardware/azure.yml
          Kubernetes:
            EtcdUseTLS: true
            Server: vip
            CloudProvider: disabled
  EOF

  meta_data = <<-EOF
    instance-id: onprem-ce-${each.key}
    local-hostname: onprem-ce-${each.key}
  EOF
}

# Declarative KVM Virtual Machines managed by Terraform
resource "libvirt_domain" "ce_node" {
  for_each  = toset(["01", "02", "03"])
  name      = "onprem-ce-${each.key}"
  memory    = 2048
  vcpu      = 2
  autostart = true

  cloudinit = libvirt_cloudinit_disk.ce_cloudinit[each.key].id

  network_interface {
    network_id     = libvirt_network.ce_bgp_net.id
    wait_for_lease = false
  }

  disk {
    volume_id = libvirt_volume.ce_disk[each.key].id
  }

  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }

  graphics {
    type        = "vnc"
    listen_type = "address"
    autoport    = true
  }
}

terraform/locals.tf

Where the derived object names live. Terraform variable defaults cannot reference other variables, so each name variable defaults to null and is resolved here.

terraform/locals.tf

locals {
  # --- F5 XC tenant endpoint ---
  # Every F5 XC tenant is served at https://<tenant>.console.ves.volterra.io, so
  # naming the tenant is enough to name the API. providers.tf feeds this to the
  # xcsh provider's api_url, which is what makes the TENANT A PROPERTY OF THE
  # CONFIGURATION rather than of whatever XCSH_API_URL the shell happens to hold.
  xc_api_url = "https://${var.expected_xc_tenant}.console.ves.volterra.io"

  # --- Deployer resolution (4-tier fallback) ---
  # 1. Explicit override via var.deployer
  # 2a. Azure AD: given_name initial + surname
  # 2b. Azure AD: mail prefix (guest/external accounts)
  # 3. Object ID hash (service principals, managed identities)
  deployer_from_name = (
    var.deployer == "" && length(data.azuread_user.current) > 0
    ? try(
      lower("${substr(data.azuread_user.current[0].given_name, 0, 1)}${data.azuread_user.current[0].surname}"),
      ""
    )
    : ""
  )

  deployer_from_mail = (
    var.deployer == "" && length(data.azuread_user.current) > 0 && local.deployer_from_name == ""
    ? try(
      lower(split("@", data.azuread_user.current[0].mail)[0]),
      ""
    )
    : ""
  )

  deployer_from_oid = substr(sha1(data.azuread_client_config.current.object_id), 0, 8)

  deployer_resolved = coalesce(
    var.deployer,
    local.deployer_from_name,
    local.deployer_from_mail,
    local.deployer_from_oid
  )

  deployer = replace(lower(local.deployer_resolved), "/[^a-z0-9]/", "")

  # --- Derived object names ---
  # Every name in the deployment descends from var.component (plus the resolved
  # deployer for the resource group, which is per-person by nature). Terraform
  # variable defaults cannot reference other variables, so each of these variables
  # defaults to null and is resolved here instead; an explicit value always wins.
  #
  # The point is that NO object name is a literal anyone has to maintain, and none
  # can carry a customer's or an individual's name by accident: change
  # var.component and the sites, load balancer, origin pool, Route Server, Bastion
  # and resource group all follow.
  region_short        = coalesce(var.region_short, var.location)
  site_prefix         = coalesce(var.site_prefix, var.component)
  resource_group_name = coalesce(var.resource_group_name, "rg-${var.component}-${local.deployer}")
  route_server_name   = coalesce(var.route_server_name, "${var.component}-rs")
  bastion_name        = coalesce(var.bastion_name, "${var.component}-bastion")
  client_vm_name      = coalesce(var.client_vm_name, "${var.component}-client")
  origin_pool_name    = coalesce(var.origin_pool_name, "${var.component}-pool")
  # `-f5se` matches the convention this tenant's other load balancers already use.
  lb_name = coalesce(var.lb_name, "${var.component}-f5se")

  # --- Derived Canada object names ---
  ca_region_short        = coalesce(var.ca_region_short, var.ca_location)
  ca_site_prefix         = coalesce(var.ca_site_prefix, "${var.component}-ca")
  ca_resource_group_name = coalesce(var.ca_resource_group_name, "rg-${var.component}-ca-${local.deployer}")
  ca_route_server_name   = coalesce(var.ca_route_server_name, "${var.component}-ca-rs")
  ca_bastion_name        = coalesce(var.ca_bastion_name, "${var.component}-ca-bastion")
  ca_client_vm_name      = coalesce(var.ca_client_vm_name, "${var.component}-ca-client")
  ca_origin_pool_name    = coalesce(var.ca_origin_pool_name, "${var.component}-ca-pool")
  ca_lb_name             = coalesce(var.ca_lb_name, "${var.component}-ca-f5se")
  ca_re_vsite_name       = coalesce(var.ca_re_vsite_name, "${var.component}-ca-re-vsite")
  ca_ce_vsite_name       = coalesce(var.ca_ce_vsite_name, "${var.component}-ca-ce-vsite")

  # --- Standard tags (applied to every Azure resource) ---
  standard_tags = {
    component   = var.component
    environment = var.environment
    deployer    = local.deployer
    managed_by  = "terraform"
  }

  tags = merge(local.standard_tags, var.tags)

  # --- SSH public key material, read once at the root ---
  # When ssh_public_key material is supplied (e.g. by the plan tests) it wins and
  # no file is read; otherwise read the key file once and pass the string down.
  ssh_public_key = var.ssh_public_key != "" ? var.ssh_public_key : file(pathexpand(var.ssh_public_key_path))

  # --- CE site registration token fed to cloud-init ---
  # Prefer the provider-generated xcsh_token.ce.uid (the Computed token VALUE);
  # an explicit var.registration_token still wins when supplied (break-glass /
  # externally-minted token). Empty var (default) => the generated token.
  ce_registration_token = var.registration_token != "" ? var.registration_token : xcsh_token.ce.uid

  # --- CE cloud-init, rendered once per node ---
  # Rendered here rather than inline in the module block so the document is
  # addressable as local.ce_cloud_init in `terraform test` — the rendered YAML is
  # the whole contract with the appliance, and it is only worth asserting if it can
  # be read. See tests/cloud_init.tftest.hcl.
  ce_cloud_init = {
    for key, node in module.ce_topology.ce_nodes : key => templatefile("${path.module}/cloud-init/ce-node.yaml", {
      cluster_name = node.site_name
      token        = local.ce_registration_token
      # chomp: a key read from a .pub file ends in a newline, which would render a
      # second, empty line into authorized_keys under `content: |`.
      ssh_public_key = chomp(local.ssh_public_key)
    })
  }

  # --- Canada CE cloud-init, rendered once per node ---
  ca_ce_cloud_init = {
    for key, node in try(module.ce_topology_ca[0].ce_nodes, {}) : key => templatefile("${path.module}/cloud-init/ce-node.yaml", {
      cluster_name   = node.site_name
      token          = local.ce_registration_token
      ssh_public_key = chomp(local.ssh_public_key)
    })
  }
}

terraform/main.tf

Top-level wiring, the tenant guard and the VIP-outside-the-VNet check. The deploy ordering is documented at the top of this file, which is the authoritative version.

terraform/main.tf

# MCN CE-HA (BGP/ECMP) — top-level wiring.
#
# N single-node Secure Mesh v2 CE sites, each originating the LB VIP 10.250.0.10/32
# via eBGP (ASN 64512) to Azure Route Server (ASN 65515). Equal-cost advertisements
# from multiple CEs program ECMP (active/active) into the hub VNet.
#
# Deploy-time ordering: Azure (VNet/subnets/RS/NICs/VMs) -> XC site (explicit
# interface) -> token -> CE cloud-init boot -> CE registers -> registration
# approval -> CE ONLINE -> xcsh_bgp + RS bgpConnection -> LB advertise.
#
# Approval is automated, but the deploy is inherently TWO-PHASE, not a single
# hands-off apply: a CE's registration is named r-<uuid> and only exists after
# the node has booted and registered. The xc-site module resolves that name with
# the xcsh_site_registration data source and approves it with
# xcsh_registration_approval, gated on found — so the first apply plans no
# approval, and a re-apply once the CEs have registered creates them. The bgp/LB
# objects can be applied before ONLINE; they converge once the CE is up.

# Guard: the F5 XC tenant in the environment MUST be the one this deployment
# belongs to.
#
# providers.tf already pins the xcsh endpoint to var.expected_xc_tenant, so a
# stray XCSH_API_URL can no longer redirect the deployment. What it can still do
# is mean the operator's TOKEN belongs to a different tenant — and the symptom of
# that is a bare 401 from the first API call, which reads like an expired
# credential and not like "you are pointed at the wrong tenant". This turns it
# into a sentence that says so.
#
# It is not hypothetical. The MCN demo was built in f5-sales-demo; the credential
# file later started exporting an f5-amer-ent XCSH_API_URL; subsequent applies
# minted an f5-amer-ent token, the CE VMs re-registered there, and their
# f5-sales-demo registrations were abandoned. Every plan in between was clean,
# because nothing in the configuration had an opinion about the tenant (#696).
#
# The external program reads the environment and nothing else — no credential, no
# network call — which is what keeps `terraform test` and CI's
# `terraform init -backend=false` credential-free. Where XCSH_API_URL is unset the
# program reports an empty tenant and the guard abstains.
#
# postcondition, not check{}: a check block only WARNS, and a warning scrolls past
# in exactly the situation this exists to stop.
data "external" "xc_env_tenant" {
  program = ["${path.module}/scripts/xc-env-tenant.sh"]

  lifecycle {
    postcondition {
      condition     = contains(["", var.expected_xc_tenant], self.result.tenant)
      error_message = "Wrong F5 XC tenant. XCSH_API_URL in this environment names tenant '${self.result.tenant}', but this deployment belongs to '${var.expected_xc_tenant}' (state key mcn.tfstate). Source the credential file for '${var.expected_xc_tenant}', or — if you really do mean to act on '${self.result.tenant}' — say so explicitly with -var expected_xc_tenant=${self.result.tenant} and a state key of its own."
    }
  }
}

# Guard: the HA VIP MUST be outside every VNet CIDR, or Azure prefers the VNet
# system route over the more-specific BGP /32. Masks the VIP to each CIDR's prefix
# length and compares network addresses (a correct containment test for any prefix).
check "vip_outside_vnet_cidrs" {
  assert {
    condition     = cidrhost(var.hub_cidr, 0) != cidrhost("${var.vip}/${split("/", var.hub_cidr)[1]}", 0)
    error_message = "vip ${var.vip} must be OUTSIDE hub_cidr ${var.hub_cidr}."
  }
  assert {
    condition     = cidrhost(var.spoke_cidr, 0) != cidrhost("${var.vip}/${split("/", var.spoke_cidr)[1]}", 0)
    error_message = "vip ${var.vip} must be OUTSIDE spoke_cidr ${var.spoke_cidr}."
  }
}

check "ca_vip_outside_vnet_cidrs" {
  assert {
    condition     = !var.enable_canada || cidrhost(var.ca_hub_cidr, 0) != cidrhost("${var.ca_vip}/${split("/", var.ca_hub_cidr)[1]}", 0)
    error_message = "ca_vip ${var.ca_vip} must be OUTSIDE ca_hub_cidr ${var.ca_hub_cidr}."
  }
}

# Tenant-scoped, reusable site registration token. The provider now ships
# xcsh_token with a Computed `uid` (system_metadata.uid) — the token VALUE a CE
# feeds to VPM at registration (the resource `id` is the token NAME, not the
# value). The spec is empty; only metadata is needed and namespace defaults to
# system. This replaces the manual var.registration_token prerequisite (#1205).
# The console approval step (#1206 / #1210) is likewise gone: each CE's runtime
# registration is resolved by site name and approved in the post-registration
# phase (see modules/xc-site/main.tf and the deploy ordering above).
resource "xcsh_token" "ce" {
  name        = "mcn-ce-registration"
  namespace   = "system"
  description = "MCN CE-HA registration token (tenant-scoped, reusable across CE sites)"
}

# Pure expansion of ce_count into the per-CE node map (hostname, site_name,
# slo_ip, az, interface_name). Drives every for_each below.
module "ce_topology" {
  source = "./modules/ce-topology"

  ce_count           = var.ce_count
  region_short       = local.region_short
  mgmt_subnet_prefix = var.mgmt_subnet_prefix
  site_prefix        = local.site_prefix
}

# Hub: RG, VNet, four subnets, Azure Route Server.
module "azure_hub" {
  source = "./modules/azure-hub"

  resource_group_name        = local.resource_group_name
  location                   = var.location
  hub_cidr                   = var.hub_cidr
  mgmt_subnet_prefix         = var.mgmt_subnet_prefix
  external_subnet_prefix     = var.external_subnet_prefix
  internal_subnet_prefix     = var.internal_subnet_prefix
  route_server_subnet_prefix = var.route_server_subnet_prefix
  route_server_name          = local.route_server_name
  bastion_subnet_prefix      = var.bastion_subnet_prefix
  enable_bastion             = var.enable_bastion
  bastion_name               = local.bastion_name
  tags                       = local.tags
}

# One CE VM (3 NICs + identity) per node.
module "ce_node" {
  source   = "./modules/ce-node"
  for_each = module.ce_topology.ce_nodes

  hostname            = each.value.hostname
  resource_group_name = module.azure_hub.resource_group_name
  location            = module.azure_hub.location
  zone                = each.value.az
  vm_size             = var.ce_vm_size
  mgmt_subnet_id      = module.azure_hub.management_subnet_id
  external_subnet_id  = module.azure_hub.external_subnet_id
  internal_subnet_id  = module.azure_hub.internal_subnet_id
  mgmt_private_ip     = each.value.slo_ip
  admin_username      = var.admin_username
  ssh_public_key      = local.ssh_public_key

  custom_data = base64encode(local.ce_cloud_init[each.key])

  tags = local.tags
}

# Generate a distinct Site Console admin password for every CE. Binding the
# password lifecycle to the VM instance rotates it automatically whenever that
# node is rebuilt, while keeping the value only in the encrypted remote state
# and sensitive Terraform output.
resource "random_password" "site_console_admin" {
  for_each = module.ce_topology.ce_nodes

  length           = 32
  min_lower        = 1
  min_numeric      = 1
  min_special      = 1
  min_upper        = 1
  override_special = "!#%*+-=?@^_~"

  keepers = {
    ce_vm_instance_id = module.ce_node[each.key].vm_instance_id
  }
}

# Site Console Basic Auth delegates password verification to VPM on the CE host.
# The XC site's admin_user_credentials field is accepted by the API but VPM
# deliberately skips the built-in admin user, so it does not rotate this
# credential. Apply the generated value on the node as root instead.
#
# The Custom Script extension receives only an encrypted protected setting. Its
# inline script keeps the password out of command arguments and produces no
# output. The password resource is keyed by Azure's per-instance VM id, so a VM
# replacement generates a new value and updates this extension even though the
# name-derived ARM resource id stays the same.
resource "azurerm_virtual_machine_extension" "site_console_password" {
  for_each = module.ce_topology.ce_nodes

  name                       = "site-console-admin-password"
  virtual_machine_id         = module.ce_node[each.key].vm_id
  publisher                  = "Microsoft.Azure.Extensions"
  type                       = "CustomScript"
  type_handler_version       = "2.1"
  auto_upgrade_minor_version = true

  protected_settings = jsonencode({
    script = base64encode(<<-SCRIPT
      #!/usr/bin/env bash
      set -euo pipefail
      password_b64='${base64encode(random_password.site_console_admin[each.key].result)}'
      password=$(printf '%s' "$password_b64" | base64 --decode)
      printf 'admin:%s\n' "$password" | chpasswd
      unset password password_b64
    SCRIPT
    )
  })
}

# One XC SMSv2 site + bgp object per node. The MAC is wired from the mgmt NIC so
# a NIC recreate updates the site binding automatically.
module "xc_site" {
  source   = "./modules/xc-site"
  for_each = module.ce_topology.ce_nodes

  site_name      = each.value.site_name
  hostname       = each.value.hostname
  interface_name = each.value.interface_name
  mgmt_nic_mac   = module.ce_node[each.key].mgmt_nic_mac
  # Couples the site object's lifecycle to the CE VM INSTANCE (issue #674):
  # replacing the VM replaces the site, which takes the registration bound to the
  # destroyed instance with it. Must be virtual_machine_id, not the ARM resource
  # id — the latter is name-derived and identical after a replacement.
  ce_vm_instance_id    = module.ce_node[each.key].vm_instance_id
  rs_peer_ips          = module.azure_hub.rs_peer_ips
  ce_asn               = var.ce_asn
  rs_asn               = var.rs_asn
  os_version           = var.ce_os_version
  sw_version           = var.ce_sw_version
  enable_bgp           = var.enable_bgp
  approve_registration = var.approve_registration
}

# The Azure side of each eBGP session (Route Server -> CE eth0/SLO IP).
module "azure_route_server_bgp" {
  source   = "./modules/azure-route-server-bgp"
  for_each = module.ce_topology.ce_nodes

  name            = "${each.key}-bgp"
  route_server_id = module.azure_hub.route_server_id
  peer_asn        = var.ce_asn
  peer_ip         = module.ce_node[each.key].mgmt_private_ip
}

# Test client in snet-hub-internal.
module "client_vm" {
  source = "./modules/client-vm"

  name                = local.client_vm_name
  resource_group_name = module.azure_hub.resource_group_name
  location            = module.azure_hub.location
  subnet_id           = module.azure_hub.internal_subnet_id
  admin_username      = var.admin_username
  ssh_public_key      = local.ssh_public_key
  tags                = local.tags
}

# ---------------------------------------------------------
# F5 XC data-plane (app tier)
# ---------------------------------------------------------

# The application namespace (origin pool + VIP load balancer live here) is an
# input, never an owned object. `multi-cloud-networking` predates this deployment
# and outlives it, and a `resource` block would put a namespace full of other
# people's demos on this stack's destroy list — which is how issue #637 happened.
#
# Issues #634 ("cannot be recreated, POST is 403") and #639 ("defaults to a
# namespace that no longer exists") were symptoms of reading the WRONG TENANT, not
# of a permissions or lifecycle problem: the namespace exists, in f5-sales-demo,
# and always did. The tenant guard at the top of this file is the actual fix (#696).
# Reading rather than owning the namespace is still correct, for the #637 reason.
#
# Reading it means Terraform never creates, updates or destroys it. The
# read still gives the pool and the load balancer their apply-time ordering: they
# take their namespace from this data source, so a missing namespace fails the
# plan loudly rather than half-applying. `namespace` is empty because a namespace
# object is not itself contained in a namespace (the API returns
# `metadata.namespace: ""` for one).
data "xcsh_namespace" "mcn" {
  name      = var.xc_app_namespace
  namespace = ""
}

resource "xcsh_origin_pool" "this" {
  name        = local.origin_pool_name
  namespace   = data.xcsh_namespace.mcn.name
  description = "MCN reference origin pool -> ${var.origin_ip}:${var.origin_port}"

  port = var.origin_port

  origin_servers {
    labels {}
    public_ip {
      ip = var.origin_ip
    }
  }

  no_tls {}
  loadbalancer_algorithm = "ROUND_ROBIN"
  endpoint_selection     = "DISTRIBUTED"
}

resource "xcsh_http_loadbalancer" "this" {
  # The advertise_custom block below names each CE site, but it takes those names from
  # module.ce_topology — a pure computation module that only derives strings. The
  # objects themselves come from module.xc_site, and nothing in this resource
  # references them, so Terraform sees NO dependency and is free to create the load
  # balancer before any site exists. XC then rejects the dangling site reference with
  # `[BAD_REQUEST] Invalid request parameters` on POST .../http_loadbalancers.
  #
  # The bug is invisible while the sites already exist under the names being
  # referenced, which is why it stayed latent until the demo was renamed: that
  # destroyed every site and created new ones, and the load balancer raced ahead of
  # them. Making the dependency explicit is the fix; there is no cycle, because
  # xc-site does not reference the load balancer.
  depends_on = [module.xc_site]

  name        = local.lb_name
  namespace   = data.xcsh_namespace.mcn.name
  description = "BGP/ECMP HA: custom VIP ${var.vip} advertised from every CE site."

  domains = [var.lb_domain]

  http {
    port                 = 80
    dns_volterra_managed = false
  }

  # Advertise the VIP on the outside network of every CE site.
  advertise_custom {
    dynamic "advertise_where" {
      for_each = module.ce_topology.ce_nodes
      content {
        site {
          network = "SITE_NETWORK_OUTSIDE"
          site {
            namespace = "system"
            name      = advertise_where.value.site_name
          }
          ip = var.vip
        }
        use_default_port {}
      }
    }
  }

  default_route_pools {
    pool {
      namespace = data.xcsh_namespace.mcn.name
      name      = xcsh_origin_pool.this.name
    }
    weight   = 1
    priority = 1
  }

  round_robin {}
  no_challenge {}
  user_id_client_ip {}
  disable_waf {}
  disable_rate_limit {}
  disable_api_discovery {}
  disable_api_testing {}
  disable_api_definition {}
  l7_ddos_protection {}
  service_policies_from_namespace {}
  disable_trust_client_ip_headers {}
  disable_malicious_user_detection {}
  disable_malware_protection {}
  disable_threat_mesh {}
  default_sensitive_data_policy {}
}

# ---------------------------------------------------------
# Canada Regional Infrastructure & F5 XC Data-Plane (Canada Path)
# ---------------------------------------------------------

# Pure expansion of ca_ce_count into the per-CE node map for Canada.
module "ce_topology_ca" {
  count  = var.enable_canada ? 1 : 0
  source = "./modules/ce-topology"

  ce_count           = var.ca_ce_count
  region_short       = local.ca_region_short
  mgmt_subnet_prefix = var.ca_mgmt_subnet_prefix
  site_prefix        = local.ca_site_prefix
}

# Canada Hub: RG, VNet, four subnets, Azure Route Server.
module "azure_hub_ca" {
  count  = var.enable_canada ? 1 : 0
  source = "./modules/azure-hub"

  resource_group_name        = local.ca_resource_group_name
  location                   = var.ca_location
  hub_cidr                   = var.ca_hub_cidr
  mgmt_subnet_prefix         = var.ca_mgmt_subnet_prefix
  external_subnet_prefix     = var.ca_external_subnet_prefix
  internal_subnet_prefix     = var.ca_internal_subnet_prefix
  route_server_subnet_prefix = var.ca_route_server_subnet_prefix
  route_server_name          = local.ca_route_server_name
  bastion_subnet_prefix      = var.ca_bastion_subnet_prefix
  enable_bastion             = var.enable_bastion
  bastion_name               = local.ca_bastion_name
  tags                       = local.tags
}

# One Canadian CE VM (3 NICs + identity) per node.
module "ce_node_ca" {
  for_each = try(module.ce_topology_ca[0].ce_nodes, {})
  source   = "./modules/ce-node"

  hostname            = each.value.hostname
  resource_group_name = try(module.azure_hub_ca[0].resource_group_name, null)
  location            = try(module.azure_hub_ca[0].location, null)
  zone                = each.value.az
  vm_size             = var.ce_vm_size
  mgmt_subnet_id      = try(module.azure_hub_ca[0].management_subnet_id, null)
  external_subnet_id  = try(module.azure_hub_ca[0].external_subnet_id, null)
  internal_subnet_id  = try(module.azure_hub_ca[0].internal_subnet_id, null)
  mgmt_private_ip     = each.value.slo_ip
  admin_username      = var.admin_username
  ssh_public_key      = local.ssh_public_key

  custom_data = base64encode(local.ca_ce_cloud_init[each.key])

  tags = local.tags
}

resource "random_password" "site_console_admin_ca" {
  for_each = try(module.ce_topology_ca[0].ce_nodes, {})

  length           = 32
  min_lower        = 1
  min_numeric      = 1
  min_special      = 1
  min_upper        = 1
  override_special = "!#%*+-=?@^_~"

  keepers = {
    ce_vm_instance_id = module.ce_node_ca[each.key].vm_instance_id
  }
}

resource "azurerm_virtual_machine_extension" "site_console_password_ca" {
  for_each = try(module.ce_topology_ca[0].ce_nodes, {})

  name                       = "site-console-admin-password"
  virtual_machine_id         = module.ce_node_ca[each.key].vm_id
  publisher                  = "Microsoft.Azure.Extensions"
  type                       = "CustomScript"
  type_handler_version       = "2.1"
  auto_upgrade_minor_version = true

  protected_settings = jsonencode({
    script = base64encode(<<-SCRIPT
      #!/usr/bin/env bash
      set -euo pipefail
      password_b64='${base64encode(random_password.site_console_admin_ca[each.key].result)}'
      password=$(printf '%s' "$password_b64" | base64 --decode)
      printf 'admin:%s\n' "$password" | chpasswd
      unset password password_b64
    SCRIPT
    )
  })
}

# Canadian XC SMSv2 site + BGP object per node.
module "xc_site_ca" {
  for_each = try(module.ce_topology_ca[0].ce_nodes, {})
  source   = "./modules/xc-site"

  create_site          = false
  site_name            = each.value.site_name
  hostname             = each.value.hostname
  interface_name       = each.value.interface_name
  mgmt_nic_mac         = module.ce_node_ca[each.key].mgmt_nic_mac
  ce_vm_instance_id    = module.ce_node_ca[each.key].vm_instance_id
  rs_peer_ips          = try(module.azure_hub_ca[0].rs_peer_ips, [])
  ce_asn               = var.ce_asn
  rs_asn               = var.rs_asn
  os_version           = var.ce_os_version
  sw_version           = var.ce_sw_version
  enable_bgp           = var.enable_bgp
  approve_registration = var.approve_registration
}

# Azure Route Server eBGP session for Canadian CEs.
module "azure_route_server_bgp_ca" {
  for_each = try(module.ce_topology_ca[0].ce_nodes, {})
  source   = "./modules/azure-route-server-bgp"

  name            = "${each.key}-bgp"
  route_server_id = try(module.azure_hub_ca[0].route_server_id, null)
  peer_asn        = var.ce_asn
  peer_ip         = module.ce_node_ca[each.key].mgmt_private_ip
}

module "client_vm_ca" {
  count  = var.enable_canada ? 1 : 0
  source = "./modules/client-vm"

  name                = local.ca_client_vm_name
  resource_group_name = try(module.azure_hub_ca[0].resource_group_name, null)
  location            = try(module.azure_hub_ca[0].location, null)
  subnet_id           = try(module.azure_hub_ca[0].internal_subnet_id, null)
  admin_username      = var.admin_username
  ssh_public_key      = local.ssh_public_key
  tags                = local.tags
}

# ---------------------------------------------------------
# F5 XC Virtual Sites (Canada RE & Canada CE)
# ---------------------------------------------------------

resource "xcsh_virtual_site" "canada_re" {
  count     = var.enable_canada ? 1 : 0
  name      = local.ca_re_vsite_name
  namespace = data.xcsh_namespace.mcn.name

  site_type = "REGIONAL_EDGE"
  site_selector {
    expressions = ["ves.io/city in (${join(", ", var.ca_re_cities)})"]
  }
}

resource "xcsh_virtual_site" "canada_ce" {
  count     = var.enable_canada ? 1 : 0
  name      = local.ca_ce_vsite_name
  namespace = data.xcsh_namespace.mcn.name

  site_type = "CUSTOMER_EDGE"
  site_selector {
    expressions = ["ves.io/siteName in (${join(", ", [for k, v in try(module.ce_topology_ca[0].ce_nodes, {}) : v.site_name])})"]
  }
}

resource "xcsh_origin_pool" "canada" {
  count       = var.enable_canada ? 1 : 0
  name        = local.ca_origin_pool_name
  namespace   = data.xcsh_namespace.mcn.name
  description = "Canada reference origin pool -> ${var.origin_ip}:${var.origin_port}"

  port = var.origin_port

  origin_servers {
    labels {}
    public_ip {
      ip = var.origin_ip
    }
  }

  no_tls {}
  loadbalancer_algorithm = "ROUND_ROBIN"
  endpoint_selection     = "DISTRIBUTED"
}

resource "xcsh_http_loadbalancer" "canada" {
  count      = var.enable_canada ? 1 : 0
  depends_on = [module.xc_site_ca, xcsh_virtual_site.canada_re, xcsh_virtual_site.canada_ce]

  name        = local.ca_lb_name
  namespace   = data.xcsh_namespace.mcn.name
  description = "Canada Regional HA: custom VIP ${var.ca_vip} advertised strictly via Canadian Regional Edges (Toronto and Montreal) and Canadian CEs."

  domains = [var.ca_lb_domain]

  http {
    port                 = 80
    dns_volterra_managed = false
  }

  advertise_custom {
    dynamic "advertise_where" {
      for_each = try(module.ce_topology_ca[0].ce_nodes, {})
      content {
        site {
          network = "SITE_NETWORK_OUTSIDE"
          site {
            namespace = "system"
            name      = advertise_where.value.site_name
          }
          ip = var.ca_vip
        }
        use_default_port {}
      }
    }
  }

  default_route_pools {
    pool {
      namespace = data.xcsh_namespace.mcn.name
      name      = try(xcsh_origin_pool.canada[0].name, null)
    }
    weight   = 1
    priority = 1
  }

  round_robin {}
  no_challenge {}
  user_id_client_ip {}
  disable_waf {}
  disable_rate_limit {}
  disable_api_discovery {}
  disable_api_testing {}
  disable_api_definition {}
  l7_ddos_protection {}
  service_policies_from_namespace {}
  disable_trust_client_ip_headers {}
  disable_malicious_user_detection {}
  disable_malware_protection {}
  disable_threat_mesh {}
  default_sensitive_data_policy {}
}

terraform/onprem_kvm.tf

On-Premise KVM SecureMesh v2 Site (xcsh_securemesh_site_v2.onprem_kvm) and eBGP peering configuration (xcsh_bgp.onprem_ebgp). Connects on-premise CEs (ASN 64512) over eth0 to containerized FRR ToR BGP router (10.100.0.1, ASN 65515).

terraform/onprem_kvm.tf

# On-Prem KVM SecureMesh Site v2
resource "xcsh_securemesh_site_v2" "onprem_kvm" {
  name        = "onprem-kvm-site"
  namespace   = "system"
  description = "On-Prem KVM SecureMesh Site v2"

  azure {
    not_managed {}
  }

  disable_ha {}
  block_all_services {}
  no_network_policy {}
  no_forward_proxy {}
  f5_proxy {}
  no_proxy_bypass {}
  logs_streaming_disabled {}
  no_s2s_connectivity_sli {}
  no_s2s_connectivity_slo {}
  disable_url_categorization {}
  disable_management_network {}
}

# eBGP Peering configuration for On-Prem KVM Site
resource "xcsh_bgp" "onprem_ebgp" {
  name      = "onprem-kvm-ebgp"
  namespace = "system"

  where {
    site {
      network_type = "VIRTUAL_NETWORK_SITE_LOCAL"
      ref {
        name      = xcsh_securemesh_site_v2.onprem_kvm.name
        namespace = "system"
      }
      disable_internet_vip {}
    }
  }

  bgp_parameters {
    asn = 64512
    local_address {}
  }

  peers {
    metadata {
      name = "peer-router"
    }
    external {
      asn     = 65515
      address = "10.100.0.1"
      port    = 179

      interface {
        name      = "eth0"
        namespace = "system"
      }

      disable_v6 {}
    }
    passive_mode_disabled {}
    bfd_disabled {}
  }
}

terraform/outputs.tf

Everything the documentation reads instead of naming. If a page shows a command, an output here is behind it.

terraform/outputs.tf

# ---------------------------------------------------------
# Topology
# ---------------------------------------------------------

output "ce_nodes" {
  description = "Expanded per-CE node map (hostname, site_name, slo_ip, az, interface_name)."
  value       = module.ce_topology.ce_nodes
}

output "ce_count" {
  description = "Number of CE nodes deployed."
  value       = module.ce_topology.ce_count
}

# ---------------------------------------------------------
# Azure hub / Route Server
# ---------------------------------------------------------

output "resource_group_name" {
  description = "Hub resource group name."
  value       = module.azure_hub.resource_group_name
}

output "route_server_id" {
  description = "Azure Route Server resource ID."
  value       = module.azure_hub.route_server_id
}

# The four outputs below exist so the documentation can name nothing. Every
# operational value a reader needs has to be readable from the deployment rather
# than copied out of prose, because prose goes stale silently and a reader cannot
# tell. Anything documented as a command therefore needs an output behind it.
output "route_server_name" {
  description = "Azure Route Server name — the --routeserver argument of `az network routeserver peering list-learned-routes`."
  value       = local.route_server_name
}

output "client_vm_name" {
  description = "Test client VM name — the -n argument of `az vm run-command invoke` when driving traffic at the VIP from inside the VNet."
  value       = local.client_vm_name
}

output "lb_domain" {
  description = "Domain the HTTP load balancer matches on. Requests to the VIP MUST send it as the Host header; without it the load balancer has no matching domain and answers 404."
  value       = var.lb_domain
}

output "origin_ip" {
  description = "Origin the pool targets. Useful as a control: a batch straight to the origin, bypassing the VIP, separates an origin fault from a VIP/ECMP/CE fault."
  value       = var.origin_ip
}

output "route_server_peer_ips" {
  description = "Route Server BGP peer IPs (the CE external BGP peer addresses)."
  value       = module.azure_hub.rs_peer_ips
}

output "ce_mgmt_private_ips" {
  description = "Per-CE eth0/SLO private IPs (BGP local addresses / RS bgpConnection peer IPs)."
  value       = { for k, m in module.ce_node : k => m.mgmt_private_ip }
}

output "ce_vm_names" {
  description = "Per-CE VM names."
  value       = { for k, m in module.ce_node : k => m.vm_name }
}

output "ce_sli_private_ips" {
  description = "Per-CE internal/SLI private IPs — where each CE serves its Site Console web UI (TCP 65500). Informational: the Bastion tunnel is targeted by VM resource id (ce_vm_ids), because Azure does not allow a custom resource port over an IP-targeted tunnel."
  value       = { for k, m in module.ce_node : k => m.sli_private_ip }
}

output "ce_vm_ids" {
  description = "Per-CE VM resource IDs — the --target-resource-id of `az network bastion tunnel` when opening the Site Console web UI on 65500."
  value       = { for k, m in module.ce_node : k => m.vm_id }
}

output "site_console_admin_passwords" {
  description = "Generated per-CE passwords for the node-local Site Console admin user. Retrieve only for an active tunnel and keep them out of logs and published documentation."
  value       = { for k, password in random_password.site_console_admin : k => password.result }
  sensitive   = true
}

output "bastion_name" {
  description = "Azure Bastion host name, or null when enable_bastion is false. Feed it to `az network bastion tunnel --name`."
  value       = module.azure_hub.bastion_name
}

output "client_public_ip" {
  description = "Public IP of the test client."
  value       = module.client_vm.public_ip
}

output "client_nic_name" {
  description = "Test client NIC name (read effective routes here to prove ECMP)."
  value       = module.client_vm.nic_name
}

# ---------------------------------------------------------
# XC tenant
# ---------------------------------------------------------

output "xc_tenant" {
  description = "F5 XC tenant this deployment writes to. Config-controlled (var.expected_xc_tenant), not taken from XCSH_API_URL."
  value       = var.expected_xc_tenant
}

output "xc_api_url" {
  description = "F5 XC API endpoint the xcsh provider is pinned to, derived from var.expected_xc_tenant."
  value       = local.xc_api_url
}

# The other half of the tenant guard's comparison, published so an operator can
# see what their shell is claiming without having to trip the guard to find out —
# `terraform output xc_env_tenant` answers "which tenant am I sourced for?".
# Empty means XCSH_API_URL is unset, which is the CI case and which the guard
# treats as no opinion rather than as a mismatch.
output "xc_env_tenant" {
  description = "F5 XC tenant named by XCSH_API_URL in the environment this ran in, or empty when unset. Diagnostic only: xc_tenant is what the deployment actually targets."
  value       = data.external.xc_env_tenant.result.tenant
}

# ---------------------------------------------------------
# XC data-plane
# ---------------------------------------------------------

output "xc_site_names" {
  description = "Per-CE XC site names."
  value       = { for k, m in module.xc_site : k => m.site_name }
}

# Makes the site-to-node binding that closes #674 observable from the CLI. Each
# value is the CE VM instance id its XC site object is coupled to; the matching
# registration reports the same value as infra.instance_id. When the two disagree
# the site is bound to a node that no longer exists — the state in which a
# rebuilt CE can never register. (The registration side is not on the
# xcsh_site_registration data source yet: provider issue #1376.)
output "ce_bound_instance_ids" {
  description = "Per-CE VM instance id each XC site object is bound to. Compare with the registration's infra.instance_id to spot a site still bound to a destroyed node."
  value       = { for k, m in module.xc_site : k => m.bound_vm_instance_id }
}

output "xc_interface_names" {
  description = "Per-CE auto-derived network_interface object names (BGP peer bind target)."
  value       = { for k, m in module.xc_site : k => m.interface_name }
}

output "loadbalancer_name" {
  description = "HTTP load balancer name."
  value       = xcsh_http_loadbalancer.this.name
}

output "origin_pool_name" {
  description = "Origin pool name."
  value       = xcsh_origin_pool.this.name
}

output "vip" {
  description = "HA VIP advertised via eBGP/ECMP."
  value       = var.vip
}

# ---------------------------------------------------------
# Canada Regional outputs
# ---------------------------------------------------------

output "ca_lb_domain" {
  description = "Domain served by the Canada HTTP load balancer."
  value       = var.ca_lb_domain
}

output "ca_re_virtual_site_name" {
  description = "Name of the Canadian Regional Edge virtual site."
  value       = try(xcsh_virtual_site.canada_re[0].name, null)
}

output "ca_ce_virtual_site_name" {
  description = "Name of the Canadian Customer Edge virtual site."
  value       = try(xcsh_virtual_site.canada_ce[0].name, null)
}

output "ca_loadbalancer_name" {
  description = "Name of the Canadian HTTP load balancer."
  value       = try(xcsh_http_loadbalancer.canada[0].name, null)
}

output "ca_origin_pool_name" {
  description = "Name of the Canadian origin pool."
  value       = try(xcsh_origin_pool.canada[0].name, null)
}

output "ca_vip" {
  description = "HA VIP for Canadian CEs advertised via eBGP/ECMP or Azure ILB."
  value       = var.ca_vip
}

output "ca_ilb_id" {
  description = "Azure Internal Load Balancer ID for Canadian regional path."
  value       = try(azurerm_lb.ca_ilb[0].id, null)
}

output "ca_ilb_frontend_ip" {
  description = "Azure Internal Load Balancer frontend private IP for Canadian regional path."
  value       = try(azurerm_lb.ca_ilb[0].frontend_ip_configuration[0].private_ip_address, null)
}

# ---------------------------------------------------------
# CE registration token
# ---------------------------------------------------------

output "registration_token_name" {
  description = "Name (metadata id) of the generated xcsh_token used for CE registration."
  value       = xcsh_token.ce.name
}

output "registration_token_is_generated" {
  description = "True when the CE cloud-init token feed uses the generated xcsh_token.ce.uid (no override supplied)."
  # Whether an override was supplied is not itself secret (the token value is).
  value = nonsensitive(var.registration_token == "")
}

output "ce_registration_token" {
  description = "Resolved CE registration token fed to cloud-init: the generated xcsh_token.ce.uid, or var.registration_token when overridden."
  value       = local.ce_registration_token
  sensitive   = true
}

# ---------------------------------------------------------
# AWS outputs
# ---------------------------------------------------------

output "aws_vpc_id" {
  description = "AWS VPC ID."
  value       = try(aws_vpc.aws[0].id, null)
}

output "aws_ce_instance_ids" {
  description = "EC2 instance IDs of the AWS Customer Edge nodes."
  value       = aws_instance.ce[*].id
}

output "aws_ce_public_ips" {
  description = "Elastic IPs assigned to the AWS Customer Edge nodes."
  value       = aws_eip.ce[*].public_ip
}

output "aws_lb_domain" {
  description = "Domain served by the AWS HTTP load balancer."
  value       = var.aws_lb_domain
}

output "aws_loadbalancer_name" {
  description = "Name of the AWS HTTP load balancer."
  value       = try(xcsh_http_loadbalancer.aws[0].name, null)
}

output "aws_origin_pool_name" {
  description = "Name of the AWS origin pool."
  value       = try(xcsh_origin_pool.aws[0].name, null)
}

output "aws_vip" {
  description = "HA VIP for AWS CEs advertised via eBGP/ECMP."
  value       = var.aws_vip
}

terraform/providers.tf

Pins the xcsh endpoint to var.expected_xc_tenant, which is what makes the tenant a property of the configuration.

terraform/providers.tf

# The xcsh provider takes its CREDENTIAL from the environment — no secrets in
# code — but NOT its endpoint. Export one of the following before running
# Terraform:
#
#   Token auth:  XCSH_API_TOKEN
#   P12 auth:    XCSH_P12_FILE + XCSH_P12_PASSWORD
#   PEM auth:    XCSH_CERT + XCSH_KEY
#
# api_url is set here, from var.expected_xc_tenant, and deliberately overrides any
# XCSH_API_URL in the environment. The tenant is not an ambient property of the
# operator's shell: it is which deployment this is, it belongs in version control
# next to the state key, and it is the one thing a credential file must not be
# able to change silently. It once did — see the guard in main.tf and issue #696.
#
# A credential minted for a different tenant now fails against this URL instead of
# succeeding somewhere unintended. The plan-level tests mock this provider, so no
# XC credentials are needed to run `terraform test`.
provider "xcsh" {
  api_url = local.xc_api_url
}

# Azure — deploys the hub VNet, Route Server, CE VMs and the test client.
# Auth comes from the environment (az CLI login locally; ARM_* / a service
# principal in CI). Only the subscription is set here, from a variable.
provider "azurerm" {
  features {}
  subscription_id = var.subscription_id
}

# AWS — deploys the VPC, subnets, CE EC2 instances and internet gateway.
# Auth comes from the environment (aws CLI login / AWS_* env vars).
provider "aws" {
  region = var.aws_location
}

# Read-only: resolves the deployer identity for resource naming/tags.
provider "azuread" {}

terraform/providers_libvirt.tf

Configures the libvirt provider targeting the local hypervisor daemon URI (qemu:///system).

terraform/providers_libvirt.tf

provider "libvirt" {
  uri = "qemu:///system"
}

terraform/variables.tf

Component, environment, deployer and tags.

terraform/variables.tf

# General variables. Domain-specific inputs live in variables_azure.tf,
# variables_xc.tf and variables_ce.tf.

variable "component" {
  description = "Component name used in tags."
  type        = string
  default     = "mcn-ce-ha"
}

variable "environment" {
  description = "Environment label used in tags."
  type        = string
  default     = "lab"
}

variable "deployer" {
  description = "Override for the deployer identifier used in tags (auto-resolved from Azure AD when empty)."
  type        = string
  default     = ""
}

variable "tags" {
  description = "Additional tags merged with the standard tags (component/environment/deployer/managed_by)."
  type        = map(string)
  default     = {}
}

terraform/variables_aws.tf

AWS location, VPC CIDR, CE count, instance size, VIP, and load balancer domain.

terraform/variables_aws.tf

# ---------------------------------------------------------
# AWS site deployment & placement
# ---------------------------------------------------------

variable "enable_aws" {
  description = "Enable deployment of the AWS Customer Edge site, VPC, EC2 instances, and XC resources."
  type        = bool
  default     = false
}

variable "aws_location" {
  description = "AWS region for all AWS resources."
  type        = string
  default     = "us-east-2"
}

variable "aws_vpc_cidr" {
  description = "AWS VPC address space."
  type        = string
  default     = "10.150.0.0/16"
}

variable "aws_ce_count" {
  description = "Number of Customer Edge EC2 nodes to deploy in AWS."
  type        = number
  default     = 3
}

variable "aws_instance_type" {
  description = "EC2 instance size for the Customer Edge nodes."
  type        = string
  default     = "m5.2xlarge"
}

variable "aws_vip" {
  description = "HA VIP for AWS CEs advertised as a /32 via eBGP."
  type        = string
  default     = "10.150.0.10"

  validation {
    condition     = can(cidrhost("${var.aws_vip}/32", 0))
    error_message = "aws_vip must be a valid IPv4 address."
  }
}

variable "aws_lb_domain" {
  description = "Domain name for the HTTP Load Balancer serving the AWS CE site."
  type        = string
  default     = "aws.mcn-ce-ha.f5-sales-demo.com"
}

terraform/aws_vpc.tf

AWS VPC, public SLO subnets, private SLI subnets, internet gateway, route tables, and security groups.

terraform/aws_vpc.tf

# ---------------------------------------------------------
# AWS VPC, Subnets, Gateways, Route Tables & Security Groups
# ---------------------------------------------------------

data "aws_availability_zones" "available" {
  count = var.enable_aws ? 1 : 0
  state = "available"
}

resource "aws_vpc" "aws" {
  #checkov:skip=CKV2_AWS_11:Lab VPC - flow logging not required
  #checkov:skip=CKV2_AWS_12:Lab VPC - default security group managed by AWS
  count = var.enable_aws ? 1 : 0

  cidr_block           = var.aws_vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = merge(local.tags, {
    Name = "${var.component}-aws-vpc"
  })
}

resource "aws_internet_gateway" "aws" {
  count = var.enable_aws ? 1 : 0

  vpc_id = aws_vpc.aws[0].id

  tags = merge(local.tags, {
    Name = "${var.component}-aws-igw"
  })
}

# 3 Public SLO Subnets (10.150.1.0/24, 10.150.2.0/24, 10.150.3.0/24)
resource "aws_subnet" "public_slo" {
  count = var.enable_aws ? 3 : 0

  vpc_id                  = aws_vpc.aws[0].id
  cidr_block              = cidrsubnet(var.aws_vpc_cidr, 8, count.index + 1)
  availability_zone       = try(data.aws_availability_zones.available[0].names[count.index], "${var.aws_location}${element(["a", "b", "c"], count.index)}")
  map_public_ip_on_launch = true

  tags = merge(local.tags, {
    Name = "${var.component}-aws-slo-subnet-${count.index + 1}"
  })
}

# 3 Private SLI Subnets (10.150.11.0/24, 10.150.12.0/24, 10.150.13.0/24)
resource "aws_subnet" "private_sli" {
  count = var.enable_aws ? 3 : 0

  vpc_id            = aws_vpc.aws[0].id
  cidr_block        = cidrsubnet(var.aws_vpc_cidr, 8, count.index + 11)
  availability_zone = try(data.aws_availability_zones.available[0].names[count.index], "${var.aws_location}${element(["a", "b", "c"], count.index)}")

  tags = merge(local.tags, {
    Name = "${var.component}-aws-sli-subnet-${count.index + 1}"
  })
}

resource "aws_route_table" "public" {
  count = var.enable_aws ? 1 : 0

  vpc_id = aws_vpc.aws[0].id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.aws[0].id
  }

  tags = merge(local.tags, {
    Name = "${var.component}-aws-public-rt"
  })
}

resource "aws_route_table_association" "public" {
  count = var.enable_aws ? 3 : 0

  subnet_id      = aws_subnet.public_slo[count.index].id
  route_table_id = aws_route_table.public[0].id
}

resource "aws_route_table" "private" {
  count = var.enable_aws ? 1 : 0

  vpc_id = aws_vpc.aws[0].id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.aws[0].id
  }

  tags = merge(local.tags, {
    Name = "${var.component}-aws-private-rt"
  })
}

resource "aws_route_table_association" "private" {
  count = var.enable_aws ? 3 : 0

  subnet_id      = aws_subnet.private_sli[count.index].id
  route_table_id = aws_route_table.private[0].id
}

resource "aws_security_group" "ce" {
  count = var.enable_aws ? 1 : 0

  name        = "${var.component}-aws-ce-sg"
  description = "Security group for F5 XC Customer Edge nodes in AWS"
  vpc_id      = aws_vpc.aws[0].id

  ingress {
    description = "SSH access"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "Site Console Local UI"
    from_port   = 65500
    to_port     = 65500
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "BGP peering"
    from_port   = 179
    to_port     = 179
    protocol    = "tcp"
    cidr_blocks = [var.aws_vpc_cidr]
  }

  ingress {
    description = "ICMP"
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = [var.aws_vpc_cidr]
  }

  ingress {
    description = "Intra-cluster communication"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    self        = true
  }

  egress {
    description = "Allow all outbound traffic"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = merge(local.tags, {
    Name = "${var.component}-aws-ce-sg"
  })
}

terraform/aws_ce.tf

AWS Customer Edge EC2 nodes, IAM role/profile, key pair, dual network interfaces, Elastic IPs, cloud-init user_data, site registration lookup, and automatic registration approval.

terraform/aws_ce.tf

# ---------------------------------------------------------
# AWS Customer Edge (EC2 instances, IAM, dual-NIC, registration)
# ---------------------------------------------------------

data "aws_ami" "f5_xc" {
  count       = var.enable_aws ? 1 : 0
  most_recent = true
  owners      = ["679593333241", "434481986642", "aws-marketplace"]

  filter {
    name   = "name"
    values = ["f5xc-ce-*", "f5-xc-*", "f5-volterra-*"]
  }
}

resource "aws_key_pair" "ce" {
  count      = var.enable_aws ? 1 : 0
  key_name   = "${var.component}-aws-ce-key"
  public_key = local.ssh_public_key

  tags = local.tags
}

resource "aws_iam_role" "ce" {
  count = var.enable_aws ? 1 : 0
  name  = "${var.component}-aws-ce-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "ec2.amazonaws.com"
      }
    }]
  })

  tags = local.tags
}

resource "aws_iam_role_policy" "ce" {
  count = var.enable_aws ? 1 : 0
  name  = "${var.component}-aws-ce-policy"
  role  = aws_iam_role.ce[0].id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = [
        "ec2:DescribeInstances",
        "ec2:DescribeRouteTables",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs"
      ]
      Effect   = "Allow"
      Resource = "*"
    }]
  })
}

resource "aws_iam_instance_profile" "ce" {
  count = var.enable_aws ? 1 : 0
  name  = "${var.component}-aws-ce-profile"
  role  = aws_iam_role.ce[0].id

  tags = local.tags
}

# Dual NICs per Customer Edge node
resource "aws_network_interface" "slo" {
  count = var.enable_aws ? var.aws_ce_count : 0

  subnet_id         = aws_subnet.public_slo[count.index % 3].id
  security_groups   = [aws_security_group.ce[0].id]
  source_dest_check = false

  tags = merge(local.tags, {
    Name = "${var.component}-aws-ce-${count.index + 1}-slo"
  })
}

resource "aws_network_interface" "sli" {
  count = var.enable_aws ? var.aws_ce_count : 0

  subnet_id         = aws_subnet.private_sli[count.index % 3].id
  security_groups   = [aws_security_group.ce[0].id]
  source_dest_check = false

  tags = merge(local.tags, {
    Name = "${var.component}-aws-ce-${count.index + 1}-sli"
  })
}

resource "aws_eip" "ce" {
  #checkov:skip=CKV2_AWS_19:Lab Elastic IP - attached to SLO network interface on CE instance
  count = var.enable_aws ? var.aws_ce_count : 0

  domain            = "vpc"
  network_interface = aws_network_interface.slo[count.index].id

  tags = merge(local.tags, {
    Name = "${var.component}-aws-ce-${count.index + 1}-eip"
  })
}

resource "aws_instance" "ce" {
  count = var.enable_aws ? var.aws_ce_count : 0

  ami                  = data.aws_ami.f5_xc[0].id
  instance_type        = var.aws_instance_type
  iam_instance_profile = aws_iam_instance_profile.ce[0].name
  key_name             = aws_key_pair.ce[0].key_name

  network_interface {
    network_interface_id = aws_network_interface.slo[count.index].id
    device_index         = 0
  }

  network_interface {
    network_interface_id = aws_network_interface.sli[count.index].id
    device_index         = 1
  }

  user_data = <<-EOF
    #cloud-config
    hostname: ${var.component}-aws-ce-${count.index + 1}
    fqdn: ${var.component}-aws-ce-${count.index + 1}.us-east-2.compute.internal
    write_files:
      - path: /etc/vpm/config.yaml
        permissions: '0644'
        content: |
          Vpm:
            ClusterName: aws-site
            ClusterHeader: ""
            Token: ${local.ce_registration_token}
            Latitude: 0
            Longitude: 0
            CertifiedHardwareEndpoint: https://vesio.blob.core.windows.net/releases/certified-hardware/aws.yml
    ssh_authorized_keys:
      - ${chomp(local.ssh_public_key)}
  EOF

  tags = merge(local.tags, {
    Name                             = "${var.component}-aws-ce-${count.index + 1}"
    "ves-io-site-name"               = "aws-site"
    "kubernetes.io/cluster/aws-site" = "owned"
  })
}

# Site registration lookup & automatic approval
data "xcsh_site_registration" "aws" {
  count     = var.enable_aws ? var.aws_ce_count : 0
  site_name = "aws-site"
  namespace = "system"
}

resource "xcsh_registration_approval" "aws" {
  count     = var.enable_aws && length(data.xcsh_site_registration.aws) > 0 && try(data.xcsh_site_registration.aws[0].found, false) ? var.aws_ce_count : 0
  namespace = "system"
  name      = data.xcsh_site_registration.aws[count.index].name
  state     = "APPROVED"
}

terraform/aws_xc.tf

AWS SecureMesh v2 site (xcsh_securemesh_site_v2.aws), eBGP peering (xcsh_bgp.aws_ebgp), virtual site (xcsh_virtual_site.aws), origin pool (xcsh_origin_pool.aws), and HTTP load balancer (xcsh_http_loadbalancer.aws).

terraform/aws_xc.tf

# ---------------------------------------------------------
# F5 XC SecureMesh v2 Site, BGP, Virtual Site, Origin Pool & LB for AWS
# ---------------------------------------------------------

resource "xcsh_securemesh_site_v2" "aws" {
  count       = var.enable_aws ? 1 : 0
  name        = "aws-site"
  namespace   = "system"
  description = "AWS Customer Edge SecureMesh Site v2"

  aws {
    not_managed {}
  }

  disable_ha {}
  block_all_services {}
  no_network_policy {}
  no_forward_proxy {}
  f5_proxy {}
  no_proxy_bypass {}
  logs_streaming_disabled {}
  no_s2s_connectivity_sli {}
  no_s2s_connectivity_slo {}
  disable_url_categorization {}
  disable_management_network {}
}

resource "xcsh_bgp" "aws_ebgp" {
  count     = var.enable_aws ? 1 : 0
  name      = "${var.component}-aws-ebgp"
  namespace = "system"

  where {
    site {
      network_type = "VIRTUAL_NETWORK_SITE_LOCAL"
      ref {
        name      = xcsh_securemesh_site_v2.aws[0].name
        namespace = "system"
      }
      disable_internet_vip {}
    }
  }

  bgp_parameters {
    asn = 64512
    local_address {}
  }

  peers {
    metadata {
      name = "peer-aws-router"
    }
    external {
      asn     = 65515
      address = cidrhost(var.aws_vpc_cidr, 1)
      port    = 179

      interface {
        name      = "eth0"
        namespace = "system"
      }

      disable_v6 {}
    }
    passive_mode_disabled {}
    bfd_disabled {}
  }
}

resource "xcsh_virtual_site" "aws" {
  count     = var.enable_aws ? 1 : 0
  name      = "${var.component}-aws-vsite"
  namespace = data.xcsh_namespace.mcn.name

  site_type = "CUSTOMER_EDGE"
  site_selector {
    expressions = ["ves.io/siteName in (aws-site)"]
  }
}

resource "xcsh_origin_pool" "aws" {
  count       = var.enable_aws ? 1 : 0
  name        = "${var.component}-aws-pool"
  namespace   = data.xcsh_namespace.mcn.name
  description = "AWS origin pool serving MCN CE-HA demo"

  port = var.origin_port

  origin_servers {
    labels {}
    public_ip {
      ip = var.origin_ip
    }
  }

  no_tls {}
  loadbalancer_algorithm = "ROUND_ROBIN"
  endpoint_selection     = "DISTRIBUTED"
}

resource "xcsh_http_loadbalancer" "aws" {
  count     = var.enable_aws ? 1 : 0
  name      = "${var.component}-aws-lb"
  namespace = data.xcsh_namespace.mcn.name

  domains = [var.aws_lb_domain]

  http {
    port = 80
  }

  advertise_custom {
    advertise_where {
      virtual_site {
        network = "SITE_NETWORK_INSIDE_AND_OUTSIDE"
        virtual_site {
          name      = xcsh_virtual_site.aws[0].name
          namespace = data.xcsh_namespace.mcn.name
        }
      }
      use_default_port {}
    }
  }

  default_route_pools {
    pool {
      name      = xcsh_origin_pool.aws[0].name
      namespace = data.xcsh_namespace.mcn.name
    }
    weight   = 1
    priority = 1
  }

  round_robin {}
  no_challenge {}
  user_id_client_ip {}
  disable_waf {}
  disable_rate_limit {}
  disable_api_discovery {}
  disable_api_testing {}
  disable_api_definition {}
  l7_ddos_protection {}
}

terraform/variables_azure.tf

Subscription, region, CIDRs, Bastion, Canadian ILB variant option, and the Azure object names.

terraform/variables_azure.tf

# ---------------------------------------------------------
# Azure subscription / placement
# ---------------------------------------------------------

variable "subscription_id" {
  description = "Azure subscription ID used by the azurerm provider."
  type        = string
  # Default = the <AZURE_SUBSCRIPTION_ID> subscription the lab was built in.
  default = "00000000-0000-0000-0000-000000000000"
}

variable "resource_group_name" {
  description = "Resource group that holds the hub VNet, Route Server, CE VMs and test client. Leave null (the default) to derive `rg-<component>-<deployer>`, where the deployer is resolved from Azure AD — so a fresh clone names the group after whoever deploys it rather than after whoever wrote it."
  type        = string
  default     = null
}

variable "location" {
  description = "Azure region for all resources."
  type        = string
  default     = "eastus"
}

# ---------------------------------------------------------
# Canada regional Azure placement & CIDRs
# ---------------------------------------------------------

variable "enable_canada_ilb" {
  description = "Deploy Azure Internal Load Balancer (ILB) in front of Canadian CEs as an alternative to Azure Route Server (eBGP/ECMP)."
  type        = bool
  default     = true
}

variable "ca_location" {
  description = "Azure region for Canadian regional resources."
  type        = string
  default     = "canadacentral"
}

variable "ca_resource_group_name" {
  description = "Resource group that holds the Canadian hub VNet, Route Server, CE VMs and test client. Leave null (the default) to derive `rg-<component>-ca-<deployer>`."
  type        = string
  default     = null
}

variable "ca_hub_cidr" {
  description = "Canada Hub VNet address space."
  type        = string
  default     = "10.200.0.0/16"
}

variable "ca_mgmt_subnet_prefix" {
  description = "Canada snet-hub-management prefix. CE eth0/SLO NICs live here."
  type        = string
  default     = "10.200.1.0/26"
}

variable "ca_external_subnet_prefix" {
  description = "Canada snet-hub-external prefix."
  type        = string
  default     = "10.200.2.0/26"
}

variable "ca_internal_subnet_prefix" {
  description = "Canada snet-hub-internal prefix. Test client lives here."
  type        = string
  default     = "10.200.3.0/26"
}

variable "ca_route_server_subnet_prefix" {
  description = "Canada RouteServerSubnet prefix. MUST be exactly /27."
  type        = string
  default     = "10.200.4.0/27"

  validation {
    condition     = tonumber(split("/", var.ca_route_server_subnet_prefix)[1]) == 27
    error_message = "ca_route_server_subnet_prefix must be a /27."
  }
}

variable "ca_bastion_subnet_prefix" {
  description = "Canada AzureBastionSubnet prefix. MUST be /26 or larger."
  type        = string
  default     = "10.200.5.0/26"

  validation {
    condition     = tonumber(split("/", var.ca_bastion_subnet_prefix)[1]) <= 26
    error_message = "ca_bastion_subnet_prefix must be /26 or larger."
  }
}

variable "ca_vip" {
  description = "HA VIP for Canadian CEs advertised as a /32 via eBGP."
  type        = string
  default     = "10.250.1.10"

  validation {
    condition     = can(cidrhost("${var.ca_vip}/32", 0))
    error_message = "ca_vip must be a valid IPv4 address."
  }
}

variable "ca_route_server_name" {
  description = "Canada Azure Route Server name. Leave null to derive `<component>-ca-rs`."
  type        = string
  default     = null
}

variable "ca_bastion_name" {
  description = "Canada Azure Bastion host name. Leave null to derive `<component>-ca-bastion`."
  type        = string
  default     = null
}

variable "ca_client_vm_name" {
  description = "Canada test client VM name. Leave null to derive `<component>-ca-client`."
  type        = string
  default     = null
}

variable "ca_region_short" {
  description = "Short region token for Canada site names. Leave null to use var.ca_location."
  type        = string
  default     = null
}

# ---------------------------------------------------------
# Network CIDRs
# ---------------------------------------------------------

variable "hub_cidr" {
  description = "Hub VNet address space."
  type        = string
  default     = "10.0.0.0/16"
}

variable "spoke_cidr" {
  description = "Spoke VNet address space (not deployed here; used only to assert the VIP is outside it)."
  type        = string
  default     = "10.1.0.0/16"
}

variable "mgmt_subnet_prefix" {
  description = "snet-hub-management prefix. CE eth0/SLO NICs (BGP local address) live here."
  type        = string
  default     = "10.0.1.0/26"
}

variable "external_subnet_prefix" {
  description = "snet-hub-external prefix."
  type        = string
  default     = "10.0.2.0/26"
}

variable "internal_subnet_prefix" {
  description = "snet-hub-internal prefix. The test client lives here."
  type        = string
  default     = "10.0.3.0/26"
}

variable "route_server_subnet_prefix" {
  description = "RouteServerSubnet prefix. MUST be exactly /27, named literally RouteServerSubnet, with no NSG or route table."
  type        = string
  default     = "10.0.4.0/27"

  validation {
    condition     = tonumber(split("/", var.route_server_subnet_prefix)[1]) == 27
    error_message = "RouteServerSubnet must be a /27."
  }
}

variable "route_server_name" {
  description = "Azure Route Server name. Leave null (the default) to derive `<component>-rs`."
  type        = string
  default     = null
}

variable "bastion_subnet_prefix" {
  description = "AzureBastionSubnet prefix. MUST be /26 or larger, named literally AzureBastionSubnet, with no NSG or route table. 10.0.5.0/26 is the first free /26 in the hub (10.0.1.0/26 mgmt, 10.0.2.0/26 external, 10.0.3.0/26 internal, 10.0.4.0/27 RouteServerSubnet are taken)."
  type        = string
  default     = "10.0.5.0/26"

  validation {
    # Azure rejects anything smaller than /26 at APPLY time, long after the plan
    # looked healthy. Fail at plan time instead.
    condition     = tonumber(split("/", var.bastion_subnet_prefix)[1]) <= 26
    error_message = "AzureBastionSubnet must be /26 or larger (a smaller prefix, e.g. /27, is rejected by Azure)."
  }
}

# --------------------------------------------------------------------------
# Azure Bastion
# --------------------------------------------------------------------------
# Default false, deliberately. Bastion is the only resource in this deployment
# that is not load-bearing for the BGP/ECMP demo — the data path, the sites and
# the VIP all work without it — yet a Standard-SKU host bills an hourly standing
# charge from the moment it is provisioned, whether or not anyone opens a tunnel:
# USD 0.29/hour in eastus for the two included scale units (retail, 2026-07),
# roughly USD 212 a month, plus USD 0.14/hour for each scale unit beyond two.
# Opting in keeps the cost of a default `terraform apply` exactly what it is
# today and makes the spend a conscious choice by whoever needs CE console
# access. Turn it on with `enable_bastion = true` in terraform.tfvars.
variable "enable_bastion" {
  description = "Deploy Azure Bastion so developers can reach each CE's Site Console web UI (https://<sli-ip>:65500) with Azure RBAC instead of an SSH key and a jump host."
  type        = bool
  default     = false
}

variable "client_vm_name" {
  description = "Test client VM name, and the stem of its public IP, NSG and NIC. Leave null (the default) to derive `<component>-client`. Set it only to hold an existing client VM steady: the name is also its computer_name, which forces a VM replacement when it changes."
  type        = string
  default     = null
}

variable "bastion_name" {
  description = "Azure Bastion host name (the --name argument of `az network bastion tunnel`). Leave null (the default) to derive `<component>-bastion`."
  type        = string
  default     = null
}

# ---------------------------------------------------------
# CE / client VM inputs
# ---------------------------------------------------------

variable "ce_vm_size" {
  description = "VM size for the Customer Edge nodes."
  type        = string
  default     = "Standard_D8_v4"
}

variable "admin_username" {
  description = "SSH admin username for the VMs."
  type        = string
  default     = "azureuser"
}

variable "ssh_public_key" {
  description = "SSH public key MATERIAL (the key string). When empty, read from ssh_public_key_path. Passing material keeps plan tests hermetic."
  type        = string
  default     = ""
}

variable "ssh_public_key_path" {
  description = "Path to the SSH public key file, read once at the root when ssh_public_key is empty."
  type        = string
  default     = "~/.ssh/id_ed25519.pub"
}

terraform/ca_ilb.tf

Azure Internal Load Balancer (ILB) variant configuration for the Canadian regional path (enable_canada_ilb).

terraform/ca_ilb.tf

# ---------------------------------------------------------
# Canadian Regional Extension: Azure Internal Load Balancer (ILB)
# ---------------------------------------------------------

resource "azurerm_lb" "ca_ilb" {
  #checkov:skip=CKV_AZURE_27:Lab environment ILB - diagnostic logging not required
  count               = var.enable_canada && var.enable_canada_ilb ? 1 : 0
  name                = "${var.component}-ca-ilb"
  location            = var.ca_location
  resource_group_name = module.azure_hub_ca[0].resource_group_name
  sku                 = "Standard"

  frontend_ip_configuration {
    name                          = "ca-ilb-frontend"
    subnet_id                     = module.azure_hub_ca[0].management_subnet_id
    private_ip_address            = cidrhost(var.ca_mgmt_subnet_prefix, 10)
    private_ip_address_allocation = "Static"
  }

  tags = local.tags
}

resource "azurerm_lb_backend_address_pool" "ca_ce_backend" {
  count           = var.enable_canada && var.enable_canada_ilb ? 1 : 0
  name            = "ca-ce-backend-pool"
  loadbalancer_id = azurerm_lb.ca_ilb[0].id
}

resource "azurerm_network_interface_backend_address_pool_association" "ca_ce" {
  for_each = var.enable_canada && var.enable_canada_ilb ? module.ce_topology_ca[0].ce_nodes : {}

  network_interface_id    = module.ce_node_ca[each.key].mgmt_nic_id
  ip_configuration_name   = "ipconfig1"
  backend_address_pool_id = azurerm_lb_backend_address_pool.ca_ce_backend[0].id
}

resource "azurerm_lb_probe" "ca_site_console" {
  count               = var.enable_canada && var.enable_canada_ilb ? 1 : 0
  name                = "site-console-probe"
  loadbalancer_id     = azurerm_lb.ca_ilb[0].id
  protocol            = "Tcp"
  port                = 65500
  interval_in_seconds = 5
  number_of_probes    = 2
}

resource "azurerm_lb_rule" "ca_ha_ports" {
  count                          = var.enable_canada && var.enable_canada_ilb ? 1 : 0
  name                           = "ha-ports-rule"
  loadbalancer_id                = azurerm_lb.ca_ilb[0].id
  frontend_ip_configuration_name = "ca-ilb-frontend"
  backend_address_pool_ids       = [azurerm_lb_backend_address_pool.ca_ce_backend[0].id]
  probe_id                       = azurerm_lb_probe.ca_site_console[0].id
  protocol                       = "All"
  frontend_port                  = 0
  backend_port                   = 0
  floating_ip_enabled            = true
  idle_timeout_in_minutes        = 4
}

terraform/variables_ce.tf

CE count, site prefix, ASNs, image versions and VM size.

terraform/variables_ce.tf

# ---------------------------------------------------------
# CE topology / scaling
# ---------------------------------------------------------

variable "ce_count" {
  description = "Number of single-node Secure Mesh v2 CE sites (1..3). Each originates the LB VIP via eBGP to the Azure Route Server for active/active ECMP."
  type        = number
  default     = 3

  validation {
    condition     = var.ce_count >= 1 && var.ce_count <= 3
    error_message = "ce_count must be between 1 and 3."
  }
}

# ---------------------------------------------------------
# Canada CE topology / scaling
# ---------------------------------------------------------

variable "ca_ce_count" {
  description = "Number of Canadian single-node Secure Mesh v2 CE sites (1..3)."
  type        = number
  default     = 3

  validation {
    condition     = var.ca_ce_count >= 1 && var.ca_ce_count <= 3
    error_message = "ca_ce_count must be between 1 and 3."
  }
}

variable "ca_site_prefix" {
  description = "Prefix for Canadian XC site names (site = <ca_site_prefix>-<ca_region_short>0<n>). Leave null to derive `<component>-ca`."
  type        = string
  default     = null
}

variable "region_short" {
  description = "Short region token used to build the per-CE key and site name (component `mcn-ce-ha` + region `eastus` -> site `mcn-ce-ha-eastus01`). Leave null (the default) to use var.location verbatim; set it only when the Azure region name is longer than you want inside object names."
  type        = string
  default     = null
}

variable "site_prefix" {
  description = "Prefix for XC site names (site = <prefix>-<region_short>0<n>). Leave null (the default) to use var.component, which is what keeps every object name descending from one value. Set it only to hold existing site names steady: renaming a site replaces the CE VM, because the site name is the ClusterName baked into cloud-init and cloud-init only runs on first boot."
  type        = string
  default     = null
}

variable "ce_asn" {
  description = "BGP ASN for the Customer Edge nodes (eBGP local ASN)."
  type        = number
  default     = 64512
}

variable "rs_asn" {
  description = "BGP ASN of the Azure Route Server (fixed by Azure at 65515)."
  type        = number
  default     = 65515
}

variable "enable_bgp" {
  description = "Create the per-CE xcsh_bgp objects. Defaults true — BGP/ECMP is the point of this deployment — and nothing gates it: the object-ref name length blocker that once forced it false was removed in provider v3.74.0. Retained only as an escape hatch for planning or deploying the topology without BGP."
  type        = bool
  default     = true
}

variable "approve_registration" {
  description = "Approve each CE's runtime registration from Terraform (the xcsh_site_registration data source resolves the r-<uuid> registration name from the site name, and xcsh_registration_approval approves it). Two-phase by nature: the registration only exists after the CE has booted and registered, so the first apply plans no approval and a later apply creates it. Set false to leave approval to an operator."
  type        = bool
  default     = true
}

# ---------------------------------------------------------
# Site registration token (optional override)
# ---------------------------------------------------------
# The CE VM cloud-init consumes a tenant-scoped, reusable site registration
# token (ClusterName + Token). The token is now provider-generated by the
# xcsh_token.ce resource (its Computed uid); this var is an optional override.
variable "registration_token" {
  description = "Optional override for the CE cloud-init site registration token. When empty (the default) the provider-generated xcsh_token.ce.uid is used; set a non-empty value to inject an externally-minted token instead."
  type        = string
  default     = ""
  sensitive   = true
}

variable "ce_os_version" {
  description = "CE OS version, set at create time. Empty deliberately selects the newest version advertised by F5 Distributed Cloud; the clean 2026-08-03 rebuild verified that policy on the 64 GB CE disk default. A concrete value is for reproducing an older build. Terraform cannot change this field afterwards: updates are rejected in every direction, including un-pinning. The platform can change it in place through the site upgrade_os action, but the provider cannot drive that action yet (xcsh#1390)."
  type        = string
  default     = ""
}

variable "ce_sw_version" {
  description = "CE F5 Distributed Cloud software version, set at create time. Empty deliberately selects the newest advertised build; the clean 2026-08-03 rebuild verified that policy on the 64 GB CE disk default. A concrete value is for reproducing an older build. The node always installs a destination build on first boot. Terraform cannot change this field afterwards, but the platform can change it in place through the site upgrade_sw action (xcsh#1390)."
  type        = string
  default     = ""
}

terraform/variables_xc.tf

Tenant, app namespace, load balancer, origin pool and the VIP.

terraform/variables_xc.tf

# ---------------------------------------------------------
# F5 XC tenant
# ---------------------------------------------------------

variable "expected_xc_tenant" {
  description = "F5 XC tenant this deployment belongs to: the first hostname label of the console URL (`f5-sales-demo` for https://f5-sales-demo.console.ves.volterra.io). This is the ONLY place the tenant is named. The xcsh provider's api_url is derived from it, so the configuration — not the ambient environment — decides which tenant is written to, and a plan FAILS when XCSH_API_URL in the environment names a different one."
  type        = string
  default     = "f5-sales-demo"

  validation {
    # A hostname label, not a URL: the scheme and domain are added in locals.tf.
    condition     = can(regex("^[a-z0-9]([a-z0-9-]*[a-z0-9])?$", var.expected_xc_tenant))
    error_message = "expected_xc_tenant must be a bare DNS label (lowercase letters, digits and hyphens) such as f5-sales-demo — not a URL and not a hostname."
  }
}

# ---------------------------------------------------------
# F5 XC data-plane inputs
# ---------------------------------------------------------

variable "xc_app_namespace" {
  description = "Name of a PRE-EXISTING F5 XC namespace in expected_xc_tenant to place the app-tier objects (origin pool + HTTP load balancer) in. This deployment READS the namespace; it never creates or deletes it, so a namespace holding unrelated demos can never land on this stack's destroy list."
  type        = string
  default     = "multi-cloud-networking"

  validation {
    condition     = length(var.xc_app_namespace) > 0
    error_message = "xc_app_namespace must name an existing namespace; it is looked up, not created."
  }
}

variable "origin_pool_name" {
  description = "Name of the origin pool. Leave null (the default) to derive `<component>-pool`."
  type        = string
  default     = null
}

variable "origin_ip" {
  description = "Public IP of the origin server the pool targets. REQUIRED, deliberately without a default: any default here is one specific machine, and a wrong one silently sends a fresh deployment's traffic to somebody else's host instead of failing."
  type        = string

  validation {
    condition     = can(cidrhost("${var.origin_ip}/32", 0))
    error_message = "origin_ip must be a single IPv4 address."
  }
}

variable "origin_port" {
  description = "TCP port of the origin server."
  type        = number
  default     = 80
}

variable "lb_name" {
  description = "Name of the HTTP load balancer. Leave null (the default) to derive `<component>-f5se`, matching the naming this tenant's other load balancers use."
  type        = string
  default     = null
}

variable "lb_domain" {
  description = "Domain served by the HTTP load balancer for Rest of World. REQUIRED, deliberately without a default: the load balancer matches on Host, so this value is what every request must send, and it belongs to whoever runs the deployment."
  type        = string

  validation {
    condition     = can(regex("^([a-z0-9]([a-z0-9-]*[a-z0-9])?\\.)+[a-z]{2,}$", var.lb_domain))
    error_message = "lb_domain must be a fully-qualified lowercase domain name (for example mcn-ce-ha.f5-sales-demo.com)."
  }
}

# ---------------------------------------------------------
# Canada Regional F5 XC inputs
# ---------------------------------------------------------

variable "enable_canada" {
  description = "Enable parallel Canada-only regional infrastructure, Canadian virtual sites, and f5-sales-demo.ca load balancer."
  type        = bool
  default     = true
}

variable "ca_lb_domain" {
  description = "Domain served by the Canada HTTP load balancer. Defaults to mcn-ce-ha.f5-sales-demo.ca."
  type        = string
  default     = "mcn-ce-ha.f5-sales-demo.ca"

  validation {
    condition     = can(regex("^([a-z0-9]([a-z0-9-]*[a-z0-9])?\\.)+[a-z]{2,}$", var.ca_lb_domain))
    error_message = "ca_lb_domain must be a fully-qualified lowercase domain name (for example mcn-ce-ha.f5-sales-demo.ca)."
  }
}

variable "ca_re_cities" {
  description = "List of cities for the Canadian Regional Edge Virtual Site selector. Defaults to Toronto and Montreal."
  type        = list(string)
  default     = ["toronto", "montreal"]
}

variable "ca_lb_name" {
  description = "Name of the Canada HTTP load balancer. Leave null (the default) to derive `<component>-ca-f5se`."
  type        = string
  default     = null
}

variable "ca_origin_pool_name" {
  description = "Name of the Canada origin pool. Leave null (the default) to derive `<component>-ca-pool`."
  type        = string
  default     = null
}

variable "ca_re_vsite_name" {
  description = "Name of the Canadian Regional Edge virtual site. Leave null (the default) to derive `<component>-ca-re-vsite`."
  type        = string
  default     = null
}

variable "ca_ce_vsite_name" {
  description = "Name of the Canadian Customer Edge virtual site. Leave null (the default) to derive `<component>-ca-ce-vsite`."
  type        = string
  default     = null
}

variable "vip" {
  description = "HA VIP advertised as a /32 by every CE via eBGP. MUST be outside all VNet CIDRs (Azure prefers the VNet system route over a more-specific BGP /32 otherwise)."
  type        = string
  default     = "10.250.0.10"

  validation {
    # Self-contained format check. Cross-CIDR containment is asserted by the
    # check{} block in main.tf (which may reference other variables).
    condition     = can(cidrhost("${var.vip}/32", 0))
    error_message = "vip must be a valid IPv4 address."
  }
}

terraform/versions.tf

Terraform floor and provider constraints. The xcsh constraint is an open-ended floor, and the lock file is gitignored, so every init takes the latest published provider.

terraform/versions.tf

terraform {
  # >= 1.10.0 for provider-defined functions, check{} blocks, and test framework
  # mocking options (override_during = plan in .tftest.hcl).
  required_version = ">= 1.10.0"

  required_providers {
    xcsh = {
      source  = "f5-sales-demo/xcsh"
      version = ">= 3.81.1"
    }
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    azuread = {
      source  = "hashicorp/azuread"
      version = "~> 3.0"
    }
    external = {
      source  = "hashicorp/external"
      version = "~> 2.3"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.0"
    }
    libvirt = {
      source  = "dmacvicar/libvirt"
      version = "~> 0.8.0"
    }
  }
}

Modules

terraform/modules/azure-hub/

Resource group, hub VNet, the four subnets, the Azure Route Server and the optional Bastion. RouteServerSubnet deliberately carries no NSG or route table.

terraform/modules/azure-hub/main.tf

resource "azurerm_resource_group" "this" {
  name     = var.resource_group_name
  location = var.location
  tags     = var.tags
}

resource "azurerm_virtual_network" "hub" {
  name                = "hub-vnet"
  address_space       = [var.hub_cidr]
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  tags                = var.tags
}

resource "azurerm_subnet" "management" {
  name                 = "snet-hub-management"
  resource_group_name  = azurerm_resource_group.this.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [var.mgmt_subnet_prefix]
}

resource "azurerm_subnet" "external" {
  name                 = "snet-hub-external"
  resource_group_name  = azurerm_resource_group.this.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [var.external_subnet_prefix]
}

resource "azurerm_subnet" "internal" {
  name                 = "snet-hub-internal"
  resource_group_name  = azurerm_resource_group.this.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [var.internal_subnet_prefix]
}

# RouteServerSubnet: name is literal, /27, and has NO NSG and NO route table
# association (both are unsupported on the Route Server subnet).
resource "azurerm_subnet" "route_server" {
  name                 = "RouteServerSubnet"
  resource_group_name  = azurerm_resource_group.this.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [var.route_server_subnet_prefix]
}

resource "azurerm_public_ip" "route_server" {
  name                = "${var.route_server_name}-pip"
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  allocation_method   = "Static"
  sku                 = "Standard"
  tags                = var.tags
}

# Azure Route Server. ASN is fixed by Azure at 65515; virtual_router_ips are the
# two BGP peer addresses (10.0.4.4 / 10.0.4.5) the CEs peer to.
resource "azurerm_route_server" "this" {
  name                 = var.route_server_name
  location             = azurerm_resource_group.this.location
  resource_group_name  = azurerm_resource_group.this.name
  sku                  = "Standard"
  public_ip_address_id = azurerm_public_ip.route_server.id
  subnet_id            = azurerm_subnet.route_server.id
  tags                 = var.tags
}

# --------------------------------------------------------------------------
# Azure Bastion — reachability for the CE Site Console web UI (TCP 65500)
# --------------------------------------------------------------------------
# F5's documented CE troubleshooting path is the per-node Site Console at
# https://<node-ip>:65500 as `admin`. Verified on this deployment: of a CE's three
# private addresses only the SLI one (snet-hub-internal) answers at all — mgmt/SLO
# and external are closed on every port probed — and it is reachable only from
# inside the VNet. Bastion closes the gap without distributing SSH keys and
# without touching the CE: who may connect becomes an Azure RBAC decision.
#
# AzureBastionSubnet: name is literal, /26 or larger, and — like
# RouteServerSubnet above — deliberately carries NO NSG and NO route table.
# Bastion works without an NSG; adding one obliges you to maintain Azure's exact
# required allow-rule set (GatewayManager, AzureLoadBalancer, the 8080/5701
# intra-subnet pair) for no security gain in a lab whose VMs already sit behind
# their own NSGs.
resource "azurerm_subnet" "bastion" {
  count = var.enable_bastion ? 1 : 0

  name                 = "AzureBastionSubnet"
  resource_group_name  = azurerm_resource_group.this.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [var.bastion_subnet_prefix]
}

resource "azurerm_public_ip" "bastion" {
  count = var.enable_bastion ? 1 : 0

  name                = "${var.bastion_name}-pip"
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  allocation_method   = "Static"
  sku                 = "Standard"
  tags                = var.tags
}

# Standard SKU is not a preference — Basic supports neither native-client
# tunneling nor IP-based connection.
#
#   tunneling_enabled  -> load-bearing. `az network bastion tunnel` forwards an
#                         arbitrary TCP port, which is the only way to reach 65500.
#   ip_connect_enabled -> NOT the path to 65500, and deliberately not claimed to
#                         be. Microsoft documents that "custom ports and protocols
#                         aren't currently supported when connecting to a virtual
#                         machine via native client with IP-based connections"
#                         (learn.microsoft.com/azure/bastion/connect-ip-address),
#                         and the CLI enforces it: --target-ip-address with
#                         --resource-port 65500 is refused with "Allowed ports for
#                         Tunnel with IP connect is 22, 3389". It is on because it
#                         costs nothing and lets an operator target an in-VNet IP
#                         directly for portal RDP/SSH. The 65500 tunnel targets the
#                         CE by VM resource id instead — verified reaching the Site
#                         Console (HTTP 200) on this deployment.
resource "azurerm_bastion_host" "this" {
  count = var.enable_bastion ? 1 : 0

  name                = var.bastion_name
  location            = azurerm_resource_group.this.location
  resource_group_name = azurerm_resource_group.this.name
  sku                 = "Standard"
  tunneling_enabled   = true
  ip_connect_enabled  = true

  ip_configuration {
    name                 = "configuration"
    subnet_id            = azurerm_subnet.bastion[0].id
    public_ip_address_id = azurerm_public_ip.bastion[0].id
  }

  tags = var.tags
}

terraform/modules/azure-hub/outputs.tf

output "resource_group_name" {
  description = "Name of the hub resource group."
  value       = azurerm_resource_group.this.name
}

output "location" {
  description = "Azure region."
  value       = azurerm_resource_group.this.location
}

output "vnet_name" {
  description = "Hub VNet name."
  value       = azurerm_virtual_network.hub.name
}

output "vnet_id" {
  description = "Hub VNet resource ID."
  value       = azurerm_virtual_network.hub.id
}

output "management_subnet_id" {
  description = "snet-hub-management subnet ID."
  value       = azurerm_subnet.management.id
}

output "external_subnet_id" {
  description = "snet-hub-external subnet ID."
  value       = azurerm_subnet.external.id
}

output "internal_subnet_id" {
  description = "snet-hub-internal subnet ID."
  value       = azurerm_subnet.internal.id
}

output "route_server_subnet_id" {
  description = "RouteServerSubnet subnet ID."
  value       = azurerm_subnet.route_server.id
}

output "route_server_id" {
  description = "Azure Route Server resource ID."
  value       = azurerm_route_server.this.id
}

output "rs_peer_ips" {
  description = "The two Route Server BGP peer IPs (virtual_router_ips) the CEs peer to."
  value       = azurerm_route_server.this.virtual_router_ips
}

output "rs_asn" {
  description = "Route Server ASN (fixed by Azure at 65515)."
  value       = azurerm_route_server.this.virtual_router_asn
}

output "bastion_name" {
  description = "Azure Bastion host name, or null when Bastion is not deployed. This is the --name argument of `az network bastion tunnel`."
  value       = one(azurerm_bastion_host.this[*].name)
}

output "bastion_subnet_id" {
  description = "AzureBastionSubnet subnet ID, or null when Bastion is not deployed."
  value       = one(azurerm_subnet.bastion[*].id)
}

output "bastion_subnet_prefix" {
  description = "The AzureBastionSubnet prefix this module was given (echoed so callers can assert on it without reading a count-gated resource)."
  value       = var.bastion_subnet_prefix
}

terraform/modules/azure-hub/variables.tf

variable "resource_group_name" {
  description = "Resource group to create for the hub."
  type        = string
}

variable "location" {
  description = "Azure region."
  type        = string
}

variable "hub_cidr" {
  description = "Hub VNet address space."
  type        = string
}

variable "mgmt_subnet_prefix" {
  description = "snet-hub-management prefix."
  type        = string
}

variable "external_subnet_prefix" {
  description = "snet-hub-external prefix."
  type        = string
}

variable "internal_subnet_prefix" {
  description = "snet-hub-internal prefix."
  type        = string
}

variable "route_server_subnet_prefix" {
  description = "RouteServerSubnet prefix (/27, no NSG, no route table)."
  type        = string
}

variable "route_server_name" {
  description = "Azure Route Server name."
  type        = string
  default     = "ce-ha-lab-rrs"
}

variable "bastion_subnet_prefix" {
  description = "AzureBastionSubnet prefix (/26 or larger, named literally AzureBastionSubnet, no NSG and no route table). Only read when enable_bastion is true."
  type        = string
}

variable "enable_bastion" {
  description = "Deploy Azure Bastion (AzureBastionSubnet + public IP + Standard-SKU host) so the CE Site Console web UI on TCP 65500 is reachable without a jump host. Opt-in: Bastion bills hourly whether or not anyone connects."
  type        = bool
  default     = false
}

variable "bastion_name" {
  description = "Azure Bastion host name (the --name argument of `az network bastion tunnel`)."
  type        = string
  default     = "ce-ha-lab-bastion"
}

variable "tags" {
  description = "Tags applied to every resource."
  type        = map(string)
  default     = {}
}

terraform/modules/azure-hub/versions.tf

terraform {
  required_version = ">= 1.8"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
}

terraform/modules/azure-route-server-bgp/

The Azure side of each eBGP session — one bgpConnection per CE, peering the Route Server to that CE eth0/SLO address.

terraform/modules/azure-route-server-bgp/main.tf

# The Azure side of the eBGP session: a Route Server BGP connection to a CE's
# eth0/SLO private IP. Peer ASN = the CE ASN (64512). The CE originates the LB
# VIP /32; equal-cost advertisements from multiple CEs program ECMP into the VNet.

variable "name" {
  description = "BGP connection name (e.g. eastus01-bgp)."
  type        = string
}

variable "route_server_id" {
  description = "Azure Route Server resource ID."
  type        = string
}

variable "peer_asn" {
  description = "Peer (CE) ASN."
  type        = number
  default     = 64512
}

variable "peer_ip" {
  description = "Peer (CE) eth0/SLO private IP."
  type        = string
}

resource "azurerm_route_server_bgp_connection" "this" {
  name            = var.name
  route_server_id = var.route_server_id
  peer_asn        = var.peer_asn
  peer_ip         = var.peer_ip
}

output "connection_id" {
  description = "Route Server BGP connection resource ID."
  value       = azurerm_route_server_bgp_connection.this.id
}

output "connection_name" {
  description = "Route Server BGP connection name."
  value       = azurerm_route_server_bgp_connection.this.name
}

terraform/modules/azure-route-server-bgp/versions.tf

terraform {
  required_version = ">= 1.8"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
}

terraform/modules/ce-node/

One CE VM per node: three NICs with IP forwarding, the marketplace plan block, and the cloud-init handed over as custom_data.

terraform/modules/ce-node/main.tf

# Per-CE Azure resources: managed identity, three NICs (mgmt/external/internal,
# all with IP forwarding on and accelerated networking OFF), and the volterra-node
# VM. The mgmt NIC is the VM's FIRST NIC = eth0 = the SLO/BGP local address.

resource "azurerm_user_assigned_identity" "this" {
  name                = "${var.hostname}-identity"
  resource_group_name = var.resource_group_name
  location            = var.location
  tags                = var.tags
}

resource "azurerm_public_ip" "mgmt" {
  name                = "${var.hostname}-mgmt-pip"
  resource_group_name = var.resource_group_name
  location            = var.location
  allocation_method   = "Static"
  sku                 = "Standard"
  tags                = var.tags
}

# eth0 / SLO — mgmt subnet, static private IP (the BGP local address), public IP.
resource "azurerm_network_interface" "mgmt" {
  name                           = "${var.hostname}-mgmt-nic"
  resource_group_name            = var.resource_group_name
  location                       = var.location
  ip_forwarding_enabled          = true
  accelerated_networking_enabled = false

  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = var.mgmt_subnet_id
    private_ip_address_allocation = "Static"
    private_ip_address            = var.mgmt_private_ip
    public_ip_address_id          = azurerm_public_ip.mgmt.id
  }

  tags = var.tags
}

resource "azurerm_network_interface" "external" {
  name                           = "${var.hostname}-external-nic"
  resource_group_name            = var.resource_group_name
  location                       = var.location
  ip_forwarding_enabled          = true
  accelerated_networking_enabled = false

  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = var.external_subnet_id
    private_ip_address_allocation = "Dynamic"
  }

  tags = var.tags
}

resource "azurerm_network_interface" "internal" {
  name                           = "${var.hostname}-internal-nic"
  resource_group_name            = var.resource_group_name
  location                       = var.location
  ip_forwarding_enabled          = true
  accelerated_networking_enabled = false

  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = var.internal_subnet_id
    private_ip_address_allocation = "Dynamic"
  }

  tags = var.tags
}

resource "azurerm_linux_virtual_machine" "this" {
  name                = var.hostname
  computer_name       = var.hostname
  resource_group_name = var.resource_group_name
  location            = var.location
  size                = var.vm_size
  zone                = var.zone

  admin_username                  = var.admin_username
  disable_password_authentication = true

  admin_ssh_key {
    username   = var.admin_username
    public_key = var.ssh_public_key
  }

  # eth0 first = mgmt/SLO NIC (BGP local + MAC-bound to the XC site interface).
  network_interface_ids = [
    azurerm_network_interface.mgmt.id,
    azurerm_network_interface.external.id,
    azurerm_network_interface.internal.id,
  ]

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.this.id]
  }

  os_disk {
    name                 = "${var.hostname}-osdisk"
    caching              = "ReadWrite"
    storage_account_type = "StandardSSD_LRS"

    # Explicit, because the image default is the one size measured to fail the
    # advertised version pair (#714). See variables.tf for the measurements.
    disk_size_gb = var.os_disk_size_gb
  }

  source_image_reference {
    publisher = "volterraedgeservices"
    offer     = "volterra-node"
    sku       = "volterra-node"
    version   = "latest"
  }

  # Marketplace plan is REQUIRED for the volterra-node image or VM create fails.
  plan {
    name      = "volterra-node"
    product   = "volterra-node"
    publisher = "volterraedgeservices"
  }

  custom_data = var.custom_data

  # Required for Azure Serial Console, which is the only way into a CE that has not
  # finished its first boot: the vpm/debug API used for every other diagnostic is
  # reached through the XC control plane, so it answers only once the node is ONLINE,
  # and operator SSH depends on cloud-init having written admin's authorized_keys and
  # on the SLI interface being up. Empty block = Azure-managed storage, so there is
  # no diagnostics storage account or access key to own.
  boot_diagnostics {}

  tags = var.tags
}

terraform/modules/ce-node/outputs.tf

output "mgmt_nic_id" {
  description = "Resource ID of the eth0/SLO (mgmt) NIC."
  value       = azurerm_network_interface.mgmt.id
}

output "mgmt_nic_mac" {
  description = "MAC address of the eth0/SLO (mgmt) NIC. Wired into the XC site interface binding so a NIC recreate updates the site."
  value       = azurerm_network_interface.mgmt.mac_address
}

output "mgmt_private_ip" {
  description = "Private IP of the eth0/SLO (mgmt) NIC — the BGP local address and the Route Server BGP connection peer_ip."
  value       = azurerm_network_interface.mgmt.private_ip_address
}

output "sli_private_ip" {
  description = "Private IP of the internal/SLI NIC — the only one of the CE's three private addresses that answers, and where the Site Console web UI (TCP 65500) is served. Reach it from a workstation with `az network bastion tunnel --target-resource-id <this CE's vm_id> --resource-port 65500`; the mgmt/SLO and external addresses serve nothing."
  value       = azurerm_network_interface.internal.private_ip_address
}

output "vm_name" {
  description = "CE VM name."
  value       = azurerm_linux_virtual_machine.this.name
}

output "vm_id" {
  description = "CE VM ARM resource ID. Addresses the VM (this is what `az network bastion tunnel --target-resource-id` wants) — it is NOT a per-instance identity: see vm_instance_id."
  value       = azurerm_linux_virtual_machine.this.id
}

# The two ids above and below are easy to confuse and are not interchangeable.
# vm_id is the ARM resource id, ".../virtualMachines/<hostname>" — derived from
# the VM name and therefore byte-identical before and after a replacement, which
# makes it the right handle for ADDRESSING the VM (Bastion tunnels) and useless
# as a "this node was rebuilt" signal. This one is regenerated for every new
# instance, and it is the same value the CE reports to F5 XC as the
# registration's infra.instance_id — so it is the identity that binds an XC site
# object to a specific node. modules/xc-site keys the site's replace_triggered_by
# on it (see issue #674).
output "vm_instance_id" {
  description = "128-bit unique id of the CE VM INSTANCE (regenerated on replacement; equals the XC registration's infra.instance_id). Not the ARM resource id, which is name-derived and survives a replacement."
  value       = azurerm_linux_virtual_machine.this.virtual_machine_id
}

output "identity_id" {
  description = "User-assigned managed identity resource ID."
  value       = azurerm_user_assigned_identity.this.id
}

terraform/modules/ce-node/variables.tf

variable "hostname" {
  description = "CE VM hostname (also the Azure VM name), e.g. f5-xc-ce-vm-01."
  type        = string
}

variable "resource_group_name" {
  description = "Resource group (created by the hub module)."
  type        = string
}

variable "location" {
  description = "Azure region."
  type        = string
}

variable "zone" {
  description = "Availability zone for the VM (1, 2 or 3)."
  type        = string
}

variable "vm_size" {
  description = "VM size."
  type        = string
  default     = "Standard_D8_v4"
}

variable "mgmt_subnet_id" {
  description = "snet-hub-management subnet ID (eth0/SLO)."
  type        = string
}

variable "external_subnet_id" {
  description = "snet-hub-external subnet ID."
  type        = string
}

variable "internal_subnet_id" {
  description = "snet-hub-internal subnet ID."
  type        = string
}

variable "mgmt_private_ip" {
  description = "Static private IP for the eth0/SLO (mgmt) NIC — the BGP local address (e.g. 10.0.1.4)."
  type        = string
}

variable "admin_username" {
  description = "SSH admin username."
  type        = string
  default     = "azureuser"
}

variable "ssh_public_key" {
  description = "SSH public key MATERIAL (string), passed down from the root."
  type        = string
}

variable "custom_data" {
  description = "Base64-encoded cloud-init custom data for the volterra-node bootstrap."
  type        = string
}

variable "tags" {
  description = "Tags applied to every resource."
  type        = map(string)
  default     = {}
}

# Sized from measurement, not from a rule of thumb. Issue #714 ran one disposable
# single-node Azure Secure Mesh v2 site per size, all from marketplace image 0.9.2,
# installing the pair the tenant advertises (crt-20260201-0179 + OS 9.2026.14):
#
#     31 GiB (the image default)   FAIL — voucher DaemonSet 0/1, site stuck
#                                  PROVISIONING with nothing installed
#     33 GB                        PASS
#     36 / 40 / 48 / 64 GB         PASS
#
# The default below is deliberately NOT the measured 33 GB floor. 33 works for that
# pair on that image today; a build with a marginally larger payload would fail there
# with no configuration change and no obvious cause — exactly the position the image
# default is in now.
#
# Do NOT size this from F5's documented "20 GB plus 15% of capacity" pre-upgrade
# figure. That check gates Console UI upgrades only and does not apply to the API
# path; applied here it predicts 48 GB would fail, and 48 GB installs cleanly.
variable "os_disk_size_gb" {
  description = "CE OS disk size in GB. The marketplace image default of 31 GiB is measured to FAIL the version pair F5 advertises (#714); 33 GB is the smallest size that works and this default carries headroom above it."
  type        = number
  default     = 64

  validation {
    condition     = var.os_disk_size_gb >= 40
    error_message = "os_disk_size_gb must be at least 40 GB. The image default of 31 GiB fails the advertised version pair, and 33 GB — the measured minimum — leaves no margin for a larger future payload (#714)."
  }
}

terraform/modules/ce-node/versions.tf

terraform {
  required_version = ">= 1.8"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
}

terraform/modules/ce-topology/

Pure computation, no providers and no data sources: expands ce_count into the per-CE map that drives every for_each. Plan-testable at any N without credentials.

terraform/modules/ce-topology/main.tf

# Pure computation module: expands ce_count into the per-CE node map that drives
# the ce-node, xc-site and azure-route-server-bgp for_each loops. Has NO
# providers and NO data sources, so it is fully hermetic and plan-testable at
# any N without Azure/XC credentials (see tests/n_scaling.tftest.hcl).

terraform {
  required_version = ">= 1.8"
}

variable "ce_count" {
  description = "Number of CE nodes (1..3)."
  type        = number
  default     = 3

  validation {
    condition     = var.ce_count >= 1 && var.ce_count <= 3
    error_message = "ce_count must be between 1 and 3."
  }
}

variable "region_short" {
  description = "Short region token (e.g. eastus)."
  type        = string
  default     = "eastus"
}

variable "mgmt_subnet_prefix" {
  description = "Management subnet prefix. The eth0/SLO IP is derived from it (4 + index)."
  type        = string
  default     = "10.0.1.0/26"
}

variable "hostname_prefix" {
  description = "Prefix for CE VM hostnames (hostname = <prefix>-0<n>)."
  type        = string
  default     = "f5-xc-ce-vm"
}

variable "site_prefix" {
  description = "Prefix for XC site names (site = <prefix>-<region_short>0<n>). REQUIRED, deliberately without a default: the root passes var.component, so every object name in the deployment derives from one place and no deployment-specific prefix can hide in a module default."
  type        = string

  validation {
    condition     = can(regex("^[a-z0-9]([a-z0-9-]*[a-z0-9])?$", var.site_prefix))
    error_message = "site_prefix must be a DNS-style label: lowercase alphanumerics and hyphens, not starting or ending with a hyphen (it becomes part of an XC object name)."
  }
}

locals {
  ce_nodes = {
    for i in range(var.ce_count) : "${var.region_short}0${i + 1}" => {
      index     = i
      hostname  = "${var.hostname_prefix}-0${i + 1}"
      site_name = "${var.site_prefix}-${var.region_short}0${i + 1}"
      # eth0/SLO IP: 10.0.1.4, .5, .6 for i = 0, 1, 2.
      slo_ip = cidrhost(var.mgmt_subnet_prefix, 4 + i)
      # Availability zone 1, 2, 3.
      az = element(["1", "2", "3"], i)
      # The network_interface object XC auto-creates for the explicit SLO
      # interface. The bgp peer references it by this exact name.
      interface_name = "ves-io-securemesh-site-v2-${var.site_prefix}-${var.region_short}0${i + 1}-network-${var.hostname_prefix}-0${i + 1}-eth0-0"
    }
  }
}

output "ce_nodes" {
  description = "Map keyed by <region_short>0<n> of per-CE attributes (hostname, site_name, slo_ip, az, interface_name, index)."
  value       = local.ce_nodes
}

output "ce_count" {
  description = "Number of CE nodes expanded."
  value       = length(local.ce_nodes)
}

terraform/modules/client-vm/

The test client inside the VNet, used to read effective routes and drive traffic at the VIP.

terraform/modules/client-vm/main.tf

# Simple Ubuntu test client in snet-hub-internal. Used to generate HTTP traffic
# to the VIP and to read the VNet effective route table (ECMP proof).

resource "azurerm_public_ip" "this" {
  name                = "${var.name}PublicIP"
  resource_group_name = var.resource_group_name
  location            = var.location
  allocation_method   = "Static"
  sku                 = "Standard"
  tags                = var.tags
}

resource "azurerm_network_security_group" "this" {
  #checkov:skip=CKV_AZURE_10:Lab NSG - SSH open for demo access
  #checkov:skip=CKV_AZURE_160:Lab NSG - HTTP port 80 required for traffic
  #checkov:skip=CKV_AZURE_220:Lab NSG - SSH open for demo access
  name                = "${var.name}NSG"
  resource_group_name = var.resource_group_name
  location            = var.location

  security_rule {
    name                       = "SSH"
    priority                   = 1001
    direction                  = "Inbound"
    access                     = "Allow"
    protocol                   = "Tcp"
    source_port_range          = "*"
    destination_port_range     = "22"
    source_address_prefix      = "*"
    destination_address_prefix = "*"
  }

  tags = var.tags
}

resource "azurerm_network_interface" "this" {
  #checkov:skip=CKV_AZURE_119:Lab NIC - public IP required for demo access
  name                = "${var.name}VMNic"
  resource_group_name = var.resource_group_name
  location            = var.location

  ip_configuration {
    name                          = "ipconfig1"
    subnet_id                     = var.subnet_id
    private_ip_address_allocation = "Dynamic"
    public_ip_address_id          = azurerm_public_ip.this.id
  }

  tags = var.tags
}

resource "azurerm_network_interface_security_group_association" "this" {
  network_interface_id      = azurerm_network_interface.this.id
  network_security_group_id = azurerm_network_security_group.this.id
}

resource "azurerm_linux_virtual_machine" "this" {
  #checkov:skip=CKV_AZURE_50:Lab VM - no extensions required
  #checkov:skip=CKV_AZURE_93:Lab VM - platform-managed encryption sufficient
  name                = var.name
  computer_name       = var.name
  resource_group_name = var.resource_group_name
  location            = var.location
  size                = var.vm_size

  admin_username                  = var.admin_username
  disable_password_authentication = true

  admin_ssh_key {
    username   = var.admin_username
    public_key = var.ssh_public_key
  }

  network_interface_ids = [azurerm_network_interface.this.id]

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Premium_LRS"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-jammy"
    sku       = "22_04-lts-gen2"
    version   = "latest"
  }

  custom_data = var.custom_data != "" ? var.custom_data : null

  tags = var.tags
}

output "public_ip" {
  description = "Public IP of the test client."
  value       = azurerm_public_ip.this.ip_address
}

output "private_ip" {
  description = "Private IP of the test client."
  value       = azurerm_network_interface.this.private_ip_address
}

output "vm_name" {
  description = "Test client VM name."
  value       = azurerm_linux_virtual_machine.this.name
}

output "nic_name" {
  description = "Test client NIC name (read effective routes from this NIC)."
  value       = azurerm_network_interface.this.name
}

terraform/modules/client-vm/variables.tf

variable "name" {
  description = "Test client VM name; every child resource derives from it (<name>PublicIP, <name>NSG, <name>VMNic). REQUIRED, deliberately without a default: the root passes a value derived from var.component, so no deployment-specific name can hide in a module default."
  type        = string
}

variable "resource_group_name" {
  description = "Resource group (created by the hub module)."
  type        = string
}

variable "location" {
  description = "Azure region."
  type        = string
}

variable "subnet_id" {
  description = "snet-hub-internal subnet ID the client attaches to."
  type        = string
}

variable "vm_size" {
  description = "VM size."
  type        = string
  default     = "Standard_B2s"
}

variable "admin_username" {
  description = "SSH admin username."
  type        = string
  default     = "azureuser"
}

variable "ssh_public_key" {
  description = "SSH public key MATERIAL (string), passed down from the root."
  type        = string
}

variable "custom_data" {
  description = "Base64-encoded cloud-init custom data."
  type        = string
  default     = ""
}

variable "tags" {
  description = "Tags applied to every resource."
  type        = map(string)
  default     = {}
}

terraform/modules/client-vm/versions.tf

terraform {
  required_version = ">= 1.8"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"
    }
  }
}

terraform/modules/xc-site/

The XC site object, its BGP peering, and the registration approval that resolves a r-<uuid> registration by site name.

terraform/modules/xc-site/main.tf

# Parks the identity of the CE VM instance the site's node runs on, so the site
# object has something to be coupled to. Nothing else reads it — its only job is
# to be named in the site's replace_triggered_by below.
#
# WHY A SEPARATE RESOURCE. replace_triggered_by may only name managed resources
# declared in the SAME module as the resource carrying the lifecycle block, and
# the CE VM lives in modules/ce-node. Parking the id here is the standard way to
# carry an external value across that boundary.
#
# WHY input AND NOT triggers_replace. This resource must never itself be
# replaced — only observed. A changed `input` is an in-place UPDATE, which is
# what replace_triggered_by reacts to; that keeps the resource cheap and its
# behaviour on ADOPTION correct (see below).
#
# ADOPTION IS INERT. Adding this resource to a deployment that already exists
# plans it as a CREATE, and a create of the referenced resource does NOT fire
# replace_triggered_by — only a subsequent change to its value does. Verified on
# Terraform v1.10.5 and again on v1.15.0:
# adding the pair to a populated state plans "1 to add, 0 to change,
# 0 to destroy". So this fix does not itself trigger the fleet-wide rebuild it
# exists to prevent — no import, no targeted apply, no seeding.
resource "terraform_data" "ce_vm" {
  input = var.ce_vm_instance_id
}

# Single-node Secure Mesh v2 CE site with an EXPLICIT eth0/SLO interface. The
# explicit interface is what makes XC auto-create the network_interface object
# (var.interface_name) that the BGP peer binds to — without it a standalone bgp
# object is accepted but never renders to FRR (see xcsh #1207).
resource "xcsh_securemesh_site_v2" "this" {
  count       = var.create_site ? 1 : 0
  name        = var.site_name
  namespace   = "system"
  description = "MCN CE-HA (BGP/ECMP) single-node SMSv2 site ${var.site_name} — explicit eth0 SLO interface for BGP peer binding."
  # `null`, not `{}`, when no labels are set. xcsh #1286 makes the provider preserve a
  # config-declared empty map on the POST-APPLY read-back, but import has no config to
  # read: the state carries only id/name/namespace and `ReadRequest` exposes nothing
  # else, so a literal `{}` would still re-plan as `+ labels = {}` on the first
  # post-import plan. Sending `null` when the map is empty stops asking the provider to
  # distinguish "declared empty" from "absent" — something it cannot observe on import.
  # (The nested `interface_list.labels {}` marker is a separate class, fixed by xcsh #1244.)
  #
  # EXPECTED ONE-TIME DRIFT AFTER A NODE (RE)REGISTERS. F5 XC stamps the node's
  # hardware facts onto the site as labels (host-os-version, hw-model,
  # hw-serial-number, hw-vendor, hw-version) once the CE registers. The next plan
  # therefore shows `- labels = {...} -> null` for that site, and applying it
  # settles — XC does not re-stamp them. It is one more convergence pass in the
  # already two-phase deploy, not drift to chase; observed on the site rebuilt by
  # the #674 CE replacement.
  labels = length(var.labels) > 0 ? var.labels : null

  azure {
    not_managed {
      node_list {
        hostname  = var.hostname
        type      = "Control"
        public_ip = ""

        interface_list {
          name = "eth0"

          ethernet_interface {
            device = "eth0"
            mac    = var.mgmt_nic_mac
          }

          # Site Local Outside (SLO) — required on every site; BGP peers from here.
          network_option {
            site_local_network {}
          }

          dhcp_client {}
        }
      }
    }
  }

  block_all_services {}
  disable_ha {}

  dns_ntp_config {
    f5_dns_default {}
    f5_ntp_default {}
  }

  local_vrf {
    default_config {}
    default_sli_config {}
  }

  logs_streaming_disabled {}
  no_forward_proxy {}
  no_network_policy {}
  no_s2s_connectivity_sli {}
  no_s2s_connectivity_slo {}

  offline_survivability_mode {
    no_offline_survivability_mode {}
  }

  performance_enhancement_mode {
    perf_mode_l7_enhanced {
      # Provider v3.80.0 gave perf_mode_l7_enhanced a {jumbo_disabled | jumbo_enabled}
      # sub-oneof. F5 materialises jumbo_disabled server-side, so leaving both members
      # undeclared makes the site land and then re-plan the marker as a removal on
      # every subsequent plan — it never reaches 0 changes. Declaring the server
      # default explicitly is what settles it (same fix coverage/smsv2 took in #625).
      jumbo_disabled {}
    }
  }

  re_select {
    geo_proximity {}
  }

  # CE software and OS selection is create-time configuration. The node always
  # installs a destination build on first boot; an empty variable arms the
  # default_* marker and means "install the newest version the server advertises."
  # That is the deployment policy, not an accidental omission. The clean
  # 2026-08-03 rebuild selected the advertised pair on all three 64 GB nodes and
  # brought all three sites ONLINE. Issue #714 separately proves why the disk
  # default carries headroom: the same pair failed on the marketplace image's
  # 31 GiB disk and installed at every tested size from 33 GB upwards.
  #
  # Terraform cannot update these fields after creation: PUT is rejected when
  # pinning forward, pinning backward, or clearing a pin. The platform can perform
  # an in-place change through the site upgrade_sw and upgrade_os actions, but the
  # provider cannot drive those actions yet (xcsh#1390). Set a concrete value only
  # when deliberately reproducing an older build.
  software_settings {
    os {
      dynamic "default_os_version" {
        for_each = var.os_version == "" ? [1] : []
        content {}
      }
      operating_system_version = var.os_version == "" ? null : var.os_version
    }
    sw {
      dynamic "default_sw_version" {
        for_each = var.sw_version == "" ? [1] : []
        content {}
      }
      volterra_software_version = var.sw_version == "" ? null : var.sw_version
    }
  }

  # Rebuild the site object whenever the CE VM instance it describes is rebuilt
  # (issue #674).
  #
  # THE FAILURE THIS PREVENTS. A CE's runtime registration is bound to one node
  # instance and holds the control plane's unique
  # (tenant, cluster_name, hostname) index. Destroying the VM does NOT retire
  # that registration, so the replacement node — same site, same hostname —
  # cannot create its own: the create fails with UniqueSecondaryIndexViolation
  # and retries on a ~65 s loop forever. Nothing recovers on its own, and worse,
  # nothing in the graph noticed: with no reference to the node's identity
  # anywhere, `terraform plan` reported "No changes" for the whole time the
  # fleet was down.
  #
  # WHY REPLACING THE SITE IS THE FIX. Deleting the site object takes its
  # registrations with it — observed live while replacing one CE: the site
  # 404ed and the registration bound to the outgoing instance disappeared in the
  # same poll — so the replacement node registers into a site whose index key is
  # free. Deleting only the registration is not enough: the site keeps a status
  # object that then rejects the node's workload request.
  #
  # THE ORDER IS THE POINT. Terraform runs this as: destroy site -> destroy VM
  # -> create VM -> create site. The stale registration is therefore gone before
  # the replacement node ever boots, and the site is back before the node
  # finishes booting and registers. The outgoing node cannot slip a fresh
  # registration into the gap: once its own registration is deleted it 404-loops
  # against the name it persisted in registration-obj.yml instead of creating a
  # new one.
  #
  # SAFE WHILE OTHER OBJECTS REFERENCE THE SITE. xcsh_bgp and the root HTTP load
  # balancer's advertise_where both name this site, and F5 XC resolves those
  # references lazily: deleting a site that both of them reference returns HTTP
  # 200 and leaves them intact, and re-creating it under the same name re-binds
  # them (verified against the live tenant with a throwaway site).
  #
  # THAT LAZINESS DOES NOT EXTEND TO CREATION, and the difference has bitten once.
  # An EXISTING load balancer tolerates a dangling site reference; POSTing a NEW one
  # whose advertise_where names a site that does not exist yet is rejected outright
  # with `[BAD_REQUEST] Invalid request parameters`. Renaming the deployment does
  # exactly that — every site is destroyed and re-created under a different name, so
  # the load balancer is created fresh — which is why the root resource now carries
  # an explicit `depends_on = [module.xc_site]`. Do not remove it on the strength of
  # the paragraph above: it is about deletion, not creation.
  lifecycle {
    replace_triggered_by = [terraform_data.ce_vm]
  }
}

# The approve API takes the runtime registration name ("r-<uuid>"), NOT the site
# name (GET .../registrations/<site> -> 404). registrations_by_site returns
# HTTP 200 with items:[] for a site whose CE has not registered yet, so this read
# never fails an early apply — it just reports found = false.
#
# NOTE: this data source must never carry depends_on. Its inputs are statically
# derived from ce_topology, so it resolves at plan time; a resource dependency
# would make the count below unknown at plan time ("The count value depends on
# resource attributes that cannot be determined until apply").
data "xcsh_site_registration" "this" {
  site_name = var.site_name # == passport.cluster_name (cloud-init ClusterName)
  hostname  = var.hostname  # discriminator for multi-node sites
  namespace = "system"
}

# Approve the CE registration so the node reaches ONLINE without the manual
# console step (#1206 / #1210). The registration exists only after the CE boots
# and registers via the token, so the first apply plans no approval; re-apply
# once the CE has registered (see the deploy ordering in main.tf).
#
# ADOPTING AN ALREADY-APPROVED CE: the approve action only legitimately moves a
# registration out of NEW, so applying this against a CE that is already
# APPROVED/ONLINE would POST a redundant approve (which the API may reject —
# xcsh #1278). Import the existing approval instead of letting Terraform create
# it, using namespace/name with the RUNTIME registration name (the site name
# 404s — read it from the data source's `registration_name` output):
#
#   terraform import 'module.xc_site["eastus01"].xcsh_registration_approval.this[0]' \
#     system/r-dcec2400-52d5-4154-9fd0-4b042d3fe18d
#
# Or set approve_registration = false to keep approval out of the graph entirely.
resource "xcsh_registration_approval" "this" {
  count = var.approve_registration && data.xcsh_site_registration.this.found ? 1 : 0

  namespace = "system"
  name      = data.xcsh_site_registration.this.name
  state     = "APPROVED"
}

# One bgp object per CE site: eBGP from the CE (ASN var.ce_asn) to the Azure
# Route Server (ASN var.rs_asn), one external peer per Route Server virtual
# router IP, each bound to the explicit SLO interface.
#
# NOT BLOCKED — and nothing about this arm is gated any more. The object-ref name
# length limit that used to block it is gone: the provider relaxed it to
# stringvalidator.LengthBetween(1, 128) in v3.74.0, so the 71-char interface object
# XC auto-generates for the explicit SLO interface
# (ves-io-securemesh-site-v2-<site>-network-<hostname>-eth0-0) validates. The floor
# that guarantees it is declared once, in versions.tf — do not restate the number.
#
# var.enable_bgp therefore defaults true and every test now runs with that default;
# it survives only as an escape hatch for deploying the topology without BGP. It is
# NOT an ordering gate: var.interface_name is derived statically from ce_topology, and
# XC accepts a bgp object naming an interface that does not exist yet (it converges
# once the CE is up — see the deploy ordering in the root main.tf).
resource "xcsh_bgp" "this" {
  count = var.enable_bgp ? 1 : 0

  name        = "${var.site_name}-bgp"
  namespace   = "system"
  description = "CE ${var.site_name} BGP to Azure Route Server via explicit SLO interface."

  where {
    site {
      ref {
        namespace = "system"
        name      = var.site_name
      }
      network_type = "VIRTUAL_NETWORK_SITE_LOCAL"
      disable_internet_vip {}
    }
  }

  bgp_parameters {
    asn = var.ce_asn
    # local_address {} = derive the BGP router ID from the interface's local
    # address (the JSON's BGP_ROUTER_ID_FROM_INTERFACE; there is no separate
    # bgp_router_id_type attribute in the provider schema).
    local_address {}
  }

  # Iterate over a plan-KNOWN peer count (rs_peer_count) and index into
  # rs_peer_ips. The IP values may be unknown until the Route Server is applied,
  # but the number of peers is fixed, so the block expands cleanly at plan time.
  dynamic "peers" {
    for_each = { for i in range(var.rs_peer_count) : "azure-rrs-${i + 1}" => i }
    content {
      metadata {
        name = peers.key
      }

      external {
        asn     = var.rs_asn
        address = try(var.rs_peer_ips[peers.value], "")
        port    = var.peer_port

        interface {
          namespace = "system"
          name      = var.interface_name
        }

        disable_v6 {}
      }

      passive_mode_disabled {}
      bfd_disabled {}
    }
  }
}

terraform/modules/xc-site/outputs.tf

output "site_name" {
  description = "XC securemesh_site_v2 name."
  value       = var.site_name
}

output "bgp_name" {
  description = "XC bgp object name (null when enable_bgp is false)."
  value       = one(xcsh_bgp.this[*].name)
}

output "interface_name" {
  description = "Auto-derived network_interface object name the BGP peer binds to."
  value       = var.interface_name
}

# The node identity the site object is currently coupled to. Compare it against
# the registration's own infra.instance_id (visible in the F5 XC API, not yet on
# the xcsh_site_registration data source — provider issue #1376) to tell whether
# a site is still bound to a node that no longer exists.
output "bound_vm_instance_id" {
  description = "CE VM instance id the site object is bound to. Replacing that instance replaces the site (issue #674)."
  value       = terraform_data.ce_vm.output
}

output "registration_name" {
  description = "Runtime registration name (r-<uuid>) resolved from the site name; null until the CE has registered."
  value       = data.xcsh_site_registration.this.found ? data.xcsh_site_registration.this.name : null
}

output "registration_state" {
  description = "Current registration state reported by XC (NEW, APPROVED, ONLINE, ...); null until the CE has registered."
  value       = data.xcsh_site_registration.this.found ? data.xcsh_site_registration.this.state : null
}

output "registration_approval_name" {
  description = "Name of the approved registration (null when approve_registration is false or the CE has not registered yet)."
  value       = one(xcsh_registration_approval.this[*].name)
}

output "peer_count" {
  description = "Number of external BGP peers configured (one per Route Server IP; 0 when enable_bgp is false)."
  value       = var.enable_bgp ? var.rs_peer_count : 0
}

terraform/modules/xc-site/variables.tf

variable "site_name" {
  description = "XC securemesh_site_v2 name (e.g. mcn-ce-ha-eastus01), created in namespace system."
  type        = string
}

variable "hostname" {
  description = "CE node hostname, must match the Azure VM computer name."
  type        = string
}

variable "interface_name" {
  description = "Auto-derived network_interface object name the BGP peer references: ves-io-securemesh-site-v2-<site>-network-<hostname>-eth0-0."
  type        = string
}

variable "mgmt_nic_mac" {
  description = "MAC address of the CE eth0/SLO NIC. Pins the SMSv2 interface to the NIC."
  type        = string
}

variable "ce_vm_instance_id" {
  description = "128-bit unique id of the CE VM instance this site's node runs on (azurerm_linux_virtual_machine.virtual_machine_id — the same value the CE reports as the registration's infra.instance_id). Required, and deliberately NOT the ARM resource id, which is name-derived and identical after a replacement. The site object's lifecycle is coupled to it so that rebuilding the node rebuilds the site instead of leaving a registration bound to a destroyed instance (issue #674)."
  type        = string

  validation {
    condition     = trimspace(var.ce_vm_instance_id) != ""
    error_message = "ce_vm_instance_id must be the CE VM's virtual_machine_id; an empty value would couple every node's site to the same key."
  }
}

variable "rs_peer_ips" {
  description = "List of Azure Route Server BGP peer IPs (virtual_router_ips), e.g. [10.0.4.4, 10.0.4.5]. Values may be unknown until apply; the count of peers comes from rs_peer_count so the peers block expands at plan time."
  type        = list(string)
}

variable "rs_peer_count" {
  description = "Number of external BGP peers (Azure Route Server always exposes exactly 2 virtual router IPs). Drives the peers block with a plan-known count so for_each never sees an unknown value."
  type        = number
  default     = 2
}

variable "ce_asn" {
  description = "CE (local) BGP ASN."
  type        = number
  default     = 64512
}

variable "rs_asn" {
  description = "Route Server (peer) BGP ASN."
  type        = number
  default     = 65515
}

variable "peer_port" {
  description = "BGP peer TCP port."
  type        = number
  default     = 179
}

variable "labels" {
  description = "Labels applied to the site object."
  type        = map(string)
  default     = {}
}

variable "enable_bgp" {
  description = "Create the xcsh_bgp object. Defaults true, and nothing gates it: the object-ref name length blocker that once forced it false was removed in provider v3.74.0. Retained only as an escape hatch for planning or deploying the graph without BGP. See main.tf."
  type        = bool
  default     = true
}

variable "create_site" {
  description = "Create the xcsh_securemesh_site_v2 resource object. Set false when the site object is auto-provisioned by registration approval."
  type        = bool
  default     = true
}

variable "approve_registration" {
  description = "Approve the CE's runtime registration (named r-<uuid>, resolved from the site name by the xcsh_site_registration data source) instead of clicking Approve in the console. Two-phase by nature: the registration does not exist until the CE has booted and registered via the token, so the first apply plans no approval and a later apply creates it. For a CE that is ALREADY approved/ONLINE, import the existing approval (namespace/r-<uuid>, see the note in main.tf) rather than letting Terraform create it, since re-approving a non-NEW registration may be rejected. Set false to leave approval to an operator."
  type        = bool
  default     = true
}

variable "os_version" {
  description = "CE operating system version. Empty deliberately selects the server-advertised latest version; set a value only to reproduce an older build."
  type        = string
  default     = ""
}

variable "sw_version" {
  description = "CE F5 Distributed Cloud software version. Empty deliberately selects the server-advertised latest build; set a value only to reproduce an older build."
  type        = string
  default     = ""
}

terraform/modules/xc-site/versions.tf

terraform {
  required_version = ">= 1.8"

  required_providers {
    xcsh = {
      source = "f5-sales-demo/xcsh"
      # This module declares the registration pair, so its floor must carry them:
      # >= 3.77.3 ships the xcsh_site_registration data source (resolves a CE's
      # r-<uuid> runtime registration name from its site name) alongside
      # xcsh_registration_approval. It also includes >= 3.74.0's object-ref name
      # validator relaxed 63 -> 128 chars, so the real 71-char auto-derived SLO
      # interface name the BGP peer binds to validates (xcsh_bgp is not
      # length-gated). >= 3.77.5 raises the floor to the two labels fixes this
      # module now relies on instead of `ignore_changes`: import-marker suppression
      # for the nested interface_list `labels {}` block (#1244) and preservation of
      # a config-declared empty top-level metadata `labels` map (#1286).
      # Keep in lock-step with the root terraform/versions.tf.
      version = ">= 3.77.5"
    }
  }
}