Deploy the demo

Prerequisites

Allow tens of minutes for bootstrap and hours for a full serial-upgrade rehearsal.

• Terraform at the version floor in terraform/versions.tf — >= 1.16.1, for provider-defined functions and the check block that guards the VIP.
• The xcsh provider at exactly v11.0.2 for normal deployments. This deployment uses only the clean-break SMSv2 contract. A release-candidate acceptance run may substitute one reviewed local binary through the explicit procedure below. The root .terraform.lock.hcl is committed; its checksums and the exact source constraint are authoritative.
• An F5 XC API credential for the tenant in var.expected_xc_tenant, exported as XCSH_API_TOKEN (or the P12/PEM pair). Export the credential only, never a URL — see below.
• Azure credentials with rights to create a resource group, VNet, Internal Load Balancer and VMs.
• A pre-existing F5 XC namespace for the app tier. The deployment reads it and never creates or destroys it, so a namespace holding unrelated demos can never land on this stack’s destroy list.

dev_overrides silently replaces the provider you think you are testing

If ~/.terraformrc contains a dev_overrides block for xcsh, every local Terraform command uses the binary in that directory and ignores both the version constraint and the lock file. A build a few days old is enough to fail with Blocks of type "…" are not expected here on configuration that is perfectly valid against the published provider.

Select a task-specific CLI configuration through TF_CLI_CONFIG_FILE for registry installation. Keep candidate overrides in a separate file, as shown below; do not rename a shared global configuration. See HashiCorp CLI configuration.

Supply variable values via tfvars

Every variable has a default or derived default. For internet readers and reference deployments, supply your own values via terraform.tfvars so deployment-specific names, subscriptions, domains, and addresses are not hardcoded into configuration:

cd terraform
cp terraform.tfvars.example terraform.tfvars

Variable	Default / Role	Why it should be customized
origin_ip	(no default)	Public IP of your origin server (e.g. 203.0.113.10 from RFC 5737 documentation space).
lb_domain	(no default)	FQDN for Rest of World traffic (e.g. mcn-ce-ha.f5-sales-demo.com or mcn.example.com).
ca_lb_domain	mcn-ce-ha.f5-sales-demo.ca	FQDN for Canada-only traffic (e.g. mcn.example.ca).
enable_canada	true	Toggle to deploy parallel Canadian regional infrastructure, virtual sites, and load balancer.
ca_location	canadacentral	Azure region for Canadian CEs and Route Server.
ca_hub_cidr	10.200.0.0/16	VNet CIDR block for Canadian Hub VNet (RFC 1918 space).
ca_vip	10.250.1.10	HA VIP advertised by Canadian CEs via eBGP (must be outside VNet CIDRs).
ca_re_cities	["toronto", "montreal"]	List of Regional Edge cities for Canadian RE Virtual Site selector expression.

Both origin_ip and domains are format-validated so a typo fails the plan rather than runtime routing.

terraform.tfvars is gitignored to protect sensitive environment variables, subscription IDs, and custom credentials.

The tenant is configuration, not whatever your shell is exporting

var.expected_xc_tenant names the tenant and is the only place it is named. providers.tf derives the xcsh provider’s api_url from it, which deliberately overrides any XCSH_API_URL in the environment, and a postcondition in terraform/main.tf fails the plan when the environment names a different tenant.

Read either half without running anything that changes state:

terraform output -raw xc_tenant       # what the deployment targets
terraform output -raw xc_env_tenant   # what your shell claims — diagnostic only

Illustrative output for a correctly matched configuration:

example-corp
example-corp

This guardrail exists because the demo silently re-homed itself into the wrong tenant

The demo was built in one tenant. A credential file later began exporting a different tenant’s XCSH_API_URL; subsequent applies minted a token for that tenant, the CE VMs re-registered there, and their original registrations were abandoned. Every plan in between was clean, because nothing in the configuration had an opinion about the tenant (issue 696, fixed in PR 697). Recovering it meant destroying the stack in the wrong tenant and rebuilding it in the right one.

The symptom to recognize, because it is not self-describing: a token minted for another tenant fails the pinned endpoint with a bare 401, which reads exactly like an expired credential. If a credential you believe is good returns 401, compare the two outputs above before you mint another token.

AWS SMSv2 preflight is mandatory

The AWS Transit Gateway and KVM paths are pinned to the immutable xcsh v11.0.2 release, which aligns mutation ALPN negotiation with the HTTP/1 transport used for non-idempotent requests. That release publishes contract f5xc-smsv2-api/v1 version 7.0.0 from api-specs-enriched v8.0.0. Its five AWS SMSv2 capabilities, including the evidence-backed aws_node_configuration discovery_rebuild strategy, must all report available; do not bypass a failed gate.

Use the AWS SMSv2 procedure for site-bound bootstrap, certified image ownership, MAC/device discovery, GRE transport, inside payload routing, listener realization, and cumulative recovery stages. The general lifecycle explains when to upgrade, stage, or replace a site.

Before applying an AWS plan, run the non-mutating preflight below. It checks the reviewed plan against the expected cloud account, region, XC tenant, site set, and provider artifact; it rejects Azure resource changes. The only supported lifecycle is three saved plans in this order: bootstrap for all three -bootstrap sites and CEs, bootstrap_retirement in destroy mode for only the disposable CE/site/material objects, then configured for the three distinct final sites using the private checksummed registration/ENI mapping. The preflight requires --lifecycle-phase and rejects a direct same-site transition. Continue only on ready / preflight_passed. Use a new empty evidence directory outside the repository, keep saved plans and the mapping private, and delete raw registration/mapping diagnostics after final creation.

For this lifecycle, use scripts/aws-smsv2-lifecycle-plan.sh instead of a direct terraform apply. It uses terraform-with-aws-sso.sh, makes a phase-specific saved plan, runs preflight, records only a plan digest, rechecks that digest, and applies only when passed --apply. Use the same private tfvars and exact three expected site names in each phase. For configured, first write the two sensitive bootstrap outputs aws_smsv2_bootstrap_registration_projection and aws_smsv2_owned_eni_projection to private files, then pass those files and a private mapping path. The wrapper invokes the mapping generator and deletes all three private artifacts after a successful configured apply.

Full UAT requires six physical interfaces, six Connect peers, and twelve established BGP sessions. --execute-uat is a consequential operation: it stops/restarts one CE, drives workload traffic, and verifies the runtime pair configured at first boot. It never introduces an intermediate version before the CE is healthy. See the dated acceptance baseline for the result and limits.

Apply

Run the bootstrap commands below from the repository root. After the AWS backend is configured, run the saved-plan sequence from terraform/ with terraform.tfvars and backend.hcl. Replace uppercase placeholders with private paths and the exact identities from your reviewed plan. On a new deployment, site outputs do not exist yet; take the expected names from the plan, not from stale state.

Bootstrap the dedicated AWS state backend

The former Azure state was retired with the prior deployment. Start with a fresh, dedicated S3 bucket; do not reuse a shared bucket or resurrect an old state object. The bootstrap stack creates KMS encryption, version recovery, public-access blocks, TLS-only bucket access, Terraform’s native S3 lockfile, regional server-access logs, EventBridge object notifications, and a separately encrypted cross-region recovery replica. Replica and log bucket names are derived from the unique primary name, so the bootstrap remains non-interactive; replica_region defaults to us-west-2 and must differ from the primary region.

Run this once with reviewed AWS identity and a globally unique bucket name. Run the commands from the repository root through the credential-isolation wrapper. The wrapper asks the AWS CLI for process credentials from the selected authenticated SSO profile, writes no credential value to disk, and prevents stale shared credentials from shadowing that profile. It uses the default profile unless --profile NAME precedes --. The bootstrap initially uses local state only because it creates the backend itself:

./scripts/terraform-with-aws-sso.sh -- -chdir=terraform/bootstrap/state-backend init -backend=false -input=false
./scripts/terraform-with-aws-sso.sh -- -chdir=terraform/bootstrap/state-backend plan -out='<PRIVATE_BOOTSTRAP_PLAN>' -var 'bucket_name=<UNIQUE_BUCKET_NAME>'
./scripts/terraform-with-aws-sso.sh -- -chdir=terraform/bootstrap/state-backend show '<PRIVATE_BOOTSTRAP_PLAN>'
./scripts/terraform-with-aws-sso.sh -- -chdir=terraform/bootstrap/state-backend apply '<PRIVATE_BOOTSTRAP_PLAN>'
./scripts/terraform-with-aws-sso.sh -- -chdir=terraform/bootstrap/state-backend output -json showcase_backend_hcl

After the reviewed apply, generate both ignored backend configurations and migrate the bootstrap stack’s local state programmatically:

SOURCE_REF="$(git symbolic-ref --quiet HEAD)"
SOURCE_COMMIT_SHA="$(git rev-parse HEAD)"
./scripts/configure-aws-state-backend.sh \
  --profile '<AWS_SSO_PROFILE>' \
  --source-ref "$SOURCE_REF" \
  --source-commit-sha "$SOURCE_COMMIT_SHA" \
  --deployment-owner-id 'showcase-team' \
  --deployment-actor-id 'terraform-cli'

The script verifies that the selected AWS identity owns the KMS key, creates the bootstrap and showcase state-key configurations emitted by the module without credentials, and uses Terraform’s supported -force-copy migration. Repeating the command verifies the existing remote metadata and reconfigures it idempotently. Production keeps the existing showcase key; a preview receives its deterministic environment-specific key before init. The showcase has no old state to migrate.

AWS saved-plan sequence

Create a registry-only CLI configuration for this task before initialization or planning:

REGISTRY_TFRC='<REGISTRY_CLI_CONFIG>'
PLAN_FILE='<PLAN_FILE>'
EVIDENCE_DIR='<NEW_PRIVATE_EVIDENCE_DIRECTORY>'
COLLISION_EVIDENCE_DIR='<NEW_PRIVATE_COLLISION_EVIDENCE_DIRECTORY>'
DEPLOYMENT_GENERATION='<NEW_IMMUTABLE_GENERATION>'
umask 077
printf 'provider_installation {\n  direct {}\n}\n' > "$REGISTRY_TFRC"
export TF_CLI_CONFIG_FILE="$REGISTRY_TFRC"
../scripts/terraform-with-aws-sso.sh -- init -upgrade -backend-config=backend.hcl
# The AWS preflight rejects Azure actions. This supported graph boundary omits
# Azure US, Azure Canada, Marketplace, and KVM resources without Terraform targeting.
../scripts/terraform-with-aws-sso.sh -- plan -var='enable_azure=false' -var='enable_kvm=false' \
  -var="deployment_generation=$DEPLOYMENT_GENERATION" -out="$PLAN_FILE"
../scripts/terraform-with-aws-sso.sh -- show "$PLAN_FILE"

# The collision guard renders this exact saved plan to a separate private
# directory. It rejects any existing AWS or F5 object with the planned name,
# including objects that are demonstrably owned by this deployment identity.
# A collision is never imported or adopted.
mkdir -p "$COLLISION_EVIDENCE_DIR"
../scripts/terraform-with-aws-sso.sh -- show -json "$PLAN_FILE" > "$COLLISION_EVIDENCE_DIR/reviewed-plan.json"
../scripts/aws-smsv2-owned-collision-preflight.sh \
  --plan-json "$COLLISION_EVIDENCE_DIR/reviewed-plan.json" \
  --aws-region '<AWS_REGION>' \
  --aws-account-id '<AWS_ACCOUNT_ID>' \
  --xc-tenant '<XC_TENANT>' \
  --creator-id "$XCSH_USERNAME" \
  --component 'mcn-ce-ha' \
  --deployment-generation "$DEPLOYMENT_GENERATION" \
  --manifest "$COLLISION_EVIDENCE_DIR/manifest.json"

../scripts/aws-smsv2-uat-preflight.sh \
  --evidence-dir "$EVIDENCE_DIR" \
  --lifecycle-phase configured \
  --terraform-dir . \
  --plan-file "$PLAN_FILE" \
  --expected-aws-account '<AWS_ACCOUNT_ID>' \
  --expected-aws-region '<AWS_REGION>' \
  --expected-xc-tenant '<XC_TENANT>' \
  --expected-site '<AWS_SITE_01>' \
  --expected-site '<AWS_SITE_02>' \
  --expected-site '<AWS_SITE_03>'

Inspect both preflight results. Continue only when the collision guard reports ready and the UAT guard reports ready with reason preflight_passed, then apply that same saved plan:

../scripts/terraform-with-aws-sso.sh -- apply "$PLAN_FILE"

If the collision guard finds a prerelease generation whose Terraform state was lost by an earlier provider deletion failure, do not select another name. Re-run the guard against the exact historical plan with --legacy-unlabelled-recovery, then use terraform/recovery/aws-smsv2-orphans. That root accepts only a schema-v2 manifest bound to the expected plan digest, AWS account and region, Sales Demo tenant, creator, component, generation, provider IDs, and creation evidence. Its configuration-driven import blocks must first produce an import-only saved plan. Verify that plan with scripts/verify-aws-smsv2-orphan-recovery-plan.sh before applying it. Destroy the adopted resources only with provider v11.0.2 or newer, then prove the recovery state is empty and every manifest identity is absent. CLI import, direct API deletion, state editing, taint, console cleanup, and broad tag-based deletion are not recovery mechanisms.

The recovery sequence uses its own S3 state key. Generate backend.hcl with scripts/configure-aws-state-backend.sh, initialize the recovery root, and pass the immutable identity values explicitly. Render the saved plan to JSON, then require the import verifier to produce a ready receipt before applying that exact plan. After adoption, reuse the unchanged recovery root to create a saved destroy plan with -destroy -parallelism=1, and verify it in --mode destroy before applying it. All plans, hashes, receipts, and state inventories stay in the private evidence directory.

For a partial recovery, set cumulative aws_bootstrap_site_keys in the planned inputs and pass only those admitted sites as --expected-site. Repeat planning and preflight for each stage. The full topology’s site identities can subsequently be read with terraform output -json aws_site_names.

After the complete graph converges, repeat the preflight command with a new evidence directory and --execute-uat to run the authorized disruption/traffic/failover and configured- version convergence rehearsal. Keep the reviewed plan and identity flags unchanged. The sites request the proven runtime pair on first boot; the UAT never introduces an intermediate version that could leave a newly created CE waiting for an upgrade before runtime health exists.

For a reviewed one-site action, set only aws_upgrade_software_version or aws_upgrade_os_version to an advertised target, set aws_upgrade_observed_sites = ["01"], and invoke action.xcsh_site_upgrade_sw.aws["01"] or action.xcsh_site_upgrade_os.aws["01"] in separate saved plans. Keep aws_software_version and aws_os_version fixed; changing the create-time pair can mutate sites and is not an action-only upgrade. Verify each saved plan using scripts/verify-aws-upgrade-action-plan.py and the task-owned backend-key receipt before applying that exact plan.

Deliberately selecting a provider candidate

Normal deployments retain the registry configuration above. For a reviewed local candidate, compute its digest and create a separate override file:

CANDIDATE_BIN='<CANDIDATE_PROVIDER_BINARY>'
CANDIDATE_DIR=$(dirname "$CANDIDATE_BIN")
CANDIDATE_SHA256="sha256:$(sha256sum "$CANDIDATE_BIN" | awk '{print $1}')"
CANDIDATE_TFRC='<CANDIDATE_CLI_CONFIG>'
cat > "$CANDIDATE_TFRC" <<EOF_CONFIG
provider_installation {
  dev_overrides {
    "f5-sales-demo/xcsh" = "$CANDIDATE_DIR"
  }
  direct {}
}
EOF_CONFIG
chmod 600 "$CANDIDATE_TFRC"
export TF_CLI_CONFIG_FILE="$CANDIDATE_TFRC"
terraform plan -out="$PLAN_FILE"

Review this new plan. Repeat preflight with both --candidate-provider-binary "$CANDIDATE_BIN" and --candidate-provider-sha256 "$CANDIDATE_SHA256", a new evidence directory, and the same expected identities. Apply only that candidate plan after a passing gate. Pass both candidate flags again for live UAT. The gate independently installs its registry contract probe and checks candidate bytes around live mutations; a missing, ambiguous, changed, or mismatched candidate fails closed.

Azure registration phases

The AWS preflight rejects Azure changes; do not use it to authorize an Azure apply. Review a separately scoped Azure plan and apply it. Once the intended nodes register, make and review another plan so approval can be included:

terraform plan -out='<AZURE_PLAN_FILE>'
terraform show '<AZURE_PLAN_FILE>'
terraform apply '<AZURE_PLAN_FILE>'

Repeat that sequence after registration, inspecting the changed plan before applying it. The Azure procedure describes the current implementation and the limits of its dated acceptance evidence.

Verify

Run the platform-scoped health checks before presenting the deployment.

Azure registration approval is automated and two-phase

The Azure path uses automated registration approval: the xcsh_registration_approval resource performs it, and the site module resolves which registration to approve.

That still does not make the deployment a single hands-off apply, and the reason is worth understanding before you watch a first run and conclude it failed:

• A CE’s registration object is named r-<uuid>. It is never named after the site, so nothing can predict the name at plan time.
• The registration only exists after the VM has booted and vpm has registered.
• So the first apply plans no approval at all. Re-apply once the CEs have registered and the approvals are created.

terraform/main.tf documents this ordering at the top of the file, which is the authoritative version — the deploy sequence lives with the code that implements it.

A brand-new site is not immediately reachable over the operate API

For roughly 25 to 30 minutes after a new site reports ONLINE, the vpm/debug endpoints answer with a BIG-IP 503 page reading The requested URL was rejected. It looks like a web application firewall block or a revoked token and is neither. Wait, then retry.

This affects a freshly built demo, not a running one.

Azure operator SSH is a build-time decision

The deployment writes an operator key to the appliance admin account, whose login shell is the Site CLI. It is written by cloud-init write_files, which runs once on first boot.

It cannot be added to CEs that are already running

custom_data is replace-only, so adopting this on running CEs replaces every CE VM. That is a fleet rebuild, not a configuration change. The same is true of anything else in the cloud-init template — including a comment, because custom_data is base64encode of the rendered file.

It used to be worse: a replaced CE could never re-register, because the destroyed node’s registration still held the control plane’s unique (tenant, cluster_name, hostname) index — and terraform plan reported “No changes” for the whole time the fleet was down. Recovery took two full rebuilds. That is fixed (issue 674, closed by PR 693): the site object’s lifecycle is now coupled to the CE VM instance, so replacing the VM replaces the site, and deleting the site takes its registrations with it before the replacement node boots.

Anyone adopting this elsewhere should read the SSH page first.

Clean up

terraform destroy

The app namespace survives: it is read, never managed, so destroy cannot take a namespace holding unrelated demos with it. Everything else in the resource group goes.