Software versions and rebuilds

Two Terraform variables select a Customer Edge’s software: ce_os_version for the operating system and ce_sw_version for the F5 Distributed Cloud build. Empty values deliberately select the newest versions that the service advertises. That is this deployment’s normal policy; a concrete value is only for reproducing an older build.

Verified 2026-08-03 in a from-zero rebuild: with both values empty, all three 64 GB nodes selected software crt-20260201-0179 and operating system 9.2026.14, reached ONLINE, and left a clean Terraform plan.

What the version fields do, in two phases

Separating these phases is critical; conflating them makes the operational behavior seem contradictory.

On first boot, a node installs the selected destination. terraform/modules/ce-node deploys the marketplace image with version = "latest", so the initial build on the node reflects the current marketplace image release (which changes over time). ce_sw_version specifies the target build version; leaving it empty instructs the platform to select the newest advertised build. The same applies to ce_os_version. Leaving either variable empty does not preserve existing versions on new deployments; the server automatically selects target versions for both.

Latest requires capacity headroom. On 2026-07-29, the advertised pair failed on the marketplace image’s 31 GiB disk and installed on every tested size from 33 GB through 64 GB. The pinned Marketplace image f5xc-ce-crt-20260201:20260201.0178.1 now requires at least 78 GiB at Azure VM creation. The deployment therefore sets an explicit 80 GB default and rejects values below 78 GB before Azure can reject the VM. With that prerequisite enforced, leaving the version values empty is both intentional and verified.

After first boot, nothing upgrades on its own. F5 Distributed Cloud advertises a newer build and waits. That is where “the node stays where it landed” holds — after the create, not during it.

Both phases are visible on the site object. During the clean rebuild on 2026-08-03, each site selected:

volterra_software_status.available_version   crt-20260201-0179
operating_system_status.available_version    9.2026.14

After installation, last_installed_version matched those selected values on all three sites. The site does not move again merely because a later version becomes available.

Create-time settings and upgrade actions

Provider v8 now exposes software and operating-system upgrade actions and a bounded status reader. MCN implements and verified the serial AWS path in aws_upgrade.tf; see the SMSv2 lifecycle. The Azure API observations below are historical and do not constitute v8 Azure upgrade acceptance.

Historical Azure API behavior

These are two separate facts, and conflating them produces the wrong plan.

Create-time selectors cannot. ce_os_version and ce_sw_version are effectively create-time only. Change either and apply, and the API rejects the update with [BAD_REQUEST] Invalid request parameters. Observed 2026-07-29 on disposable sites, in all three directions — pinning forward to a newer build, pinning backward to an older one, and un-pinning by clearing both fields. Forward is not a special case.

The API can. F5 Distributed Cloud exposes a dedicated upgrade action per site, which starts the change in place. The following historical direct-API example bypasses Terraform:

# software build
curl -X POST -H "Authorization: APIToken $TOKEN" -H 'Content-Type: application/json' \
  --data '{"version": "<software-version>"}' \
  "$API_URL/api/config/namespaces/system/sites/<site-name>/upgrade_sw"

# operating system
curl -X POST -H "Authorization: APIToken $TOKEN" -H 'Content-Type: application/json' \
  --data '{"version": "<os-version>"}' \
  "$API_URL/api/config/namespaces/system/sites/<site-name>/upgrade_os"

Observed 2026-07-29: the software call returned 200, the site moved to UPGRADING with deployment_state.phase UPGRADE_IN_PROGRESS, and the site object’s requested version changed to the one posted. Omitting the field returns 400 with version empty in the request, which is how the field name was confirmed.

Plan for hours rather than minutes, and allow retries to resolve. On a default-disk node, the upgrade to crt-20260201-0179 took roughly an hour, briefly reporting UPGRADE_FAILED with result Failed before recovering and completing successfully on the target build. The platform retries failed steps automatically. Consequently, monitoring and automation scripts should treat initial failure reports as transient states rather than final verdicts.

Note the group in that path: these live under config, not operate. The same paths under operate return 404 API Group could not be determined, which is a routing message and not a statement that no upgrade exists — a distinction that cost this project a wrong conclusion.

Direct API upgrades require reconciliation

A direct API call bypasses the Terraform action and status workflow. Verify the installed version independently, then inspect a refreshed plan. A subsequent from-zero deployment with empty version values selects whatever is newest at that later time.

Two earlier claims on this page, withdrawn

This page previously said that moving a CE to a different version means recreating the site, and that a rejected update “still clears the field on the server” leaving Terraform proposing a failing change forever. Neither survived testing: the API upgrade above starts a version change in place, without recreating anything, and after each of the three rejected updates the site’s spec was unchanged and the site stayed ONLINE. The divergence described in provider issue xcsh#1387 was observed on a site that had already auto-upgraded past its pin, so it may need that divergent starting state; it was not reproduced on a site whose spec matched its pin.

If you do rebuild rather than upgrade, every consequence of replacing a CE applies.

Every consequence of replacing a CE applies

New SSH host keys, reassigned internal addresses, a fresh registration, and the loss of anything that lived only on the node. Cloud-init runs again on the new VM, so SSH access configured that way survives — but it runs only on first boot, so anything added to the old node by hand is gone.

Read SSH on the host-key and address consequences before scheduling this.

An undersized disk can make installation fail

Accepting the pin is not the same as installing it. On a freshly created single-node Azure Secure Mesh v2 Customer Edge pinned to crt-20260201-0179, the site object reported the pinned version immediately — and the install then failed:

site_state        PROVISIONING
phase             UPGRADE_FAILED
result            Failed
last_installed    (empty)
message           stage: 10, app: voucher obj: voucher objKind: DaemonSet failed ...
                  required replicas: 1, current replicas: 0

Observed 2026-07-28, and reproduced twice on 2026-07-29. The operating system selection installed normally in the same run (9.2024.6 to 9.2026.14, UPGRADE_COMPLETED); only the software install failed. The site never reached ONLINE and last_installed_version stayed empty, so nothing rolled back to a working build — there was no earlier successful install to roll back to.

A failure at create is not like a failure during an upgrade. The two behave differently and the difference matters when you are deciding whether to intervene:

	at create	during an API upgrade
does it retry into success?	no — held Failed for over 20 minutes, twice	yes — recovered and completed
where does the node end up?	PROVISIONING, nothing installed	ONLINE on a working build
is it safe to leave?	no, it is stuck	yes, it recovers or keeps the old build

A create-time failure requires rebuilding the node with increased disk capacity, whereas an upgrade reporting Failed should be allowed time to retry before intervening.

The cause was the disk, not the version. A matrix of software × OS × disk size, one disposable single-node Azure Secure Mesh v2 site per combination and all from the same marketplace image, isolates it. Observed 2026-07-29:

software	OS 9.2024.6 (what the image ships)	OS 9.2026.14
crt-20250613-3382	installs	installs
crt-20260201-0179	installs	fails, on the default disk only

Neither version fails on its own. Only the pair does, and only on the image’s default disk — the same pair installs on 33 GB and every larger size tested. So the newer build is not unsupported here, and neither is the newer operating system; together they need slightly more disk than a default node has.

terraform/modules/ce-node sets disk_size_gb explicitly. For the pinned f5xc-ce-crt-20260201:20260201.0178.1 Marketplace image, the variable defaults to 80 GB and rejects values below 78 GB. Azure therefore cannot reject the VM for an image-disk mismatch.

The margin is the surprising part, and it is why this looked like a version problem for so long. The default is 31 GiB (the health command reports size_gb: 31, and /var is 29 G with 3.5 G free on a node in the failing state). 33 GB installs cleanly. So the default is short by roughly two gigabytes, not by a wide margin.

Do not set the measured minimum

33 GB is the smallest size observed to work — for this version pair, on this image, today. A build with a slightly larger payload would fail again at 33 GB with no configuration change and no obvious cause, exactly as the default does now. Choose headroom over the floor: the sizes verified here are 33, 36, 40, 48 and 64 GB, and the cost of the larger ones is a few gigabytes of managed disk.

Test a future version change on a disposable site when possible: a fleet that fails this way is stuck in PROVISIONING with recreation as the only exit.

Do not size the disk from F5's pre-upgrade figure

F5’s documented pre-upgrade check wants 20 GB plus 15% of capacity free. That check gates Console UI upgrades and explicitly does not apply to API-initiated ones, so it does not describe this failure. Applied here it predicts a 48 GB disk would fail; 48 GB installs cleanly. Size from the measured outcomes above, not from that formula.

An explanation this page previously offered, now withdrawn

An earlier version of this page suggested the install failed because the requested build was older than the image’s — that the node was asked to move backwards. That is wrong. This fleet was created from the same image with a pin thirteen months older and installed normally, and crt-20260201-0179 installs too when the OS pin is left at the image’s own version. Direction of travel is not the variable.

Which build the documentation describes

The command reference describes the current deployment’s software build, crt-20260201-0179. On 2026-08-03, that build exposed 43 debug API commands and 89 on-box execcli commands. The catalogs record the build and capture timestamp so a later surface change is detected rather than silently described as current. See the command reference and on-box commands.

Latest was verified from zero

The previous deployment used an older software and operating-system pair. The 2026-08-03 rebuild left both version variables empty, selected the advertised pair, brought all three sites ONLINE, and finished with a clean Terraform plan. The earlier installation failures were reproduced only on the undersized marketplace disk described above.