A site that never came online

Captured 2026-07-28 from one CE of this deployment. sitecli/capture-manifest.json records which node, and scripts/capture-sitecli.sh --check re-verifies the command surface against a live CE.

The API cannot debug this

The debug API reaches the node through the F5 Distributed Cloud control plane, over the tunnel that registration establishes. If registration failed there is no tunnel, so every command on this site is unavailable for this exact failure. Work from Site Console if the node still has a network path, and from serial console if it does not.

1. Confirm you are asking the right tenant. This costs one command and it is first because getting it wrong makes every later step lie to you: a healthy fleet in the tenant you are not looking at is indistinguishable from a fleet that never registered.

   cd terraform
   terraform output -raw xc_tenant       # the tenant this deployment belongs to
   terraform output -raw xc_env_tenant   # the tenant your shell is exporting

   f5-sales-demo
   f5-sales-demo

   Mismatched, or a credential you believe is good returning a bare 401? That is the symptom of a token minted for another tenant, and it has already cost this deployment one full rebuild (issue 696).

2. Confirm what the tenant thinks. Absent from the site list is a different problem from present but not ONLINE.

   curl -sS -H "Authorization: APIToken $XCSH_API_TOKEN" \
     "$XCSH_API_URL/api/config/namespaces/system/sites" | jq -r '.items[].name'

   If the site exists and reports ONLINE, registration succeeded and you are in the wrong workflow.

3. Find the site’s registration and its state. A node can register and then wait for approval, which looks identical to a failure from the outside.

   curl -sS -H "Authorization: APIToken $XCSH_API_TOKEN" \
     "$XCSH_API_URL/api/register/namespaces/system/registrations_by_site/<site>" \
     | jq -r '.items[] | "\(.name)  \(.object.status.current_state)  \(.object.spec.gc_spec.infra.hostname)"'

   r-a97e20c8-b7e1-483d-bd76-34a64ff8bc78  ONLINE  f5-xc-ce-vm-01

   No items at all means the node has not registered yet — go to the serial console. An item in a state other than ONLINE means it registered and something after that failed, which is a different investigation.

   To sweep the whole namespace for anything waiting on approval, listregistrationsbystate takes the state as a POST body:

   curl -sS -X POST -H "Authorization: APIToken $XCSH_API_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"namespace":"system","state":"PENDING"}' \
     "$XCSH_API_URL/api/register/namespaces/system/listregistrationsbystate" | jq -r '.items[].name'

   On a healthy fleet that returns nothing, because approval is automated.

4. Confirm the VM is actually running, before assuming a software fault.

   az vm list -d -g <resource-group> --query "[].{name:name,power:powerState}" -o table

   Name              Power
   ----------------  ----------
   f5-xc-ce-vm-01    VM running
   f5-xc-ce-vm-02    VM running
   f5-xc-ce-vm-03    VM running
   mcn-ce-ha-client  VM running

5. Attach the serial console. Interactively this needs a real terminal; the attach itself can be scripted over the websocket instead — see serial console.

   az serial-console connect -g <resource-group> -n <vm-name>

   If it refuses, check boot diagnostics is enabled on the VM before concluding the node is unreachable; that prerequisite and its failure modes are covered on the serial console page.

6. Read what the console shows, in this order. cloud-init first — a node whose cloud-init never completed has no configuration to register with. Then the registration attempts themselves, then DNS: if the node cannot resolve register.ves.volterra.io, nothing downstream can work.

7. Once the node is ONLINE, verify with health and expect state: PROVISIONED.

A registration is named for a UUID, never after its site

Registrations are named r-<uuid>, and there is no endpoint that takes a site name where a registration name is expected. GET .../registrations/<site> returns 404, which reads as “this site has no registration” and is really “that is not a registration name”. The site-to-registration mapping only comes from registrations_by_site/<site>, as in step 3, or by listing registrations and matching on gc_spec.passport.cluster_name.

The same trap made it look as though a healthy fleet had zero registrations.

The usual causes

• You are pointed at the wrong tenant, so a fleet that is fine looks absent. Step 1.
• cloud-init did not complete, so /etc/vpm/config.yaml is absent or wrong.
• The registration token is expired, already consumed, or from another tenant.
• No egress route to register.ves.volterra.io.
• Clock skew large enough to invalidate certificates — quick to rule out later with chronyc-sources, but not reachable until the node is online. The SSH login banner reports NTP: Synced and the resolver’s status before you run anything, when SSH is available.