Customer Edge diagnostics

A Customer Edge (CE) is the F5 Distributed Cloud node this repository provisions into Azure — terraform/modules/ce-node, built from the volterraedgeservices/volterra-node marketplace image.

It runs the data plane (Argo), the control plane (Vega), an Envoy proxy, a Kubernetes stack, and vpm, the agent that registers the node and manages everything else on it.

When a CE misbehaves, the Site CLI is the instrument. The first decision is which access path applies, and getting it wrong wastes time in a way that looks like a broken node.

Four access paths, and they are not interchangeable

Debug APIFor a node that is ONLINE. Scriptable, with a 43-command catalog and an enforced capture allow-list.

Site ConsoleF5's own per-node troubleshooting UI, over an Azure Bastion tunnel. No SSH key, no jump host.

SSHThe full on-box Site CLI. Needs a key present from first boot and a host inside the VNet.

Serial consoleFor a node that never registered. The only route when the node has no network path.

The debug API reaches the node through the F5 Distributed Cloud control plane. That is the whole reason the split matters: the API can only answer once the node has registered and reports ONLINE.

A node that failed to register — bad cloud-init, an expired token, no route to register.ves.volterra.io — is exactly the case where you need diagnostics, and exactly the case the API cannot serve.

Site Console is the one to try first when you want F5’s own troubleshooting UI rather than specific commands. It asks least of you — no key, no jump host, no public IP on the node — because who may connect becomes an Azure RBAC decision, and it answers whether or not the site has registered. It needs Azure Bastion, which this deployment gates behind enable_bastion and which is not deployed by default.

SSH is the only route that reaches the appliance’s full command surface — the debug API exposes 43 commands and the node itself offers many more. Two things constrain it. sshd answers on the node’s internal (SLI) address only, so it needs a host inside the VNet; and the key is written by cloud-init at first boot, because the ssh_key field on the site object is inert — vpm skips the admin user and never applies it. Enabling it on running nodes therefore replaces them.

So: if the site is ONLINE and the debug catalog is enough, use the debug API. For a UI, or for a node that never registered but still has a network path, use Site Console. If you need the rest of the command surface and can reach the VNet, use SSH. If the node has no working network path at all, serial console is the only way in.

Check the site state first — and check you are asking the right tenant

One request tells you which path you are on:

curl -sS -H "Authorization: APIToken $XCSH_API_TOKEN" \
  "$XCSH_API_URL/api/config/namespaces/system/sites/<site>" \
  | jq -r '.spec.site_state'

ONLINE means the debug API will work. Anything else means Site Console or serial console.

A site that does not exist has a second explanation before “it never registered”: you are pointed at the wrong tenant. The tenant this deployment belongs to is pinned by var.expected_xc_tenant, and cd terraform && terraform output -raw xc_tenant reports it without touching anything; its console is https://<XC_TENANT>.console.ves.volterra.io. A credential minted for another tenant returns a bare 401, which reads as an expired token.

Safety rules

These hold for every command on every page below.

• Read-only unless you are certain. The command surface is split into two privilege tiers, and the Exec tier either mutates the node or reads a state marker. Nothing on these pages runs an Exec command, and neither does the capture harness.
• Never assume a command is safe from its name. ip-link-set reads like a query and takes an interface down. systemctl-restart-crio restarts the container runtime under a live data plane.
• Prefer the narrowest command that answers the question. health and diagnosis summarise the node cheaply; flow-l dumps every live flow, and flow-l-match answers the same question about one connection.
• A CE serves live traffic. These are shared demo environments. Assume someone is presenting from the site you are debugging.

What is documented here

Every command reachable through the debug API on the software build this tenant runs, each with output captured from a live node rather than transcribed from elsewhere.

The command reference lists all of them with their category, privilege tier and transport; the workflows chain them into the sequences you actually use when something is wrong.

The on-box reference separately records the six top-level commands and 89 execcli subcommands exposed by the same build. scripts/sitecli_ssh_harvest.py drives the appliance’s completion menu over SSH and records the description the appliance gives for each command, so the surface is measured rather than guessed at. Its execution is default-denied: it enumerates everything and runs only an allow-list.

Alongside the command surface, the interface model covers what the three network interface cards on each node actually are: why the Site Local Outside (SLO) interface is the one reserved for reaching F5, what the Secure Mesh Site v2 management network is and the constraints that decide whether you can use it, and why the mgmt NIC here is not an out-of-band management interface.

For architecture rather than diagnostics, Customer Edge high availability separates multi-node clustering from several independent sites joined by BGP and ECMP, then explains route withdrawal, path limits, flow affinity, and application persistence.