SSH

The on-box examples on this page were captured 2026-07-28. The command catalog was refreshed 2026-08-03 after the from-zero rebuild and records the current node and software build.

SSH reaches the appliance admin account, whose login shell is the Site CLI (/opt/bin/vpmu). This is the only route to the full on-box command surface — the debug API exposes 43 commands, and the appliance itself offers many more that are not on that API at all.

cd terraform
SLI=$(terraform output -json ce_sli_private_ips | jq -r '.eastus01')
ssh -tt -i ~/.ssh/id_ed25519 -J azureuser@<operator-vm> "admin@$SLI"

Every part of that command line is load-bearing, and the next three sections explain which failure each one prevents.

There is no shell prompt to land on: admin’s login shell is the Site CLI, so you arrive at its >>> prompt. Run on-box commands as execcli <name> there — see on-box commands.

Look the internal address up — never carry one over from a previous session

Azure reassigns the internal subnet on every rebuild, and not in node order. After the most recent one f5-xc-ce-vm-01, -02 and -03 hold 10.0.3.7, 10.0.3.5 and 10.0.3.4; a previous build had them in a different order entirely, and the test client’s NIC is in the same subnet and has held an address a CE held before. Connecting to a remembered address therefore lands on the wrong node, on a VM that is not a CE, or nowhere — and the last of those looks identical to “SSH is not enabled”.

terraform output -json ce_sli_private_ips is the mapping. Without state to hand, the NIC name is stable even though the address is not:

az network nic show -g "$(terraform output -raw resource_group_name)" \
  -n "$(terraform output -json ce_vm_names | jq -r '.eastus01')-internal-nic" \
  --query ipConfigurations[0].privateIPAddress -o tsv

Bootstrap-only — an existing CE cannot gain SSH without being recreated

cloud-init write_files runs once, on first boot. Afterwards a /etc/cloud/cloud-init.disabled marker exists on the node and cloud-init does not run again.

So there is no in-place path onto a running CE. custom_data is replace-only in Azure, which means adopting this on an existing deployment replaces every CE VM. Granting operator SSH to nodes that are already up is a rebuild, not a configuration change — and the next warning is what that rebuild used to cost, and why it no longer does.

Replacing a CE VM used to wedge registration permanently — that is fixed, and it is why the site object is coupled to the VM

Worth knowing before you plan a rebuild, because the shape of the failure explains the coupling you will see in the plan.

It used to be that the destroyed node’s registration kept the control plane’s unique (tenant, cluster_name, hostname) index, so the replacement node’s registration was refused with UniqueSecondaryIndexViolationError and retried forever. Deleting the registration was not sufficient on its own: a leftover StatusObject then returned 500 for the workload request, and by that point each node had persisted a registration-obj.yml pinning it to the registration you deleted, so it looped on 404 with no self-recovery. Recovery took two full rebuilds. Throughout all of it terraform plan reported “No changes”, because from Terraform’s point of view the desired state had been reached.

Issue 674 is closed, by PR 693. modules/xc-site now names the CE VM instance in replace_triggered_by, so replacing the VM replaces the site object, and deleting a site takes its registrations with it — the index key is free before the replacement node boots. The fleet has since been rebuilt from scratch and all three sites came up ONLINE without intervention.

The coupling is observable rather than assumed. Each site is bound to a VM instance id, and the matching registration reports the same value as infra.instance_id:

cd terraform && terraform output -json ce_bound_instance_ids

Disagreement between the two means a site is still bound to a node that no longer exists, which is the state a rebuilt CE can never register out of. Reading the registration’s half still needs a direct API call — provider issue 1376 is what would put it on the data source.

Why cloud-init writes the key, and not the API

admin_user_credentials.ssh_key on the site object looks exactly like the field for this. It is accepted, it survives a read-back, and it configures nothing. vpm owns the node’s local users and declines to touch this one — from the node’s own log, three times over a single boot:

vpm  users.go:165: Won't do any change for user admin (internal skip)

Everything else follows from that one line. /var/home/admin/.ssh never exists, before or after the site reaches ONLINE. The admin shadow last-change date stays at the image build date while vesbkp and vesopcon show the current date, because vpm set those two and skipped admin. And vpm logs nothing about ssh_key or authorized_keys at any point.

So the file has to be written outside vpm. admin is uid 2202, baked into the node image, so it exists before cloud-init runs and owner: admin:admin resolves at write time — no runcmd, no ownership fix-up:

- path: /var/home/admin/.ssh/authorized_keys
  permissions: "0600"
  owner: admin:admin
  content: |
    ${ssh_public_key}

sshd was always willing. sshd -T reports pubkeyauthentication yes, and admin appears in AllowUsers in both sshd_config files that ship on the node. There was never anything to enable — only a missing file.

Do not re-add admin_user_credentials, and do not edit sshd_config

admin_user_credentials was wired into terraform/modules/xc-site and reverted once this evidence was collected (#660, PR #663). It is not a gap waiting to be filled.

sshd_config is replaced during registration, so an edit to it is discarded at the least convenient moment.

Why it answers on one address only

sshd binds 0.0.0.0:22, but only the internal (SLI) address is on the host network stack in a way sshd will answer on. eth0 is renamed a-i-eth0 and carries no host IP at all — the Argo data plane owns that interface, and the SLO address it would have carried appears on vhost0 instead. The other two NICs stay on the host stack under their own names.

Probed from a VM inside the VNet, against one CE:

SLO address              timed out
external address         timed out
internal/SLI address     OPEN     SSH-2.0-OpenSSH_8.7
an unused address        timed out    (control)

A probe against the address you know the node by therefore returns exactly what a closed security group returns. Nothing is blocking it; there is no listener on that address.

Do not read the guest interface name to work out which subnet you are on

Which of eth1 and eth2 carries the internal address differs between nodes in this same deployment. Read live from all three CEs:

f5-xc-ce-vm-01    eth1 -> external      eth2 -> internal/SLI
f5-xc-ce-vm-02    eth2 -> external      eth1 -> internal/SLI
f5-xc-ce-vm-03    eth1 -> external      eth2 -> internal/SLI

Only two things hold on every node: a-i-eth0 has no address, and vhost0 carries the SLO address. For anything else, match on the address in ip addr output rather than on the interface name.

The CE public addresses have no listener on 22 either, which is why the command above goes through -J: an operator VM inside the VNet, in the same subnet as the SLI addresses. This deployment builds one for the purpose — terraform output -raw client_vm_name names it.

-J does not pass your options to the jump leg

Options given on the command line apply to the final hop. -o UserKnownHostsFile=/dev/null alongside -J therefore still consults and writes your real known_hosts for the jump host, which is the surprise if you are trying to keep a rebuilt fleet out of it. Spell the first hop out when that matters:

ssh -tt -o ProxyCommand="ssh -W %h:%p -o UserKnownHostsFile=/dev/null azureuser@<operator-vm>" \
    -o UserKnownHostsFile=/dev/null -i ~/.ssh/id_ed25519 "admin@$SLI"

Two measurement mistakes that produce confident wrong answers

Probing eth0, the external interface, or the public address for port 22 reports closed whatever the node’s SSH state, so it cannot tell you anything at all. Only the SLI address answers.

And a probe from a corporate workstation can report a port open that is closed: Netskope completes the TCP handshake itself on common ports. Ports 80 and 443 read as open on the CE public addresses from a workstation and are closed when probed from inside the VNet. Probe from a VM in the VNet.

The Site CLI needs a terminal, and a carriage return

The admin login shell is not a shell. It is a go-prompt application, which puts the terminal into raw mode and reads keystrokes rather than lines. Four consequences, each of which fails in a way that looks like a different problem:

Mechanic	What happens without it
Allocate a terminal (ssh -tt)	panic: no such device or address from go-prompt.NewStandardInputParser, which reads as a crashed appliance
Send a carriage return, not a line feed, for Enter	The line is never submitted, and the session closes on end-of-input having printed nothing
Keep standard input open	The connection ends before the command has rendered, so a working command looks silent
Write the command text and the Enter byte separately	The newline lands in the buffer as a literal character and the CLI answers unknown command, which reads as though the command does not exist

ssh host 'some-command' therefore does not work: the argument is ignored and the interactive prompt starts anyway. Drive it as a terminal or not at all. scripts/sitecli_ssh_harvest.py is the reference implementation.

A panic reading 'no such device or address' means authentication SUCCEEDED

vpmu emits it after login, when it finds no terminal to attach to. Read as a crash it looks like the worst possible outcome, and it is in fact the last error before success. Add -tt and connect again.

A CE's host key changes for two different reasons

The next connection warns REMOTE HOST IDENTIFICATION HAS CHANGED and refuses. Clear the old entry for that address and reconnect. Two things trigger it:

• Replacing the node. The replacement presents a different key at the same address. Since granting SSH replaces the VMs at all, expect this on the very first connection after adopting it.
• An in-place operating-system upgrade, with no replacement at all. Observed 2026-07-28: a node whose ce_os_version upgrade completed presented a new host key at the same address, on the same virtual machine.

StrictHostKeyChecking=accept-new does not help with either. It accepts keys for hosts it has never seen; a changed key is exactly what it refuses. Only removing the stale entry clears it.

Addresses can also move. Recreating a deployment reassigns them from the same subnet, so a CE and a jump host can end up swapping addresses — after which a stale entry is not merely outdated but attached to the wrong machine.

The banner is a free health check

Before you type anything, the login banner has already answered several of the questions you would otherwise spend commands on. From f5-xc-ce-vm-01, ASCII art and the public IP elided:

UNAUTHORIZED ACCESS TO THIS DEVICE IS PROHIBITED
All actions performed on this device are audited

Using https://register.ves.volterra.io
OS: rhel-9.2024.6
Memory: 32768MiB
Storage: sda: 31GiB
CPU: Model: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz | CPUs: 8
Software: crt-20250613-3382
DNS: 168.63.129.16: OK
NTP: Synced
Uptime: 0 days, 5 hours, 20 minutes
Registration Status: PROVISIONED
SLO IP: 10.0.1.4/26
WELCOME IN SITE CLI

Registration Status distinguishes a node still coming up (PROVISIONING) from one that is done (PROVISIONED). Software is the build string that decides which commands exist. DNS and NTP cover the two dependencies that break registration first, so a node that never came online has usually already told you why here — before chronyc-sources or dig are needed at all.

Supplying the keypair

1. Generate a keypair, if you do not already have one. Ed25519 rather than RSA: shorter, and accepted by the appliance sshd.

   ssh-keygen -t ed25519 -C "ce-operator" -f ~/.ssh/id_ed25519

2. Point the deployment at the public half. The root module reads the file once and passes the string down to every node:

   terraform/terraform.tfvars

   ssh_public_key_path = "~/.ssh/id_ed25519.pub"

   ssh_public_key takes the material inline instead, which is what the plan tests use. The private half never leaves your workstation.

3. Apply. Read the warning at the top of this page first — on an existing deployment this replaces the CE VMs.

The VM's own admin_ssh_key is a different thing

azurerm_linux_virtual_machine.admin_ssh_key in modules/ce-node authorises the Azure VM administrator, not the appliance admin account. It grants no Site CLI access, and having it set is not evidence that CE SSH is configured.

What this is not

Each of these was tested and ruled out while the route was still believed impossible. Each is plausible enough to cost a day.

Not the cause	How it was ruled out
The credential only applies at create time	sshd starts about 90 seconds into first boot, before the site object exists. admin_user_credentials is also present in ReplaceSpecType and applies in place.
block_all_services	The port-22 closure on the management address is identical whether services are blocked or not.
A trailing newline on the key	Tested both ways, no change. Cloud-init still strips it, because a literal block would otherwise render a second, empty line.
Pinned, older CE software	Reproduced on three builds, including one running OpenSSH 9.9.
A missing admin_password	Setting admin_password alongside ssh_key on a from-scratch CE changed nothing.

Two API fields that look like discriminators and are not

gc_spec.admin_ssh_key is empty or absent on all 58 sites in f5-sales-demo — including the two that do set admin_user_credentials.ssh_key. Comparing it between a working and a non-working site tells you nothing, because it reads empty even for the sites where the credential is configured.

registrations_by_site omits the field entirely and returns None for it, which reads as “not configured” rather than “not reported”.

The other routes

Debug APIScripted, needs no access inside the VNet, and the right choice for repeatable evidence. Exposes a 43-command catalog and needs the site ONLINE.

Site ConsoleF5's node troubleshooting UI over an Azure Bastion tunnel. No key, no jump host, and it answers before the site registers.

Serial consoleFor a node that never registered, or has no working network path. Needs a terminal, or the websocket harness.

A newly created site needs about half an hour before the debug API answers

For roughly 25 to 30 minutes after a new site reports ONLINE, the vpm/debug endpoints return a BIG-IP 503 page reading The requested URL was rejected. That looks like a web application firewall block, or a revoked token, and is neither: the site is not ready to relay yet. Wait and retry rather than changing anything.

SSH has no such warm-up. It answers as soon as sshd is up, which is why it is the route that works while a node is still registering.