Diagnostics & Verification

This page provides a layered UAT verification matrix for validating your Client-Side Defense deployment end-to-end. Each test case follows the infrastructure dependency chain — from DNS resolution through CSD telemetry — so you can systematically prove every component is working correctly.

These commands are the API equivalent of the CSD Console Walkthrough — use them when you need to verify from the terminal, automate monitoring, or demonstrate CSD capabilities without the UI.

Prerequisites

Set up your environment variables as described in API Automation — Environment Setup:

set -a && source .env && set +a

All commands below use the xTOKENx placeholder format. Substitute with your environment variables ($XCSH_API_TOKEN, $XCSH_NAMESPACE, etc.) or use the interactive form at the top of the page.

Caution

Choose one ownership mode. Never run the API create/update/delete workflow against resources present in Terraform state.

The commands on this page are read-only observations unless a remediation step explicitly says otherwise. For Terraform-owned resources, remediate in Terraform, review a fresh saved plan, and apply it against the same backend and state. Do not use API mutation as a fallback.

API Behavior Notes

Note

PUT and DELETE return empty \{\} on HTTP 200 — this is the expected success response, not an error. Only POST returns the created object.

List endpoints (e.g., /healthchecks) and individual GET endpoints (e.g., /healthchecks/\{name\}) return different response structures. See API Reference — API Conventions for the correct jq paths.

Time Range Helpers

Many CSD endpoints require epoch timestamps (seconds since Unix epoch). These one-liners compute start/end times for common ranges.

Cross-platform (macOS + Linux):

# Current time as epoch seconds
NOW=$(date +%s)

# 1 hour ago
START_1H=$(( NOW - 3600 ))

# 24 hours ago
START_24H=$(( NOW - 86400 ))

# 7 days ago
START_7D=$(( NOW - 604800 ))

# 30 days ago
START_30D=$(( NOW - 2592000 ))

Preset	Seconds	Shell expression
1 hour	3,600	$(( $(date +%s) - 3600 ))
24 hours	86,400	$(( $(date +%s) - 86400 ))
7 days	604,800	$(( $(date +%s) - 604800 ))
30 days	2,592,000	$(( $(date +%s) - 2592000 ))

Note

CSD endpoints (/api/shape/csd/) use epoch seconds (e.g., 1710000000). Access log endpoints (/api/data/) use ISO 8601 timestamps (e.g., 2024-03-10T00:00:00Z). The commands in each section use the correct format for that endpoint.

Test Case Format

Each test below follows this structure:

Field	Description
Test ID	Layer number + sequential ID (e.g., DNS-1, TLS-2)
What it proves	The specific infrastructure fact being verified
Command	Ready-to-run curl or dig command
PASS / FAIL	Expected output for healthy vs. unhealthy state
Fix	Link to the relevant setup or troubleshooting section

---

Layer 1: DNS Resolution

DNS is the foundation — if the domain does not resolve to the load balancer VIP, nothing else works.

DNS-1: A Records Match the Current LB VIPs

What it proves: Every A record for the exact protected domain resolves to a VIP currently reported by the exact load balancer in the expected namespace.

LB=$(curl -fsS \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/http_loadbalancers/xXCSH_LB_NAMEx")
DNS_A=$(dig +short xXCSH_DOMAINNAMEx A | sort -u)
LB_VIPS=$(printf '%s' "$LB" | jq -r '.spec.dns_info[]?.ip_address // empty' | sort -u)

printf '%s' "$LB" | jq -e \
  --arg name "xXCSH_LB_NAMEx" \
  --arg namespace "xXCSH_NAMESPACEx" \
  --arg domain "xXCSH_DOMAINNAMEx" \
  '.metadata.name == $name and .metadata.namespace == $namespace and (.spec.domains | index($domain) != null)' >/dev/null
test -n "$DNS_A"
test "$DNS_A" = "$LB_VIPS"

Result	Meaning
PASS — identity/domain assertion succeeds and the non-empty sorted A-record set exactly equals the current LB VIP set	DNS names the intended LB and no stale or foreign VIP is present
FAIL — identity/domain mismatch, empty set, or unequal sets	Correct the domain, namespace, LB selection, or DNS records before continuing

Note

A returned IP by itself is not proof: it must equal the current VIP set on the named LB. If the domain previously returned NXDOMAIN, repeat the A query against the authoritative nameserver to distinguish stale recursive cache from authoritative state.

DNS-2: ACME Challenge Records

What it proves: The exact challenge owner has the record type and value required by the selected DNS ownership mode.

ACME_OWNER="_acme-challenge.xXCSH_DOMAINNAMEx"
ACME_CNAME=$(dig +short "$ACME_OWNER" CNAME)
ACME_TXT=$(dig +short "$ACME_OWNER" TXT)
printf 'owner=%s\nCNAME=%s\nTXT=%s\n' "$ACME_OWNER" "$ACME_CNAME" "$ACME_TXT"

Result	Meaning
PASS (external DNS) — the exact owner has the configured F5 automatic-certificate CNAME target	External ACME delegation is present
PASS (F5 Distributed Cloud managed DNS) — the exact owner has the platform-managed TXT value and no conflicting external CNAME	Managed ACME validation is present
FAIL — expected value missing, wrong owner, conflicting records, or both empty	Certificate validation is not proven

Do not accept any non-empty CNAME or TXT as PASS. Compare the observed value with the value reported for this domain by the current LB certificate metadata or the DNS configuration. For external DNS, create the exact CNAME instructed by that metadata. For managed DNS, enable allow_http_lb_managed_records and verify the platform-managed TXT record.

DNS-3: Nameserver Authority

What it proves: Whether F5 XC is the authoritative DNS provider for the root domain.

dig +short NS xXCSH_ROOT_DOMAINx

Result	Meaning
Includes ns1.f5clouddns.com and ns2.f5clouddns.com	F5 XC managed DNS — records can be auto-created
Other nameservers	External DNS — records must be created manually

Fix: Phase 1 — DNS, Certificate, and Origin Recovery

DNS-4: F5 XC Managed Records Enabled

What it proves: The F5 XC DNS zone allows automatic record creation for load balancers (F5 XC managed DNS only).

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/dns/namespaces/system/dns_zones/xXCSH_ROOT_DOMAINx" \
  | jq '.spec.primary.allow_http_lb_managed_records'

Result	Meaning
PASS — true	Platform will auto-create A and ACME records for LBs
FAIL — false or null	Managed records disabled; enable via zone update

Caution

A false result is a common misconfiguration when using F5 XC managed DNS. Without managed records enabled, the platform cannot create the A record pointing to the VIP or the ACME challenge record. The load balancer then remains in VIRTUAL_HOST_PENDING_A_RECORD and the certificate remains in PreDomainChallengePending.

Enable managed records on the DNS zone or create the records manually.

Fix: Phase 1 — DNS, Certificate, and Origin Recovery

---

Layer 2: TLS Certificate

The single load balancer named by XCSH_LB_NAME terminates HTTPS and redirects HTTP to HTTPS. HTTP is redirect behavior on this object, not a second load balancer or a fallback path.

TLS-1: Certificate State

What it proves: The automatic TLS certificate has been issued on the load balancer.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/http_loadbalancers/xXCSH_LB_NAMEx" \
  | jq '.spec.cert_state'

Result	Meaning
PASS — "CertificateValid" or "AutoCertRenewing"	Certificate is valid and active
WAIT — "DomainChallengePending" or "DomainChallengeStarted"	ACME challenge is still progressing
FAIL — "PreDomainChallengePending" or "AutoCertDomainRateLimited"	HTTPS is not ready; resolve the ACME record or rate limit before end-to-end validation

TLS-2: Certificate Details

What it proves: The certificate metadata and ACME records correspond to the configured domain.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/http_loadbalancers/xXCSH_LB_NAMEx" \
  | jq '{
    cert_state: .spec.cert_state,
    auto_cert_info: .spec.auto_cert_info,
    vip_ip: .spec.dns_info[0].ip_address
  }'

TLS-3: Live TLS Handshake

What it proves: A client can complete TLS to the protected domain with a valid certificate.

curl -sv "https://xXCSH_DOMAINNAMEx/" 2>&1 \
  | grep -E 'SSL connection|subject:|expire date:|issuer:'

Result	Meaning
PASS — TLS connection, expected subject, and valid dates	Client-to-F5 XC TLS is ready
FAIL — connection or certificate error	Check DNS, LB virtual-host state, and certificate state

---

Layer 3: HTTP Load Balancer

All checks in this layer read the same unsuffixed load balancer object.

LB-1: Operational State

What it proves: The virtual host is deployed and ready.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/http_loadbalancers/xXCSH_LB_NAMEx" \
  | jq '.spec.state'

Result	Meaning
PASS — "VIRTUAL_HOST_READY"	LB virtual host is operational
FAIL — pending state	Inspect DNS, certificate state, and virtual-host error details

LB-2: Domain and HTTPS Redirect Configuration

What it proves: The LB serves the expected domain with automatic HTTPS and HTTP redirect enabled.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/http_loadbalancers/xXCSH_LB_NAMEx" \
  | jq '{domains: .spec.domains, https_auto_cert: .spec.https_auto_cert, http_redirect: .spec.https_auto_cert.http_redirect}'

Result	Meaning
PASS — expected domain and http_redirect: true	One LB owns HTTPS and redirects HTTP
FAIL — missing domain, HTTPS, or redirect	Remediate through the selected ownership mode

LB-3: CSD Enabled on LB

What it proves: Client-Side Defense injects JavaScript on all pages.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/http_loadbalancers/xXCSH_LB_NAMEx" \
  | jq '{
    csd_enabled: (if .spec.client_side_defense then true else false end),
    js_policy: .spec.client_side_defense.policy
  }'

Result	Meaning
PASS — csd_enabled: true with js_insert_all_pages	CSD injection is configured for all pages
FAIL — absent CSD configuration	Remediate through the selected ownership mode

LB-4: Default Route Pool

What it proves: The LB references the expected origin pool.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/http_loadbalancers/xXCSH_LB_NAMEx" \
  | jq '.spec.default_route_pools[] | {pool: .pool.name, namespace: .pool.namespace, weight, priority}'

LB-5: Deployment Status

What it proves: The virtual host, VIP, certificate, and route configuration are reported by the same LB object.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/http_loadbalancers/xXCSH_LB_NAMEx" \
  | jq '{
    name: .metadata.name,
    state: .spec.state,
    cert_state: .spec.cert_state,
    domains: .spec.domains,
    dns_info: .spec.dns_info,
    http_redirect: .spec.https_auto_cert.http_redirect,
    csd_enabled: (if .spec.client_side_defense then true else false end),
    route_pools: [.spec.default_route_pools[] | .pool.name],
    virtual_host_errors: [.status[]? | select(.virtual_host_status != null) | .virtual_host_status | {state, error_description, suggested_action}]
  }'

---

Layer 4: Origin Pool

The origin pool supports either an AWS ALB hostname (public_name) or an Azure/public lab IP (public_ip). Set XCSH_ORIGIN_KIND to public_name or public_ip; set the matching XCSH_ORIGIN_HOSTNAME or XCSH_ORIGIN_IP variable.

OP-1: Origin Pool Configuration

What it proves: The configured oneOf origin kind and value match the intended backend.

ORIGIN_POOL=$(curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/origin_pools/xXCSH_ORIGIN_POOLx")

case "xXCSH_ORIGIN_KINDx" in
  public_name)
    echo "$ORIGIN_POOL" | jq --arg expected "xXCSH_ORIGIN_HOSTNAMEx" '{
      name: .metadata.name,
      origin_kind: "public_name",
      origin: [.spec.origin_servers[] | .public_name.dns_name],
      expected: $expected,
      port: .spec.port
    }'
    ;;
  public_ip)
    echo "$ORIGIN_POOL" | jq --arg expected "xXCSH_ORIGIN_IPx" '{
      name: .metadata.name,
      origin_kind: "public_ip",
      origin: [.spec.origin_servers[] | .public_ip.ip],
      expected: $expected,
      port: .spec.port
    }'
    ;;
  *) echo "FAIL: XCSH_ORIGIN_KIND must be public_name or public_ip" >&2; exit 1 ;;
esac

Result	Meaning
PASS — origin contains the matching expected hostname or IP and the expected port	Origin pool oneOf choice is correct
FAIL — wrong kind, value, or port	Remediate the origin pool through its selected owner

OP-2: TLS to Origin

echo "$ORIGIN_POOL" | jq '{
  tls_config: (if .spec.no_tls then "no_tls (plaintext)" elif .spec.use_tls then "use_tls (encrypted)" else "unknown" end)
}'

OP-3: Healthcheck Association

echo "$ORIGIN_POOL" | jq '.spec.healthcheck'

An empty array is valid when no optional healthcheck was selected.

OP-4: Direct Origin Observation

What it proves: The origin responds from the operator’s current network. It does not prove F5 XC-to-origin health.

case "xXCSH_ORIGIN_KINDx" in
  public_name) ORIGIN_URL="http://xXCSH_ORIGIN_HOSTNAMEx:xXCSH_ORIGIN_PORTx/" ;;
  public_ip)   ORIGIN_URL="http://xXCSH_ORIGIN_IPx:xXCSH_ORIGIN_PORTx/" ;;
  *) echo "FAIL: invalid XCSH_ORIGIN_KIND" >&2; exit 1 ;;
esac

curl -sS -o /dev/null -w '%{http_code}\n' "$ORIGIN_URL"

Note

A successful direct request proves only client-to-origin reachability. F5 XC may use a different path, source address, DNS resolver, or policy. Use LB access logs and the protected HTTPS request to prove the F5 XC-to-origin path.

---

Layer 5: Health Check

Health checks monitor backend availability. They are optional for CSD but useful for production deployments.

HC-1: Healthcheck Configuration

What it proves: The healthcheck exists with the correct type, path, timing, and thresholds.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/healthchecks/xXCSH_HC_NAMEx" \
  | jq '{
    name: .metadata.name,
    type: (if .spec.http_health_check then "HTTP" elif .spec.tcp_health_check then "TCP" else "unknown" end),
    path: .spec.http_health_check.path,
    expected_status: .spec.http_health_check.expected_status_codes,
    timeout: .spec.timeout,
    interval: .spec.interval,
    unhealthy_threshold: .spec.unhealthy_threshold,
    healthy_threshold: .spec.healthy_threshold
  }'

Result	Meaning
PASS — shows HTTP type with path / and expected 200	Healthcheck configured correctly
FAIL — 404 response	Healthcheck does not exist (it may have been omitted; see Phase 1 Step 3)

HC-2: List All Healthchecks

What it proves: Enumerates all healthchecks in the namespace to verify naming and count.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/healthchecks" \
  | jq -r '
    ["NAME", "NAMESPACE", "DESCRIPTION"],
    (.items[] | [
      .name,
      .namespace,
      (.description | if length == 0 then "—" else . end)
    ])
    | @tsv' | column -t

Note

The list endpoint returns summary data only (name, namespace, description). To see full configuration details (type, path, interval, timeout), query an individual healthcheck using HC-1.

---

Layer 6: CSD Configuration

CSD must be enabled at the tenant level and have the JavaScript injection tag configured.

CSD-1: Tenant CSD Status

What it proves: CSD is configured and enabled for the tenant.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/status" \
  | jq '{isConfigured, isEnabled}'

Result	Meaning
PASS — both true	CSD is active for this tenant
FAIL — isConfigured: false	CSD not enabled at tenant level — contact F5 XC administrator
FAIL — isEnabled: false	CSD configured but not active

CSD-2: JS Injection Tag

What it proves: The CSD JavaScript injection tag is generated and ready for injection.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/js_configuration" \
  | jq '{has_script_tag: (.scriptTag | length > 0)}'

Result	Meaning
PASS — has_script_tag: true	JS injection tag is configured
FAIL — has_script_tag: false	No script tag — verify CSD is enabled and a protected domain is registered

CSD-3: JS Tag Content

What it proves: Shows the full script tag for verification or manual injection.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/js_configuration" \
  | jq '.scriptTag'

CSD-4: Protected Domain Registration

What it proves: The exact protected-domain name and namespace match the expected application.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/protected_domains" \
  | jq -e --arg name "xXCSH_DOMAINNAMEx" --arg namespace "xXCSH_NAMESPACEx" \
    '.items[] | select(.name == $name and .namespace == $namespace) | {name, namespace, description}'

Result	Meaning
PASS — an item exactly matches both XCSH_DOMAINNAME and XCSH_NAMESPACE	The intended domain is registered in the intended namespace
FAIL — no exact match	A different or merely non-empty item does not prove protection for this application

CSD-5: Live Application and JS Injection Verification

What it proves: The protected HTTPS response is the expected scenario application and contains the injected CSD path. XCSH_CSD_DEPLOYMENT_MODE selects ownership (api or terraform) only; it never selects AWS or Azure evidence.

case "xXCSH_CSD_DEPLOYMENT_MODEx" in
  api|terraform) ;;
  *) echo "FAIL: XCSH_CSD_DEPLOYMENT_MODE must be api or terraform" >&2; exit 1 ;;
esac

APPLICATION_MARKER='xXCSH_APPLICATION_MARKERx'
test -n "$APPLICATION_MARKER" || { echo "FAIL: XCSH_APPLICATION_MARKER is required for every scenario" >&2; exit 1; }
case "$APPLICATION_MARKER" in
  'replace-with-application-marker'|'replace-with-azure-application-marker'|'xXCSH_APPLICATION_MARKERx')
    echo "FAIL: replace the XCSH_APPLICATION_MARKER documentation placeholder" >&2
    exit 1
    ;;
esac

APPLICATION_HTML=$(curl -fsS "https://xXCSH_DOMAINNAMEx/")
printf '%s' "$APPLICATION_HTML" | grep -Fqi "$APPLICATION_MARKER" \
  && echo "PASS: expected application rendered" \
  || { echo "FAIL: expected application marker missing" >&2; exit 1; }
printf '%s' "$APPLICATION_HTML" | grep -q '__imp_apg__' \
  && echo "PASS: __imp_apg__ injected" \
  || { echo "FAIL: __imp_apg__ missing" >&2; exit 1; }

Both assertions must succeed. XCSH_APPLICATION_MARKER always identifies the expected application/scenario independently of XCSH_ORIGIN_KIND. The repository reference defaults to OWASP Juice Shop; every non-reference API or Azure scenario must supply its own stable, deployment-specific marker. Empty and documentation-placeholder values fail closed. XCSH_CSD_DEPLOYMENT_MODE continues to identify ownership only.

An HTTPS status alone does not identify the intended origin, and direct-origin reachability does not prove the F5 path.

---

Layer 7: Traffic Verification

Verify that live traffic is reaching the load balancer and being processed correctly.

TV-1: Request Count (Last 24 Hours)

What it proves: Traffic is reaching the load balancer. Zero results means no traffic is arriving.

curl -s -X POST \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  -H "Content-Type: application/json" \
  -d '{
    "start_time": "'"$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-24H +%Y-%m-%dT%H:%M:%SZ)"'",
    "end_time": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'"
  }' \
  "xXCSH_API_URLx/api/data/namespaces/xXCSH_NAMESPACEx/access_logs/aggregation" \
  | jq '{total_requests: .total_hits}'

Result	Meaning
PASS — total_requests is a non-zero string (e.g., "380")	Traffic is flowing through the LB
FAIL — "0" or no data	No traffic reaching the LB — check DNS-1 and LB-1

Note

The total_hits value is returned as a string (e.g., "380"), not an integer. The namespace is inferred from the URL path — do not include it in the request body.

TV-2: Status Code Distribution

What it proves: The breakdown of response status codes reveals error patterns.

curl -s -X POST \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  -H "Content-Type: application/json" \
  -d '{
    "start_time": "'"$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-24H +%Y-%m-%dT%H:%M:%SZ)"'",
    "end_time": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'",
    "sort": "DESCENDING",
    "limit": 100
  }' \
  "xXCSH_API_URLx/api/data/namespaces/xXCSH_NAMESPACEx/access_logs" \
  | jq -r '
    [.logs[] | fromjson | .rsp_code_class]
    | group_by(.) | map({class: .[0], count: length})
    | sort_by(-.count)
    | ["STATUS_CLASS", "COUNT"], (.[] | [.class, .count])
    | @tsv' | column -t

Note

The aggregation endpoint does not support terms bucketing — aggs always returns empty. This command fetches the most recent 100 log entries and computes the distribution client-side. Each log entry is a JSON string that must be parsed with fromjson.

Expected output for a healthy site:

STATUS_CLASS                  COUNT
2xx                           82
downstream_remote_disconnect  18

Result	Meaning
PASS — majority 2xx	Site is serving successful responses
WARN — high 4xx count	Client errors (bad URLs, missing resources)
FAIL — high 5xx count	Server errors — check origin server health

The downstream_remote_disconnect class indicates the client closed the connection before the response completed (common for long-polling or WebSocket upgrade requests).

TV-3: Recent Application Log Event

What it proves: A recent request for the protected domain was recorded after the validation request.

VALIDATION_START=$(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-5M +%Y-%m-%dT%H:%M:%SZ)
curl -fsS "https://xXCSH_DOMAINNAMEx/" >/dev/null
curl -s -X POST \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  -H "Content-Type: application/json" \
  -d '{
    "start_time": "'"$VALIDATION_START"'",
    "end_time": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)'",
    "sort": "DESCENDING",
    "limit": 100
  }' \
  "xXCSH_API_URLx/api/data/namespaces/xXCSH_NAMESPACEx/access_logs" \
  | jq -e --arg domain "xXCSH_DOMAINNAMEx" \
    '[.logs[] | fromjson | select((.authority // .host // .req_host) == $domain)] | first | {timestamp: .["@timestamp"], method, path: .req_path, status: .rsp_code}'

PASS requires a matching event inside this time window for the exact protected host. Old namespace traffic or an unrelated host is not application evidence. Field availability can vary by log schema; inspect one parsed event and select the documented host field if none of authority, host, or req_host is present.

TV-4: CSD JS Injection in Access Logs

What it proves: Access logs confirm CSD JavaScript is being injected into responses.

curl -s -X POST \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  -H "Content-Type: application/json" \
  -d '{
    "start_time": "'"$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)"'",
    "end_time": "'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'",
    "sort": "DESCENDING",
    "limit": 20
  }' \
  "xXCSH_API_URLx/api/data/namespaces/xXCSH_NAMESPACEx/access_logs" \
  | jq '[.logs[] | fromjson | select(.csd_js_injection == "true")] | length as $injected |
    {injected_count: $injected, total_sampled: 20}'

Result	Meaning
PASS — injected_count > 0	CSD JS is being injected into page responses
FAIL — injected_count: 0	JS not injected — check CSD-1 and LB-3

Note

The csd_js_injection field appears only on entries where injection occurred. One lab sample observed 40% injected entries and 60% entries without the field; those percentages are an observation, not a target or product guarantee. Validate HTML page requests rather than expecting a fixed ratio across static assets, API calls, or WebSocket traffic.

TV-5: End-to-End Protocol and Telemetry Proof

What it proves: HTTP redirects to the single HTTPS virtual host, HTTPS serves the application through F5 XC, and the browser emits CSD telemetry.

curl -sS -o /dev/null -D - "http://xXCSH_DOMAINNAMEx/" \
  | grep -E '^HTTP/|^[Ll]ocation:'
curl -sS -o /dev/null -w 'HTTPS %{http_code}\n' \
  "https://xXCSH_DOMAINNAMEx/"

Result	Meaning
PASS — HTTP 301 with an HTTPS Location, then HTTPS 200	Redirect and protected application path are operational
FAIL — any other result	Check the one LB’s redirect, certificate, route pool, and origin health

Then open https://xXCSH_DOMAINNAMEx/ in a browser, use DevTools Network, filter for dip, and confirm a request from the page to the F5-operated CSD collection endpoint. The dip browser request proves client-side telemetry execution; curl alone cannot prove it.

---

Layer 8: CSD Telemetry & Domain Policy

These commands query the same data displayed in the CSD Console dashboard, script list, form fields, and network views.

TEL-1: Script Inventory

What it proves: CSD is detecting and cataloging scripts running on the protected domain.

NOW=$(date +%s)
START=$(( NOW - 604800 ))

curl -s -X POST \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  -H "Content-Type: application/json" \
  -d "{
    \"startTime\": \"${START}\",
    \"endTime\": \"${NOW}\"
  }" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/scripts" \
  | jq -r '
    ["SCRIPT", "RISK", "STATUS", "FIELDS", "USERS"],
    (.scripts[] | [
      (.script_name | if length > 50 then .[:47] + "..." else . end),
      .risk_level,
      .status,
      (.form_fields_read // 0),
      (.affected_users_count // 0)
    ])
    | @tsv' | column -t

Result	Meaning
PASS — scripts listed with risk levels	CSD is actively monitoring scripts
FAIL — empty array	See Troubleshooting: Empty Scripts Array

Tip

Script names are full URLs and can be very long. The command above truncates names to 50 characters for terminal readability. Remove the truncation filter to see full URLs.

TEL-2: Detected Domains

What it proves: CSD has detected script source domains and classified them by status.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/detected_domains" \
  | jq '{
    summary: {
      action_needed: .domain_summary.actionNeededCount.count,
      mitigated: .domain_summary.mitigatedDomains.count,
      allowed: .domain_summary.allowedDomains.count,
      total: .domain_summary.totalDomains.count
    },
    domains: [.domains_list[] | {
      domain: .domain,
      category: .category,
      status: .status,
      first_seen: (.firstSeenDate | tonumber | todate),
      latest_seen: (.latestSeenDate | tonumber | todate)
    }]
  }'

Result	Meaning
PASS — total > 0 with domains listed	CSD is tracking script domains
WARN — action_needed > 0	Some domains need review
FAIL — empty response	No telemetry data; check CSD-5

TEL-3: Domain Policy State

What it proves: Shows the distribution of allowed vs. mitigated domains and whether policy is consistent.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/detected_domains" \
  | jq '{
    action_needed: .domain_summary.actionNeededCount.count,
    mitigated: .domain_summary.mitigatedDomains.count,
    allowed: .domain_summary.allowedDomains.count,
    total: .domain_summary.totalDomains.count,
    domains_needing_action: [.domains_list[] | select(.status == "AN") | .domain]
  }'

Result	Meaning
PASS — action_needed: 0	All domains have been reviewed and classified
WARN — action_needed > 0	Domains listed in domains_needing_action require review

TEL-4: Form Field Inventory

What it proves: CSD has detected form fields that scripts are reading, with sensitivity classification.

NOW=$(date +%s)
START=$(( NOW - 604800 ))

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/formFields?startTime=${START}&endTime=${NOW}" \
  | jq -r '
    ["FIELD", "SENSITIVITY", "SCRIPTS"],
    (.form_fields[] | [
      .name,
      .analysis,
      (.scripts_count // 0)
    ])
    | @tsv' | column -t

Result	Meaning
PASS — form fields listed with sensitivity	CSD is tracking form field access
INFO — empty array	No form fields detected (expected if no forms on the site)

TEL-5: Script Deep Dive

What it proves: Detailed information about a specific script including risk, behaviors, and network interactions.

First, get a script ID from TEL-1, then query its details:

SCRIPT_ID="your-script-id"

# Overview (risk level, type, source domain)
curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/scripts/${SCRIPT_ID}/dashboard" \
  | jq '{script_name, risk_level, type, source_domain, status}'

# Behaviors over time
curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/scripts/${SCRIPT_ID}/behaviors" \
  | jq '.behaviors'

# Network interactions (domains the script communicates with)
curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/scripts/${SCRIPT_ID}/networkInteractions" \
  | jq '.network_interactions'

TEL-6: Affected Users

What it proves: Lists users impacted by a specific script, showing the scope of exposure.

SCRIPT_ID="your-script-id"
NOW=$(date +%s)
START=$(( NOW - 604800 ))

curl -s -X POST \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  -H "Content-Type: application/json" \
  -d "{
    \"startTime\": \"${START}\",
    \"endTime\": \"${NOW}\"
  }" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/scripts/${SCRIPT_ID}/affectedUsers" \
  | jq -r '
    ["IP_ADDRESS", "DEVICE_ID", "GEO", "CHANNEL", "USER_AGENT"],
    (.affected_users[] | [
      .ip_address,
      (.device_id | if length > 12 then .[:12] + "..." else . end),
      .geolocation,
      .channel,
      (.user_agent | if length > 30 then .[:27] + "..." else . end)
    ])
    | @tsv' | column -t

TEL-7: Script Count by Risk Level

What it proves: Aggregated risk distribution across all detected scripts.

NOW=$(date +%s)
START=$(( NOW - 604800 ))

curl -s -X POST \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  -H "Content-Type: application/json" \
  -d "{
    \"startTime\": \"${START}\",
    \"endTime\": \"${NOW}\"
  }" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/scripts" \
  | jq '[.scripts[] | .risk_level] | group_by(.) | map({risk_level: .[0], count: length}) | sort_by(-.count)'

Result	Meaning
PASS — all No Risk	No risky scripts detected
WARN — Low Risk or High Risk present	Review flagged scripts in TEL-5

TEL-8: Telemetry Freshness

What it proves: CSD telemetry data is recent, confirming active monitoring.

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/detected_domains" \
  | jq '{
    total_domains: .domain_summary.totalDomains.count,
    latest_update: (.domain_summary.totalDomains.lastUpdated // "unknown"),
    most_recent_domain: (.domains_list | sort_by(.latestSeenDate) | last | {
      domain: .domain,
      latest_seen: (.latestSeenDate | tonumber | todate)
    })
  }'

Result	Meaning
PASS — latest_seen is within the last 24 hours	Telemetry is actively collecting data
WARN — latest_seen is older than 24 hours	Traffic may have stopped or CSD processing is delayed

---

Cloud-Origin and Ownership Evidence

Run only the subsection matching the selected cloud scenario and owner. XCSH_CSD_DEPLOYMENT_MODE=api|terraform selects F5 resource ownership; it is not a cloud-provider selector. Cloud-side health is supporting origin evidence; the common DNS, F5 Distributed Cloud, TLS, protected HTTPS, access-log, and CSD checks above remain mandatory.

Terraform-Owned AWS Origin and Log Evidence

Use this subsection only for the Terraform-owned AWS reference stack. It observes current Terraform state plus AWS origin and delivery evidence; it does not prove the protected F5 path by itself.

TF_DIR="$(git rev-parse --show-toplevel)/terraform/aws"
TF_JSON=$(terraform -chdir="$TF_DIR" show -json)
AWS_REGION=$(terraform -chdir="$TF_DIR" output -raw aws_region)
VPC_ID=$(terraform -chdir="$TF_DIR" output -raw vpc_id)
TARGET_GROUP_ARN=$(terraform -chdir="$TF_DIR" output -raw target_group_arn)
KMS_KEY_ARN=$(terraform -chdir="$TF_DIR" output -raw cloudwatch_logs_kms_key_arn)
ALB_LOG_BUCKET=$(terraform -chdir="$TF_DIR" output -raw alb_access_logs_bucket)
ALB_LOG_PREFIX=$(terraform -chdir="$TF_DIR" output -raw alb_access_logs_prefix)
APPLICATION_URL=$(terraform -chdir="$TF_DIR" output -raw application_url)
XCSH_LB_NAME=$(terraform -chdir="$TF_DIR" output -raw xc_http_loadbalancer_name)
XCSH_ORIGIN_POOL=$(terraform -chdir="$TF_DIR" output -raw xc_origin_pool_name)

ECS_CLUSTER=$(printf '%s' "$TF_JSON" | jq -r '.. | objects | select(.address? == "module.origin.aws_ecs_cluster.this") | .values.name')
ECS_SERVICE=$(printf '%s' "$TF_JSON" | jq -r '.. | objects | select(.address? == "module.origin.aws_ecs_service.this") | .values.name')
APP_LOG_GROUP=$(printf '%s' "$TF_JSON" | jq -r '.. | objects | select(.address? == "module.origin.aws_cloudwatch_log_group.this") | .values.name')
FLOW_LOG_GROUP=$(printf '%s' "$TF_JSON" | jq -r '.. | objects | select(.address? == "aws_cloudwatch_log_group.vpc_flow") | .values.name')

Do not copy identifiers from a prior run. The root values come from Terraform outputs; module-only runtime identifiers are read from the current state at execution time.

AWS-1: ECS Steady State

aws ecs wait services-stable \
  --region "$AWS_REGION" \
  --cluster "$ECS_CLUSTER" \
  --services "$ECS_SERVICE"
aws ecs describe-services \
  --region "$AWS_REGION" \
  --cluster "$ECS_CLUSTER" \
  --services "$ECS_SERVICE" \
  --query 'services[0].{status:status,desired:desiredCount,running:runningCount,pending:pendingCount}'

PASS requires status ACTIVE, running equal to desired, and pending zero.

AWS-2: Healthy ALB Target

aws elbv2 describe-target-health \
  --region "$AWS_REGION" \
  --target-group-arn "$TARGET_GROUP_ARN" \
  --query 'TargetHealthDescriptions[].TargetHealth.{state:State,reason:Reason,description:Description}'

At least one target must be healthy. This is AWS-side target health; the protected HTTPS test remains necessary to prove the F5 XC path.

AWS-3: KMS-Encrypted CloudWatch and Flow Logs

aws logs describe-log-groups \
  --region "$AWS_REGION" \
  --log-group-name-prefix "$APP_LOG_GROUP" \
  --query 'logGroups[?logGroupName==`'"$APP_LOG_GROUP"'`].{name:logGroupName,kmsKeyId:kmsKeyId,retention:retentionInDays}'
aws ec2 describe-flow-logs \
  --region "$AWS_REGION" \
  --filter "Name=resource-id,Values=$VPC_ID" \
  --query 'FlowLogs[].{status:FlowLogStatus,delivery:DeliverLogsStatus,destination:LogDestination}'
aws logs describe-log-streams \
  --region "$AWS_REGION" \
  --log-group-name "$FLOW_LOG_GROUP" \
  --order-by LastEventTime --descending --max-items 1 \
  --query 'logStreams[0].{stream:logStreamName,lastEvent:lastEventTimestamp}'

PASS requires the application log group to reference KMS_KEY_ARN, 365-day retention, an active flow log with successful delivery, and a flow-log stream with a recent event after traffic is generated.

AWS-4: ALB Access-Log Delivery

aws s3api list-objects-v2 \
  --region "$AWS_REGION" \
  --bucket "$ALB_LOG_BUCKET" \
  --prefix "$ALB_LOG_PREFIX/AWSLogs/" \
  --max-items 5 \
  --query 'Contents[].{key:Key,lastModified:LastModified,size:Size}'

PASS requires a recent, non-empty object after sending traffic. Delivery can lag; absence immediately after a request is not proof of failure.

AWS-5: Protected Application and CSD

curl -sS -o /dev/null -D - "${APPLICATION_URL/https:/http:}" \
  | grep -E '^HTTP/|^[Ll]ocation:'
APPLICATION_HTML=$(curl -fsS "$APPLICATION_URL")
printf '%s' "$APPLICATION_HTML" | grep -qi 'OWASP Juice Shop'
printf '%s' "$APPLICATION_HTML" | grep -q '__imp_apg__'

PASS requires HTTP 301, HTTPS 200, rendered Juice Shop content, and __imp_apg__. Also confirm VIRTUAL_HOST_READY, CertificateValid or AutoCertRenewing, and a browser dip request using the earlier checks.

AWS-6: Final No-Drift Plan

terraform -chdir="$TF_DIR" plan -detailed-exitcode -out=csd-final.tfplan

Exit code 0 is PASS (no drift). Exit code 2 means the saved plan contains changes: review it with terraform -chdir="$TF_DIR" show csd-final.tfplan, correct the owning Terraform configuration or runtime issue, and create a fresh saved plan. Retain the same backend and state. Do not remediate Terraform-owned F5 XC resources with API mutation.

Azure Alternate Owner Evidence

For the Azure alternate, verify ownership and runtime with the Azure source of truth selected for that deployment. Record the exact subscription, resource group, resource IDs, deployment owner, provisioning state, and application endpoint used by XCSH_ORIGIN_IP.

Require an owner-backed no-drift result where declarative IaC owns the resources, and verify a recent Azure application-log event after a protected HTTPS request. Do not reuse the AWS ECS, ALB, CloudWatch, flow-log, S3, or Terraform-output assertions for Azure.

Azure origin health and owner evidence are not substitutes for DNS-to-current-VIP equality, the exact protected-domain record, certificate state, HTTP redirect, HTTPS application marker, F5 access-log event, CSD injection, and browser dip telemetry.

---

Full-Stack Dashboard

Run a single read-only command that checks critical health indicators across every layer and produces a summary table:

echo "=== CSD Verification Dashboard ==="
echo ""

# Layer 1: DNS evidence (display only; DNS-1 and DNS-2 define PASS)
DNS_A=$(dig +short xXCSH_DOMAINNAMEx A | sort -u | paste -sd, -)
DNS_ACME_CNAME=$(dig +short _acme-challenge.xXCSH_DOMAINNAMEx CNAME | paste -sd, -)
DNS_ACME_TXT=$(dig +short _acme-challenge.xXCSH_DOMAINNAMEx TXT | paste -sd, -)

# Layers 2-4: one LB, TLS, and origin
LB=$(curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/http_loadbalancers/xXCSH_LB_NAMEx")
ORIGIN_POOL=$(curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/origin_pools/xXCSH_ORIGIN_POOLx")

LB_STATE=$(echo "$LB" | jq -r '.spec.state // "UNKNOWN"')
CERT_STATE=$(echo "$LB" | jq -r '.spec.cert_state // "UNKNOWN"')
HTTP_REDIRECT=$(echo "$LB" | jq -r '.spec.https_auto_cert.http_redirect // false')
CSD_ON_LB=$(echo "$LB" | jq -r 'if .spec.client_side_defense then "ENABLED" else "DISABLED" end')
DOMAINS=$(echo "$LB" | jq -r '.spec.domains | join(", ")')
ROUTE_POOL=$(echo "$LB" | jq -r '[.spec.default_route_pools[] | .pool.name] | join(", ")')
case "xXCSH_ORIGIN_KINDx" in
  public_name) ORIGIN=$(echo "$ORIGIN_POOL" | jq -r '.spec.origin_servers[0].public_name.dns_name // "MISSING"') ;;
  public_ip) ORIGIN=$(echo "$ORIGIN_POOL" | jq -r '.spec.origin_servers[0].public_ip.ip // "MISSING"') ;;
  *) ORIGIN="INVALID_ORIGIN_KIND" ;;
esac

# Layer 6: CSD status
CSD=$(curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/status")

CSD_CONFIGURED=$(echo "$CSD" | jq -r '.isConfigured // false')
CSD_ENABLED=$(echo "$CSD" | jq -r '.isEnabled // false')

# Layer 6: JS config
JS_TAG=$(curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/js_configuration" \
  | jq -r 'if (.scriptTag | length) > 0 then "PRESENT" else "MISSING" end')

# Layer 7: Traffic (last 24h)
TRAFFIC=$(curl -s -X POST \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  -H "Content-Type: application/json" \
  -d "{
    \"start_time\": \"$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -v-24H +%Y-%m-%dT%H:%M:%SZ)\",
    \"end_time\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"
  }" \
  "xXCSH_API_URLx/api/data/namespaces/xXCSH_NAMESPACEx/access_logs/aggregation" \
  | jq -r '.total_hits // "0"')

printf "%-28s %s\n" "CHECK" "STATUS"
printf "%-28s %s\n" "----------------------------" "----------------------------"
printf "%-28s %s\n" "[DNS] A Records" "${DNS_A:-NOT_FOUND}"
printf "%-28s %s\n" "[DNS] ACME CNAME" "${DNS_ACME_CNAME:-NOT_FOUND}"
printf "%-28s %s\n" "[DNS] ACME TXT" "${DNS_ACME_TXT:-NOT_FOUND}"
printf "%-28s %s\n" "[TLS] Certificate" "$CERT_STATE"
printf "%-28s %s\n" "[LB] State" "$LB_STATE"
printf "%-28s %s\n" "[LB] HTTP Redirect" "$HTTP_REDIRECT"
printf "%-28s %s\n" "[LB] Domains" "$DOMAINS"
printf "%-28s %s\n" "[LB] Route Pool" "$ROUTE_POOL"
printf "%-28s %s\n" "[Origin] Kind" "xXCSH_ORIGIN_KINDx"
printf "%-28s %s\n" "[Origin] Value" "$ORIGIN"
printf "%-28s %s\n" "[LB] CSD on LB" "$CSD_ON_LB"
printf "%-28s %s\n" "[CSD] Configured (tenant)" "$CSD_CONFIGURED"
printf "%-28s %s\n" "[CSD] Enabled" "$CSD_ENABLED"
printf "%-28s %s\n" "[CSD] JS Script Tag" "$JS_TAG"
printf "%-28s %s\n" "[Traffic] Requests (24h)" "$TRAFFIC"

---

Access Log Field Reference

Caution

Each entry in the .logs[] array is a JSON string, not a parsed object. You must pipe through fromjson in jq before accessing fields (e.g., .logs[] | fromjson | .method).

Field	Type	Description
@timestamp	string	Request timestamp (ISO 8601). Note the @ prefix — access with .["@timestamp"] in jq
method	string	HTTP method (GET, POST, etc.)
req_path	string	Request URI path
rsp_code	string	HTTP response status code as a string (e.g., "200", "404")
rsp_code_class	string	Status code class (2xx, 3xx, 4xx, 5xx, or downstream_remote_disconnect)
src_ip	string	Client source IP address
dst_ip	string	Destination (VIP) IP address
domain	string	Request Host header value
user_agent	string	Client User-Agent string
rsp_size	string	Response body size in bytes (returned as string)
req_size	string	Request body size in bytes (returned as string)
duration_with_data_tx_delay	string	Total request duration in seconds (returned as string, e.g., "0.024219")
csd_js_injection	string	"true" when CSD JavaScript was injected (only present when active)

CSD Telemetry Field Reference

Field	Endpoint	Description
isConfigured	status	CSD enabled at tenant level
isEnabled	status	CSD active for this namespace
scripts[]	scripts	Array of detected script objects
.script_name	scripts	Full URL of the JavaScript file
.risk_level	scripts	Risk level (No Risk, Low Risk, High Risk)
.status	scripts	AN (Action Needed) or NA (No Action Needed)
.form_fields_read	scripts	Number of form fields the script reads
.affected_users_count	scripts	Number of unique users/sessions affected
domain_summary	detected_domains	Counts by status: .actionNeededCount.count, .mitigatedDomains.count, .allowedDomains.count, .totalDomains.count (each has .count and .lastUpdated)
domains_list[]	detected_domains	Array of detected domain objects with .domain, .status, .category, .firstSeenDate, .latestSeenDate (epoch seconds as strings)
form_fields[]	formFields	Array of detected form field objects
.analysis	formFields	Sensitivity classification (Sensitive, Not Sensitive)

Troubleshooting

Zero Access Logs

If TV-1 returns 0:

1. Check DNS resolution — verify the domain resolves to the LB VIP:

   dig +short xXCSH_DOMAINNAMEx A

   If empty, DNS is not configured. Follow Phase 1 — DNS, Certificate, and Origin Recovery.

2. Check LB state — the unsuffixed load balancer must be VIRTUAL_HOST_READY and its certificate must be valid:

   curl -s \
     -H "Authorization: APIToken xXCSH_API_TOKENx" \
     "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/http_loadbalancers/xXCSH_LB_NAMEx" \
     | jq '{state: .spec.state, cert_state: .spec.cert_state, http_redirect: .spec.https_auto_cert.http_redirect}'

3. Send protocol-specific test requests — HTTP must redirect and HTTPS must serve the application:

   curl -sS -o /dev/null -D - "http://xXCSH_DOMAINNAMEx/" | grep -E '^HTTP/|^[Ll]ocation:'
   curl -sS -o /dev/null -w '%{http_code}\n' "https://xXCSH_DOMAINNAMEx/"

Managed Records Disabled on DNS Zone

If DNS-4 returns false, automatic record creation is disabled even though F5 XC is the authoritative DNS provider. This is a common misconfiguration that causes both the A record and ACME CNAME to be missing, which blocks the load balancer from reaching VIRTUAL_HOST_READY and the certificate from being issued.

To enable managed records, have the established DNS-zone owner review and apply allow_http_lb_managed_records: true. See Phase 1 — DNS, Certificate, and Origin Recovery; this CSD workflow does not mutate the shared zone.

LB Stuck in VIRTUAL_HOST_PENDING_A_RECORD

The load balancer is waiting for a DNS A record pointing to its VIP. See LB Stuck in VIRTUAL_HOST_PENDING_A_RECORD for detailed resolution steps.

Certificate Stuck in PreDomainChallengePending

The automatic TLS certificate requires an ACME challenge record. The method depends on your DNS provider:

F5 XC Managed DNS: Enable allow_http_lb_managed_records on the DNS zone (DNS-4). The platform creates a TXT-based ACME challenge record in the x-ves-io-managed RR set group.

For API-owned resources, follow the documented clean-recreation workflow if issuance remains stuck. For Terraform-owned resources, retain the same backend/state and create, review, and apply a fresh Terraform plan; do not delete the LB through the API.

External DNS: Create a CNAME record at your DNS provider:

_acme-challenge.xXCSH_DOMAINNAMEx  CNAME  *.autocerts.ves.volterra.io

Verify the record exists:

dig +short _acme-challenge.xXCSH_DOMAINNAMEx CNAME
dig +short _acme-challenge.xXCSH_DOMAINNAMEx TXT

If both are empty, the ACME record is not configured. Certificate provisioning takes 5–10 minutes after the record is in place. Check the LB error status for specific ACME validation errors:

curl -s \
  -H "Authorization: APIToken xXCSH_API_TOKENx" \
  "xXCSH_API_URLx/api/config/namespaces/xXCSH_NAMESPACEx/http_loadbalancers/xXCSH_LB_NAMEx" \
  | jq '[.status[]? | select(.virtual_host_status != null) | .virtual_host_status.error_description]'

CSD isConfigured is false

CSD must be enabled at the tenant level by an F5 XC administrator. This is a tenant-wide setting that cannot be configured via the namespace API. Contact your administrator to enable Client-Side Defense.

Empty Scripts Array

If TEL-1 returns an empty array:

1. Check the protected domain — CSD monitors scripts only on registered domains:

   curl -s \
     -H "Authorization: APIToken xXCSH_API_TOKENx" \
     "xXCSH_API_URLx/api/shape/csd/namespaces/xXCSH_NAMESPACEx/protected_domains" \
     | jq '.items[] | {name, namespace}'

2. Verify JS injection — confirm the CSD script path is injected into the protected HTTPS response:

   curl -fsS "https://xXCSH_DOMAINNAMEx/" | grep -q '__imp_apg__'

   If there is no match, check that the LB has client_side_defense with js_insert_all_pages.

3. Allow processing time — after first enabling CSD or registering a new protected domain, script detection can take 5–15 minutes. Generate traffic to the site and wait before re-checking.

Deterministic Page Tamper Header Runbook

This procedure tests candidate coverage; it does not define F5’s monitored-header set. Page Tamper officially snapshots HTTP headers, uses a moving baseline built over multiple browser sessions and sequential cycles, and can produce ClientSideDefenseHttpHeaderModified or ClientSideDefenseHttpHeaderCompromised. See About Client-Side Defense and Configure Client-Side Defense.

Historical evidence must be interpreted narrowly. On 2026-09-23, a tenant ClientSideDefenseHttpHeaderModified record named x-content-type-options, x-frame-options, and cache-control, reported modification=Added, and included the exact protected path /. It proves detection for that event, not the Compromised semantic or an official monitored-header list. The 2026-09-24 six-header global-value campaign is INVALID TEST for Compromised-trigger conclusions: it had no controlled baseline/comparison cohorts and alert review initially searched current alerts rather than history. Its later bounded negative search remains an observation, not proof of unsupported coverage.

Issue #1241 owns the dedicated endpoint and deterministic controller. Use the permanent but inert-by-default /csd-page-tamper/payment endpoint; do not mutate the F5 Distributed Cloud load balancer or origin pool for an experiment.

1. Bootstrap the control baseline. Require no Terraform drift, a ready load balancer, a valid certificate, healthy ordinary application traffic, and a healthy dedicated endpoint. From workstation and worker fresh browsers, prove HTTP 200, all 12 exact candidate headers, all five empty synthetic fields, CSD injection, and a dip POST. Run control-only traffic for 45 minutes while polling both current alerts and alert history. If deployment creates a correlated Modified/Added alert, wait for resolution and then require a 15-minute quiet period.
2. Reinforce control for 15 minutes. Run 12 fresh profiles against the exact payment path without the selector. Every response must retain all 12 exact headers, CSD injection, and dip evidence.
3. Run one mixed cohort for up to 45 minutes. Alternate fresh control profiles with tampered profiles at the identical URL. Only tampered profiles send X-CSD-Page-Tamper: <header-id>. Require the selected header absent, the other 11 exact, and at least 20 complete pairs unless the exact required alert arrives sooner.
4. Poll current and history every 60 seconds. Parse JSON-string history entries. Correlate only records whose raw fields match the exact namespace, selected header case-insensitively, path /csd-page-tamper/payment, and experiment start window. Preserve sanitized alert name, modification, lifecycle state, timestamps, display name, and description. Generic, wrong-path, stale, future, or adjacent alerts do not qualify.
5. Classify conservatively. Use COMPROMISED only for an exact Compromised correlation; MODIFIED_ONLY for an exact Modified correlation without Compromised; NO_ALERT_WITHIN_WINDOW only when the complete evidence chain is valid but neither arrives; and INVALID_TEST for any baseline, cohort, telemetry, overlap, correlation, recovery, cleanup, readiness, or drift failure.
6. Apply the canary stop rule. Run X-Content-Type-Options first. If it does not reach COMPROMISED, complete recovery, stop the suite, and diagnose the Modified/Compromised distinction. Do not claim the hypothesis succeeded.
7. Recover control-only traffic for 15 minutes. Stop selector traffic and re-prove all 12 exact headers from workstation and worker, CSD injection, dip, endpoint and ordinary application health, load-balancer readiness, run-owned cleanup, and final refresh-aware Terraform no drift. Only then may another header start.

Publish only sanitized receipts and measured outcomes. Until an executed case satisfies every gate, the dedicated mixed-cohort design is a hypothesis rather than proof of Compromised behavior.

Recorded live result: 2026-09-25/26

The dedicated endpoint is deployed and bootstrap-proven. Exact-plan deployment made five additions, one task-definition replacement (1 change, 1 destroy), and zero F5 Distributed Cloud mutations. Bootstrap run dd91ea9b-3da3-4fc8-b24b-c32cdba2e863 ran from 2026-09-25T21:40:23.373Z to 2026-09-25T22:42:18.638Z with alerts=[]. Endpoint and root health, all 12 exact canonical headers, five empty payment fields, CSD script presence, both healthy ALB target groups, VIRTUAL_HOST_READY, CertificateValid, readiness, and Terraform no drift were all observed.

Canary run 69f30bd0-6dc2-4721-bb21-cae159032af2 ran from 2026-09-25T23:21:44.908Z to 2026-09-26T00:38:11.806Z. It completed 12/12 controls and 20/20 mixed control/tampered pairs with valid telemetry and alerts=[]. No exact Modified or Compromised correlation appeared; current alerts returned zero and alert history returned an empty array. Classify this run only as NO_ALERT_WITHIN_WINDOW.

Recovery proved 10 control pairs plus successful probes, readiness, cleanup, and Terraform no drift. The canary did not reach COMPROMISED, so the suite stopped as required and no remaining candidate header was run. The endpoint and test mechanics are validated; the Compromised hypothesis is not. Earlier controller-defect attempts were INVALID_TEST and are not product evidence.

Quick Reference

Time Range Presets

Period	Epoch offset	ISO 8601 (Linux)	ISO 8601 (macOS)
1 hour	$(( $(date +%s) - 3600 ))	date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ	date -u -v-1H +%Y-%m-%dT%H:%M:%SZ
24 hours	$(( $(date +%s) - 86400 ))	date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ	date -u -v-24H +%Y-%m-%dT%H:%M:%SZ
7 days	$(( $(date +%s) - 604800 ))	date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ	date -u -v-7d +%Y-%m-%dT%H:%M:%SZ
30 days	$(( $(date +%s) - 2592000 ))	date -u -d '30 days ago' +%Y-%m-%dT%H:%M:%SZ	date -u -v-30d +%Y-%m-%dT%H:%M:%SZ

Verification Layer Summary

Layer	Tests	What it covers
1. DNS Resolution	DNS-1 through DNS-4	A record, ACME CNAME, nameserver authority, managed records
2. TLS Certificate	TLS-1 through TLS-3	Cert state, cert metadata, live handshake
3. HTTP Load Balancer	LB-1 through LB-5	One LB, HTTPS redirect, CSD flag, route pool, deployment
4. Origin Pool	OP-1 through OP-4	Hostname/IP oneOf, TLS mode, HC association, direct observation
5. Health Check	HC-1 through HC-2	HC config, list all HCs
6. CSD Configuration	CSD-1 through CSD-5	Tenant status, JS tag, protected domains, live injection
7. Traffic Verification	TV-1 through TV-5	Request count, status codes, samples, JS in logs, end-to-end test
8. CSD Telemetry	TEL-1 through TEL-8	Scripts, domains, policy, form fields, deep dive, users, risk, freshness

Endpoint Summary

Diagnostic	Endpoint	Method	Time Format
Request count	/api/data/namespaces/\{ns\}/access_logs/aggregation	POST	ISO 8601
Status codes	/api/data/namespaces/\{ns\}/access_logs	POST	ISO 8601
Recent requests	/api/data/namespaces/\{ns\}/access_logs	POST	ISO 8601
LB state	/api/config/namespaces/\{ns\}/http_loadbalancers/\{name\}	GET	None
Origin pool	/api/config/namespaces/\{ns\}/origin_pools/\{name\}	GET	None
Healthcheck	/api/config/namespaces/\{ns\}/healthchecks/\{name\}	GET	None
DNS zone	/api/config/dns/namespaces/system/dns_zones/\{zone\}	GET	None
CSD status	/api/shape/csd/namespaces/\{ns\}/status	GET	None
JS configuration	/api/shape/csd/namespaces/\{ns\}/js_configuration	GET	None
Protected domains	/api/shape/csd/namespaces/\{ns\}/protected_domains	GET	None
Script list	/api/shape/csd/namespaces/\{ns\}/scripts	POST	Epoch seconds
Detected domains	/api/shape/csd/namespaces/\{ns\}/detected_domains	GET	None
Form fields	/api/shape/csd/namespaces/\{ns\}/formFields	GET	Epoch seconds (query params)
Script details	/api/shape/csd/namespaces/\{ns\}/scripts/\{id\}/dashboard	GET	None
Script behaviors	/api/shape/csd/namespaces/\{ns\}/scripts/\{id\}/behaviors	GET	None
Script network	/api/shape/csd/namespaces/\{ns\}/scripts/\{id\}/networkInteractions	GET	None
Affected users	/api/shape/csd/namespaces/\{ns\}/scripts/\{id\}/affectedUsers	POST	Epoch seconds