Phase 4 — Teardown

Phase 4 removes the deployment through the same ownership mode that created it. API-owned resources are deleted only from the Phase 1 session record. Terraform-owned resources are destroyed only through a reviewed saved Terraform plan.

Caution

Explicit approval is required. Before either path, ask: “Phase 4 will permanently delete the CSD deployment recorded by its owner. Are you ready to tear down? Reply ‘yes’ to proceed.” Do not delete or apply a destroy plan until the operator replies yes.

Choose one ownership mode. Never run the API create/update/delete workflow against resources present in Terraform state.

Select exactly one path:

• Path A — API-owned: Phase 1 produced .csd-api-session.json with ownership_mode: "api" and exact created entries.
• Path B — Terraform-owned: the remote backend and current state contain the deployment.

If ownership is ambiguous, stop. Do not infer ownership from matching names, tags, current accessibility, or a successful GET.

Path A: Delete API-Owned Objects

Validate the Session Record

Fail closed unless the ledger is structurally valid, belongs to the current environment, contains only allowed kinds/statuses, and has exactly one entry per (kind, namespace, name). Environment variables validate the ledger; they never select teardown names.

SESSION_FILE="${CSD_API_SESSION_FILE:-.csd-api-session.json}"
case "$SESSION_FILE" in /*) ;; *) SESSION_FILE="$PWD/$SESSION_FILE" ;; esac
SESSION_DIR=$(dirname -- "$SESSION_FILE")
SESSION_NAME=$(basename -- "$SESSION_FILE")
[ -d "$SESSION_DIR" ] && [ "$SESSION_NAME" != . ] && [ "$SESSION_NAME" != .. ] || {
  echo "STOP: API session ledger path is invalid"; exit 1;
}
SESSION_DIR=$(cd "$SESSION_DIR" && pwd -P) || exit 1
SESSION_FILE="$SESSION_DIR/$SESSION_NAME"
export CSD_API_SESSION_FILE="$SESSION_FILE" CSD_API_SESSION_DIR="$SESSION_DIR"
test -f "$SESSION_FILE" || { echo "STOP: API session ledger is missing"; exit 1; }

case "${XCSH_API_URL:-}" in https://*) ;; *) exit 1 ;; esac
while [ "${XCSH_API_URL%/}" != "$XCSH_API_URL" ]; do
  XCSH_API_URL=${XCSH_API_URL%/}
done
export XCSH_API_URL
XCSH_TENANT_HOSTNAME=${XCSH_API_URL#https://}
case "$XCSH_TENANT_HOSTNAME" in ''|*/*|*:*|*[!A-Za-z0-9.-]*) exit 1 ;; esac
XCSH_TENANT_IDENTITY=${XCSH_TENANT_HOSTNAME%%.*}
[ -n "$XCSH_TENANT_IDENTITY" ] || exit 1

jq -e --arg api_url "$XCSH_API_URL" --arg tenant_hostname "$XCSH_TENANT_HOSTNAME" \
  --arg tenant_identity "$XCSH_TENANT_IDENTITY" --arg ns "$XCSH_NAMESPACE" \
  --arg domain "$XCSH_DOMAINNAME" '
  .schema_version == 1 and .ownership_mode == "api" and
  .environment == {
    api_url: $api_url, tenant_hostname: $tenant_hostname, tenant_identity: $tenant_identity,
    namespace: $ns, domain: $domain
  } and
  (.resources | type == "array") and
  (all(.resources[];
    (.kind | IN("namespace","protected_domain","healthcheck","origin_pool","http_loadbalancer","mitigated_domain")) and
    (.namespace == $ns) and
    (.name | type == "string" and test("^[a-z]([a-z0-9-]{0,62}[a-z0-9])?$")) and
    (.status | IN("created","pre-existing","unknown")) and
    (.spec | type == "object")
  )) and
  ([.resources[] | [.kind,.namespace,.name] | join("\u0000")] | length) ==
  ([.resources[] | [.kind,.namespace,.name] | join("\u0000")] | unique | length) and
  ([.resources[] | select(.status == "unknown")] | length == 0)
' "$SESSION_FILE" >/dev/null || {
  echo "STOP: ledger is corrupt, mismatched, duplicated, or has unknown ownership"
  exit 1
}

TF_XC_STATE=$(terraform -chdir=terraform/aws state list) || {
  echo "STOP: Terraform state could not be read; ownership is unresolved"
  exit 1
}
while IFS= read -r state_address; do
  [ -n "$state_address" ] || continue
  resource_address=$state_address
  while [[ "$resource_address" =~ ^module\.[^.\[]+(\[[^]]+\])?\.(.+)$ ]]; do
    resource_address=${BASH_REMATCH[2]}
  done

  if [[ "$resource_address" =~ ^data\.xcsh_[A-Za-z0-9_]+\.[A-Za-z0-9_-]+(\[[^]]+\])?$ ]]; then
    continue
  fi
  if [[ "$resource_address" =~ ^xcsh_[A-Za-z0-9_]+\.[A-Za-z0-9_-]+(\[[^]]+\])?$ ]]; then
    echo "STOP: Terraform state contains managed xcsh resource $state_address; use Terraform Path B"
    exit 1
  fi
  if [[ "$resource_address" == *xcsh_* ]]; then
    echo "STOP: unrecognized Terraform address shape $state_address; ownership is unresolved"
    exit 1
  fi
done <<<"$TF_XC_STATE"

CREATED=$(jq -c '[.resources[] | select(.status == "created")]' "$SESSION_FILE")
printf '%s\n' "$CREATED" | jq .

Only entries in CREATED are eligible. pre-existing and unknown entries cannot be deleted. Do not compare names to XCSH_LB_NAME, XCSH_ORIGIN_POOL, XCSH_HC_NAME, or any other mutable environment variable to choose targets.

Capture Live Inventory

Before deleting anything, read the current namespace inventory with the native API wildcard inventory operation and preserve the response with the session record:

{
  "method": "GET",
  "paths": [
    "*"
  ],
  "params": {
    "namespace": "<XCSH_NAMESPACE>"
  }
}

The live inventory is evidence, not authorization. It may reveal unrelated or externally visible objects, but it does not expand the deletion allowlist.

Delete in Reverse Dependency Order

Enumerate targets only from the validated CREATED snapshot and reject any duplicate kind/name before mutation:

jq -er '
  def rank: {mitigated_domain:1,http_loadbalancer:2,origin_pool:3,healthcheck:4,protected_domain:5,namespace:6}[.];
  sort_by(.kind | rank)[] | [.kind,.namespace,.name] | @tsv
' <<<"$CREATED" > .csd-api-delete-targets.tsv || {
  echo "STOP: could not derive deletion targets from ledger"
  exit 1
}

Process .csd-api-delete-targets.tsv line by line in its emitted reverse-dependency order. For each row, map the recorded kind to exactly one endpoint; never substitute a current environment name:

Recorded kind	DELETE endpoint
mitigated_domain	/api/shape/csd/namespaces/{namespace}/mitigated_domains/{name}
http_loadbalancer	/api/config/namespaces/{namespace}/http_loadbalancers/{name}
origin_pool	/api/config/namespaces/{namespace}/origin_pools/{name}
healthcheck	/api/config/namespaces/{namespace}/healthchecks/{name}
protected_domain	/api/shape/csd/namespaces/{namespace}/protected_domains/{name}

The namespace row is withheld for the guarded cascade procedure below. Before each object DELETE, re-read the same ledger file and require exactly one matching entry with status == "created"; if the ledger changed, is corrupt, or the match count is not one, stop.

After each DELETE, GET that exact ledger-recorded object and require 404 before continuing. If it remains, wait once, GET once more, then stop. Never delete pre-existing or unknown entries, and never infer targets from collection counts or null names.

Namespace Cascade Delete: Fail Closed

Caution

Cascade deletion can remove every namespace-scoped object, including objects never created by this session. It is not a cleanup shortcut.

Run cascade deletion only when all of these statements are true:

• The ledger has exactly one namespace entry for the current environment and its status is created.
• Every other ledger-created entry was deleted and verified absent.
• A fresh xcsh_api wildcard inventory (GET, paths: ["*"], current namespace) reports zero confirmed namespace members.
• External-visible results and Unknown-scope results were reviewed but are not treated as namespace members or deletion targets.
• No pre-existing or unknown ledger entry exists in that namespace.

If any condition fails, preserve the namespace and report the blocker. If all pass, request a second explicit approval: “The recorded objects are absent and inventory has zero confirmed members. Cascade-delete the ledger-created namespace <recorded-name>? Reply ‘yes’ to proceed.” The first teardown approval does not authorize this cascade.

Only after the second yes, issue the namespace lifecycle operation documented for this API: POST /api/web/namespaces/{namespace}/cascade_delete. Use the namespace name read from the ledger in the path parameter and request body:

{
  "method": "POST",
  "path": "/api/web/namespaces/{namespace}/cascade_delete",
  "params": {
    "namespace": "<ledger-created namespace name>"
  },
  "payload": {
    "name": "<ledger-created namespace name>"
  }
}

Then GET /api/web/namespaces/{namespace} with that same recorded name and require 404. The standard DELETE /api/web/namespaces/{namespace} operation is not the documented deletion lifecycle and must not be substituted. Do not delete a shared DNS zone or manual DNS records unless separately approved and recorded as API-created by this exact session.

API Teardown Evidence

Retain the following evidence with the session record:

Check	Required result
Initial approval	Exact affirmative operator response recorded
Ownership	Targets derived only from unique ledger entries with status: "created"
Reverse order	Mitigations, LB, pool, optional healthcheck, protected domain
Object reads	Each deleted object returned 404 before its dependency was removed
Namespace inventory	Zero confirmed members before cascade, or namespace preserved
Cascade approval	Separate explicit affirmative response recorded
Final state	Namespace 404 only after authorized cascade; otherwise namespace preserved

Path B: Destroy Terraform-Owned Resources

Do not use any API DELETE request in this path. Keep the same remote S3 backend, state key, native lockfile, AWS profile, region, account, namespace, and domain used for deployment.

Initialize without changing backend ownership:

terraform -chdir=terraform/aws init
terraform -chdir=terraform/aws validate

Before creating any destroy plan, capture every identifier used by the post-destroy checks from current state. Validate the complete evidence object, then make it read-only. Keep it with the teardown evidence; do not commit it or put credentials, tokens, provider configuration, or other secrets in it:

IDENTIFIERS=.csd-destroy-identifiers.json
umask 077
terraform -chdir=terraform/aws output -json | jq -e '
def nonempty_string: type == "string" and length > 0;
def nonempty_string_array:
  type == "array" and length > 0 and all(.[]; type == "string" and length > 0);
{
  aws_account_id: .aws_account_id.value,
  aws_region: .aws_region.value,
  vpc_id: .vpc_id.value,
  public_subnet_ids: .public_subnet_ids.value,
  private_subnet_ids: .private_subnet_ids.value,
  alb_arn: .alb_arn.value,
  target_group_arn: .target_group_arn.value,
  alb_access_logs_bucket: .alb_access_logs_bucket.value,
  cloudwatch_logs_kms_key_arn: .cloudwatch_logs_kms_key_arn.value,
  xc_namespace: .xc_namespace.value,
  xc_protected_domain_name: .xc_protected_domain_name.value,
  xc_origin_pool_name: .xc_origin_pool_name.value,
  xc_http_loadbalancer_name: .xc_http_loadbalancer_name.value
} | select(
  (.aws_account_id | nonempty_string) and (.aws_region | nonempty_string) and
  (.vpc_id | nonempty_string) and (.public_subnet_ids | nonempty_string_array) and
  (.private_subnet_ids | nonempty_string_array) and (.alb_arn | nonempty_string) and
  (.target_group_arn | nonempty_string) and (.alb_access_logs_bucket | nonempty_string) and
  (.cloudwatch_logs_kms_key_arn | nonempty_string) and (.xc_namespace | nonempty_string) and
  (.xc_protected_domain_name | nonempty_string) and (.xc_origin_pool_name | nonempty_string) and
  (.xc_http_loadbalancer_name | nonempty_string)
 )' > "$IDENTIFIERS" || {
  rm -f "$IDENTIFIERS"
  echo "STOP: complete post-destroy identifiers could not be captured"
  exit 1
}
chmod 0400 "$IDENTIFIERS"

terraform -chdir=terraform/aws plan \
  -destroy \
  -out=csd-destroy.tfplan
terraform -chdir=terraform/aws show csd-destroy.tfplan

Review the complete plan before approval. It must destroy only resources owned by this state. Confirm the expected F5 Distributed Cloud objects are the one namespace, protected domain, origin pool, and unsuffixed HTTPS load balancer, along with the AWS origin infrastructure managed by the same state. Unexpected replacements, creates, provider errors, or unrelated destroys block teardown.

After that review, obtain and record a new explicit approval bound to the exact saved-plan bytes. An earlier general teardown approval is not apply authorization:

read -r -p "Apply the reviewed destroy plan? Type yes: " CSD_DESTROY_APPROVAL
test "$CSD_DESTROY_APPROVAL" = yes || { echo "STOP: destroy not approved"; exit 1; }
read -r -p "Record approver identity: " CSD_DESTROY_APPROVER
test -n "$CSD_DESTROY_APPROVER" || { echo "STOP: approver identity is required"; exit 1; }

PLAN_FILE=terraform/aws/csd-destroy.tfplan
PLAN_SHA256=$(shasum -a 256 "$PLAN_FILE" | awk '{print $1}')
jq -n --arg approval "$CSD_DESTROY_APPROVAL" \
  --arg approver "$CSD_DESTROY_APPROVER" --arg plan_sha256 "$PLAN_SHA256" \
  --arg approved_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  '{approval:$approval,approver:$approver,plan_sha256:$plan_sha256,approved_at:$approved_at}' \
  > .csd-destroy-approval.json
chmod 0400 .csd-destroy-approval.json

test "$(jq -r .approval .csd-destroy-approval.json)" = yes
test "$(jq -r .plan_sha256 .csd-destroy-approval.json)" = \
  "$(shasum -a 256 "$PLAN_FILE" | awk '{print $1}')"
terraform -chdir=terraform/aws apply csd-destroy.tfplan

Do not run a fresh unsaved terraform destroy after approval; the reviewed saved-plan bytes and its recorded approval are the authorization boundary.

Verify both Terraform state and live absence using only the identifiers captured before destroy:

test -z "$(terraform -chdir=terraform/aws state list)"

set +e
terraform -chdir=terraform/aws plan -destroy -detailed-exitcode -out=csd-post-destroy.tfplan
PLAN_RC=$?
set -e
test "$PLAN_RC" -eq 0

AWS_ACCOUNT_ID=$(jq -r .aws_account_id .csd-destroy-identifiers.json)
AWS_REGION=$(jq -r .aws_region .csd-destroy-identifiers.json)
VPC_ID=$(jq -r .vpc_id .csd-destroy-identifiers.json)
ALB_ARN=$(jq -r .alb_arn .csd-destroy-identifiers.json)
TARGET_GROUP_ARN=$(jq -r .target_group_arn .csd-destroy-identifiers.json)
ALB_LOG_BUCKET=$(jq -r .alb_access_logs_bucket .csd-destroy-identifiers.json)
KMS_KEY_ARN=$(jq -r .cloudwatch_logs_kms_key_arn .csd-destroy-identifiers.json)
SUBNET_IDS=$(jq -r '[.public_subnet_ids[],.private_subnet_ids[]] | join(" ")' .csd-destroy-identifiers.json)

test "$(aws sts get-caller-identity --query Account --output text)" = "$AWS_ACCOUNT_ID"
test "$(aws ec2 describe-vpcs --region "$AWS_REGION" \
  --filters "Name=vpc-id,Values=$VPC_ID" \
  --query 'length(Vpcs)' --output text)" = "0"
test "$(aws ec2 describe-subnets --region "$AWS_REGION" \
  --filters "Name=subnet-id,Values=${SUBNET_IDS// /,}" \
  --query 'length(Subnets)' --output text)" = "0"

if ALB_ERROR=$(aws elbv2 describe-load-balancers --region "$AWS_REGION" \
  --load-balancer-arns "$ALB_ARN" 2>&1); then
  echo "STOP: captured ALB still exists"
  exit 1
fi
printf '%s' "$ALB_ERROR" | grep -q 'LoadBalancerNotFound'

if TG_ERROR=$(aws elbv2 describe-target-groups --region "$AWS_REGION" \
  --target-group-arns "$TARGET_GROUP_ARN" 2>&1); then
  echo "STOP: captured target group still exists"
  exit 1
fi
printf '%s' "$TG_ERROR" | grep -q 'TargetGroupNotFound'

test "$(aws s3api list-buckets \
  --query "length(Buckets[?Name=='$ALB_LOG_BUCKET'])" --output text)" = "0"
test "$(aws kms describe-key --region "$AWS_REGION" --key-id "$KMS_KEY_ARN" \
  --query 'KeyMetadata.KeyState' --output text)" = "PendingDeletion"

Required result: state output is empty, the destroy-mode plan exits 0, and every captured AWS identifier is absent, except that the captured KMS key must be in PendingDeletion because AWS enforces a deletion waiting period. Do not use a normal configuration plan after destroy: the configuration still declares the stack, so a normal plan correctly proposes recreation.

Also issue these read-only xcsh_api GETs using values from .csd-destroy-identifiers.json; each must return 404:

Captured value	GET endpoint
xc_http_loadbalancer_name	/api/config/namespaces/{namespace}/http_loadbalancers/{name}
xc_origin_pool_name	/api/config/namespaces/{namespace}/origin_pools/{name}
xc_protected_domain_name	/api/shape/csd/namespaces/{namespace}/protected_domains/{name}
xc_namespace	/api/web/namespaces/{namespace}

Pass xc_namespace as namespace and the corresponding captured resource value as name. Any successful read is a live-remnant blocker. Do not delete the remnant through the API—retain the same backend and recover through a fresh reviewed Terraform destroy plan.

If apply is interrupted by an EIP quota, state lock, certificate/origin propagation, namespace dependency, or transient F5 Distributed Cloud 503, retain the same backend and state. Resolve the cause, then create, review, approve, and apply a fresh saved destroy plan. Never switch to API deletion to finish a Terraform-owned teardown.

Completion

Teardown is complete only when the selected owner’s verification passes:

• API path: every session-recorded API-created object is absent, while all pre-existing and unrecorded objects remain untouched.
• Terraform path: state is empty, the destroy-mode plan exits 0, and exact pre-destroy identifiers are absent from both AWS and F5 Distributed Cloud live reads.

There is no skeleton load balancer and no second load balancer to preserve. A future build creates the same single logical architecture through exactly one selected owner.