#CAS Middleware Operations

Runbooks, SLOs, and alerting for the independently deployed CAS middleware (the @unicas packages in packages/). Production topology:

Secrets live only as Worker secrets (Google, Microsoft, and GitHub client secrets, SESSION_ENCRYPTION_KEYS, OAUTH_STATE_ENCRYPTION_KEY, CAS_AUDIT_READER_KEY) — never in vars or source. Deployment credentials are supplied through CLOUDFLARE_ACCOUNT_ID / CLOUDFLARE_API_TOKEN; see Deployment and local configuration.

#SLOs and error budgets

Error-budget burn: alert at 5% of monthly budget consumed per rolling 24 h, page at 15%. Availability is measured from the unified service (/health plus a routed probe like the live smoke's lease+read).

#Metrics and events

Cloudflare's aggregate Worker metrics are authoritative for request volume, invocation outcomes, and CPU/wall-time quantiles. Persisted custom logs use a 5% head sample and are diagnostic evidence only. Generated invocation logs, native Cloudflare tracing, and production manual trace export are disabled by the observability contract. The manual OTLP code path is checked in with a zero production sample and no destination or exporter secrets.

Existing structured events (JSON to Worker stdout and sampled Workers Logs):

• cas_app_authorization — App/Space v1 authorization decisions. Events use kind ∈ authorized, rejected, fail_closed, registry_stale. fail_closed is an incident signal (registry unreachable past the hard bound, or a cold outage). Rejected requests carry stable capability error codes such as unknown_issuer and registry_unavailable in their HTTP responses.
• admin_oidc_callback_failed — administrator login callback failures with a bounded reason and no token material.
• admin_email_challenge_delivery_failed — generic Email binding failure; the event contains no address, code, invitation token, or provider payload.
• unicas_*_failed — bounded incident locations without raw errors, URLs, identifiers, storage keys, or content.

Use Worker/zone metrics or the Cloudflare GraphQL API for five-minute request and status rollups. Query sampled event names and bounded reason/error codes in Workers Logs to diagnose a metric or probe signal; do not scale the 5% sample into an authoritative count. Manual D1/R2/Durable Object waterfalls are available for synthetic validation, but production export remains dormant until destination access and retention, canary evidence, and the exact nonzero sample receive review.

#Alerting rules

#Runbooks

#Lease-driven upload cleanup

App/Space v1 creates at most 1024 active node-upload generations per Space. Presigned PUT URLs default to five minutes. Uploaded or abandoned temporary objects remain eligible for recovery until their 24-hour cleanup deadline; the bounded Space GC path removes expired records, reservations, and temporary objects without requiring another lease for that hash.

reservedBytes includes reservations whose byte length is known after validation (and retained v1 server-mediated uploads). It does not include an unvalidated direct R2 object because the v2 request declares no length. The active-generation limit, publication-time 32 MiB check, short URL lifetime, and cleanup deadline bound that pre-validation exposure instead.

Alert on repeated STORAGE_ERROR, CAS_UPLOAD_LIMIT, or ready-record/object inconsistency. Never respond to a missing canonical object for a ready row by issuing a fresh upload target. R2 PutObject does not currently support a SHA-256 FULL_OBJECT checksum, so the 32 MiB limit and SHA-256 identity are enforced when the next lease validates the temporary object, not by R2 while receiving the PUT.

#Deploy

The UniCAS service remains one middleware deployment unit, while a complete production release publishes four independently owned Workers: service, Spaces App, product site, then documentation site. A promotion pull request from main into release is the normal release boundary. Its merged release revision runs the protected deploy-production job after CI validation. That job checks out and builds the exact validated commit, runs canonical service smoke, deploys Spaces and runs its file smoke, deploys the two static Workers, and verifies all five public origins. A separate job then records a successful push deployment as an immutable production-YYYYMMDD-<workflow-run-number> annotated tag pointing to that exact revision; manual recovery runs do not create production tags.

For read-only manual release validation, dispatch CI; it runs pnpm validate:release and cannot enter Production. Spaces bootstrap or principal recovery uses the separate Recover Spaces production workflow. Supply a full commit SHA reachable from main, select exactly one recovery operation, and pass Production review. Recovery shares the unicas-production concurrency group with promotion and never creates a production tag.

Documentation-only publication uses the separate Deploy documentation workflow. Supply the full lowercase SHA of the current main tip and pass Production review. The workflow requires the checkout and refreshed origin/main to equal that SHA before it runs repository code, shares the unicas-production concurrency group with complete releases, and runs docs tests, a browser test, a Wrangler dry-run, the docs deployment, public SDK route checks, and an artifact-manifest.json source-revision check. It cannot deploy the service, Spaces, product site, npm packages, tags, or GitHub Releases.

Local emergency deployment remains explicit. Always rebuild first because Wrangler uploads dist/ and stale output silently deploys old code:

pnpm release:service:plan
pnpm release:service:production
pnpm release:spaces:plan
pnpm release:spaces:production
pnpm release:site:production
pnpm release:docs:production
pnpm release:service:smoke                    # App/Space v1; run twice 70s apart
pnpm release:spaces:smoke -- --base-url https://spaces.unicas.work

pnpm release:service:production applies pending CAS_DB D1 migrations after the service build and before Worker publication. Tenant schema initialization is not a request or cron responsibility. A migration error stops publication; do not deploy the new Worker separately. Worker rollback does not roll back the D1 migration ledger, so every future migration must state backward compatibility or a forward-repair procedure.

The Spaces Worker uses the explicit aws:ap-southeast-1 placement hint rather than fixed IAD or Smart Placement. SPACES_DB remains in ENAM, while the UniCAS tenant D1 and Durable Object data plane are in APAC. An authenticated directory read therefore pays one bounded session query across regions before executing the dominant manifest path near the APAC data plane. Do not switch back to Smart Placement without a new canary: optimizing the first query can increase the downstream Durable Object dispatch time.

Treat the placement hint as a candidate topology, not proof of latency. Before accepting a deployment, run at least 30 authenticated synthetic directory reads from the agreed APAC probes, report p50/p95/p99/max for TTFB and Worker wall time, record the observed colo/placement, and verify a controlled cold request. Compare spaces_session, spaces_manifest, spaces_unicas, spaces_root when present, cas_auth, cas_do, applicable cas_d1_*/cas_r2_*, and cas_edge. Do not report response-construction timing as full stream completion.

If explicit APAC placement misses any directory threshold, restore the last accepted Spaces version and placement configuration before evaluating an APAC SPACES_DB migration. A D1 move requires a separate copy, consistency, cutover, rollback, and data-validation plan; do not infer it from a broad region label.

The App/Space smoke script uses one dedicated /deploy-smoke Space, per-run node hashes, and per-run request IDs. After acquiring the parent Root Ref, it releases that ref in a finally path even when a later assertion fails, so subsequent runs do not accumulate positive Root Refs. It also checks an empty isolation Space and rejects both token/route version mismatches. HTTPS targets skip the paused-body concurrency probe because Cloudflare ingress buffers the request body; set UNICAS_SMOKE_ENABLE_CONCURRENCY=1 only when the target preserves streaming ingress. Running smoke twice with a 70s gap also proves the authority-cache refresh path.

The Spaces smoke uses a dedicated Principal and Space. Each run creates a temporary file Root, records paths before mutation, uploads a multi-node file, requires immutable-node reuse on a second upload, and checks exact authority and Space denials. Cleanup uses a stable Root Ref request ID, removes the catalog row only after release, and is safe to repeat. The scheduled handler retries at most ten expired or failed runs per invocation. Detailed bootstrap, rotation, and recovery steps are in Spaces file App operations.

#Failure diagnosis

Use the named failed step and its non-secret Wrangler output to identify the deployment unit. Validation failures publish nothing. A failure in the service publish or canonical smoke leaves the product and documentation Workers untouched, although the service version may already be live. A Spaces deploy or smoke failure occurs after the service smoke and blocks both static sites. A product-site failure occurs after both service smokes. A documentation failure occurs after all earlier units succeed. A final HTTPS-check failure means the publishes completed but one public route did not return a successful response.

A tag-production failure happens only after all deployment and verification steps passed, so production is live even though its audit marker is missing. Use Re-run failed jobs on that original workflow run; do not dispatch a new manual run or create a tag by hand. The retry is idempotent when the existing tag resolves to the deployed commit and fails without moving it on a conflict. Follow Deployment and local configuration for tag inspection, ruleset policy, and conflict escalation.

For any failure after a Wrangler command starts, compare the affected Worker's wrangler deployments list output with the workflow commit and timestamps. Do not print environment values or private-key files while diagnosing. The workflow's always() cleanup removes its ephemeral smoke key even when the service or smoke step fails. Fix a transient or configuration problem and use a manual release dispatch; use version rollback for a bad artifact.

#Credential rotation

To rotate the Cloudflare deployment token, create a replacement from the Edit Cloudflare Workers template with the same single-account and unicas.work zone scope. Replace the CLOUDFLARE_API_TOKEN secret in the GitHub Production environment, complete a green manual run from release, then revoke the old token.

To rotate the smoke signer, first publish the replacement public key through the tracked https://unicas.work/deploy-smoke JWKS while retaining the old key. Deploy the product-site Worker and wait for the verifier-cache overlap window. During a deployment-free window, replace UNICAS_SMOKE_PRIVATE_KEY_PKCS8 and UNICAS_SMOKE_KID in the same GitHub environment, then complete a green manual run before removing the old public key in a later product-site deployment. The issuer URL and audience do not change during signer rotation, so no new control-plane activation is needed. Rotate Worker runtime secrets separately with wrangler secret put; never copy them into GitHub deployment configuration.

#App invitation management

App members can inspect invitation history with GET /admin/apps/{appId}/member-invitations, filtering by status=pending|accepted|expired|revoked and traversing { items, nextCursor } pages with limit (1..1000). Cursors bind the App, status filter, and control snapshot. Restart traversal after INVALID_CURSOR; current membership is checked for every page. Reads never return bearer tokens, token hashes, or accept URLs.

DELETE /admin/apps/{appId}/member-invitations/{invitationId} requires that invitation's exact If-Match, not the App revision. A pending, unexpired invitation becomes revoked and returns 204 with its new ETag. A revoked record with a current ETag is a no-op; a stale version returns 412, and an accepted or expired invitation returns 409 INVITATION_NOT_PENDING.

List, accept, and revoke operations reconcile elapsed pending invitations to durable expired state. Accepted, revoked, and expired transitions advance the invitation revision and append immutable-actor audit evidence. Acceptance and revocation race under one atomic storage precondition, so they cannot both succeed. Records remain available for history rather than being deleted.

V2 creation returns only { invitationId, acceptUrl, expiresAt } and the invitation ETag. Acceptance returns only { appId } for navigation. The Console's Members section includes Administrators and Invitations views with filtering, refresh, paging, and revoke confirmation. A freshly created URL is copyable only from that creation result and is cleared when switching Apps; the invitation list cannot recover it.

#Platform Account bootstrap and reset

Administrator authorization is deny-by-default and belongs to the stable Account. platform.admin and apps.create are independent child authorities; App membership grants access only to that App. Exact provider issuer/subject is stored only on External Identity and retained with audit evidence.

Fresh control-schema initialization creates only the current Account/App model. It does not inspect, synthesize, backfill, or upgrade legacy administrator data. For the authorized internal-development production cutover, use release/service/reset-smoke.mjs from the exact release candidate. Its preview inventories the fixed production bindings and refuses unknown tables, Apps, Spaces, issuer configuration, R2 prefixes, or KV key formats. Execution requires the exact Smoke App ID and either a reviewed backup directory or the explicit DELETE-ALL-TEST-DATA-NO-BACKUP confirmation.

Execution first deploys and verifies a maintenance Worker on both service origins, directly enumerates the dedicated R2 and KV bindings, then repeats the D1 inventory and rejects any change. Pending direct-upload sessions or physical objects outside the Smoke node and temporary-upload prefixes stop the reset. It captures the authorized Google issuer/subject entirely inside D1, clears the bounded test state, creates exactly the current Account/App and App/Space schemas, and bootstraps the Account, two platform authorities, Smoke App membership, external issuer, and audit evidence. The subject is never returned through Wrangler or written to a local file. Any failure leaves maintenance active; only the held release workflow restores the normal Worker. If schema creation completed but final verification was interrupted, rerun with --verify-current and the exact App ID instead of attempting the destructive path again.

Bootstrap the first Account through the local bootstrap or a restricted, environment-specific initialization procedure that atomically creates:

1. one cas_accounts row and its cas_account_profiles row;
2. one exact cas_external_identities binding;
3. platform.admin and, when required, apps.create rows in cas_account_platform_authorities.

Do not derive an Account from email, write Principal/operator compatibility rows, or copy provider identity into membership rows. Verify bootstrap with count-only queries over active Accounts and Account authorities, then establish a second Platform Admin through the protected API.

Platform invitation idempotency replay stores its bearer token only as an encrypted value under SESSION_ENCRYPTION_KEYS; retain old keys until sessions and pending invitation receipts sealed with them have expired. If D1 is unavailable, authorization fails closed; there is no bypass flag or fallback email allowlist.

#Suspend and restore an App

An App member can pause all App Space data-plane operations without deleting data or changing memberships, issuer configuration, or Root Ref retention:

unicas apps update <appId> --status suspended --etag '"4"'
unicas apps update <appId> --status active --etag '"5"'

Read the current App with unicas apps get <appId> before selecting an ETag. The HTTP operation is PATCH /admin/apps/{appId} with If-Match; success is 204 No Content with the resulting App ETag. CLI and MCP return only { etag }. A current same-value update is a no-op; stale preconditions fail with 412 REVISION_MISMATCH. Actual transitions append app.suspended or app.restored control audit events with the immutable actor identity.

Suspension rejects reads, metadata reads, leases, Root Ref reads/updates, usage, and GC with 403 APP_SUSPENDED once the verifier observes the change. Capabilities issued before suspension are covered. Issuer authority refreshes after 30 seconds and fails closed at 60 seconds since its last successful read if registry refresh fails. Failure must never preserve old access beyond that hard bound; requests then fail with registry unavailability until current state can be read. An in-flight operation already authorized before the boundary is not rolled back.

Protected-resource metadata is served only for active Apps; suspended Apps return an unavailable-issuer response. Prior positive discovery responses may remain cached for at most their 60-second max-age, but do not override capability verification.

App administrator reads and recovery mutations remain available, including metadata repair, membership and invitation administration, issuer repair, audit, and Restore. Suspension neither releases Root Refs nor initiates GC. Restore leaves data and configuration intact; verifiers observe it on refresh. The existing authorization event stream records suspension denials without recording bearer capabilities. The App/Space verifier refuses a suspended owning App so it cannot bypass this operational stop.

#Inspect App usage

The Console App Overview reads current aggregate CAS usage through:

GET /admin/apps/{appId}/usage
Cookie: cas_admin_session=<HttpOnly OIDC-backed session>

Current App membership is required; no Space capability is used. Suspended Apps remain inspectable. The response reports node count, logical and physical ready bytes, upload reservations, missing-content nodes, and leased nodes summed across independently accounted Spaces.

The read queries cas_space_usage, one mutable projection row per usage- bearing Space, and never scans App node rows or R2. Node and reservation writes maintain that row transactionally. Scheduled maintenance emits cas_usage_reconciliation with only examined/observed/missing/failure counts while it backfills and rotates through canonical-object observations. A 503 SERVICE_UNAVAILABLE means at least one node has not completed observation or the accounting store is unavailable; retry after reconciliation advances. Do not treat it as zero usage.

The Console Usage card also exposes an App-member recovery action:

POST /admin/apps/{appId}/gc
Cookie: cas_admin_session=<HttpOnly OIDC-backed session>
X-CSRF-Token: <session token>
Content-Type: application/json

{}

The action records app.gc.requested in the control audit log, enumerates a bounded keyset page of usage-bearing Spaces, and dispatches each Space through its normal keyed mutation actor. It never bypasses the Space GC eligibility checks: only nodes with zero Root Refs, zero child references, and an expired lease may be deleted. One request has fixed Space and per-Space node budgets and may return a nextCursor; repeat with that cursor to finish the current App sweep. Run another sweep after deletions because removing parents can make their children eligible in a later pass.

#Rollback

Wrangler retains prior versions. Inspect each unit that may have changed and select the known-good version ID. If more than one unit changed, roll them back in reverse deployment order: documentation, product site, Spaces, then service. Skip units that the failed workflow never reached.

A Worker version rollback changes the code receiving traffic; it does not undo D1, R2, KV, Durable Object storage, or Worker secret changes. Inspect the target and failed versions for binding or Durable Object migration compatibility before rolling back. Routine releases in this workflow do not rewrite resources or runtime secrets, but a future migration can make an older service version unsafe to restore.

pnpm --filter @unicas/docs-site exec wrangler deployments list --config wrangler.jsonc
pnpm --filter @unicas/docs-site exec wrangler rollback <docs-version-id> --config wrangler.jsonc --yes --message "rollback failed production release"

pnpm --filter @unicas/site exec wrangler deployments list --config wrangler.jsonc
pnpm --filter @unicas/site exec wrangler rollback <site-version-id> --config wrangler.jsonc --yes --message "rollback failed production release"

pnpm --filter @unicas/spaces exec wrangler deployments list --config ../../.wrangler/spaces/wrangler.production.json
pnpm --filter @unicas/spaces exec wrangler rollback <spaces-version-id> --config ../../.wrangler/spaces/wrangler.production.json --yes --message "rollback failed production release"

pnpm --filter @unicas/service-cloudflare exec wrangler deployments list
pnpm --filter @unicas/service-cloudflare exec wrangler rollback <service-version-id> --yes --message "rollback failed production release"

After rollback, rerun the canonical smoke with provisioned operator credentials and check https://api.unicas.work/health, https://console.unicas.work/, https://spaces.unicas.work/, https://unicas.work/, and https://docs.unicas.work/. Do not use a source rebuild as a substitute for an explicit version rollback: rebuilding a moving branch does not identify the artifact receiving traffic. The service rollback procedure was verified in the admin drill on 2026-08-26; include the two static Workers in the next full rollback drill.

#Backup and restore

Backup (manual or scheduled; daily target):

wrangler d1 export unicas-control --remote --no-schema --output <cutover>/unicas-control.sql
wrangler d1 export unicas-tenant --remote --no-schema --output <cutover>/unicas-tenant.sql
pnpm --filter @unicas/spaces exec wrangler d1 export SPACES_DB --remote --config ../../.wrangler/spaces/wrangler.production.json --output ../../.wrangler/spaces/unicas-spaces.sql

--remote is mandatory (without it wrangler exports an empty local DB). Store the SQL off-box (the gitignored .wrangler/ copy is a working backup, not a durable one). R2 content is referenced by node hashes in the physical unicas-tenant D1 backup; a restore re-verifies blobs through the canonical read path.

For the split-origin smoke-only cutover, review the live reset plan without executing it:

node release/service/reset-smoke.mjs `
   --expected-stack-id <current-smoke-stack-id>

This is a physical pre-cutover tool: its Stack/Tenant flags and output match the current D1/R2 schema and are not v2 aliases. The command refuses inventories containing another physical partition in any scoped control or data table or an unexpected object-key shape. The rendered D1 commands drop the legacy physical tables; they do not merely delete rows, because retained stack_id/tenant_id columns would block the new Worker from creating the clean App/Space schema.

With --backup-dir, pass a fresh empty directory. After maintenance is active, the script creates both D1 exports itself, downloads every physically enumerated R2 object, verifies each canonical object's SHA-256, and writes backup-manifest.json with D1 and R2 sizes and hashes. Existing export or manifest files abort the operation so stale backups cannot satisfy the guard.

Run the destructive command only inside the approved maintenance window, then deploy the new Worker immediately so it can create the app_id/space_id tables. Do not reuse this one-time pre-cutover tool after the physical cutover. After reviewing the printed R2, OAuth KV, and D1 commands, execute with the same explicit physical Stack ID and either a fresh backup directory:

node release/service/reset-smoke.mjs --execute `
   --expected-stack-id <current-smoke-stack-id> `
   --backup-dir <off-machine-cutover-directory>

or, only when test-data disposal is explicitly approved, omit the directory and add --confirm DELETE-ALL-TEST-DATA-NO-BACKUP.

The operation also refuses to start while either direct-upload signing secret is configured. Revoke the underlying R2 API token first; deleting only the Worker binding does not invalidate already signed URLs.

This tool targets only the isolated unicas-* resources committed in this repository. It has no legacy Worker, route, database, bucket, or namespace parameter.

Restore (disaster drill; destructive — clears target tables first):

wrangler d1 execute unicas-control --remote --command "<clear tables>"
wrangler d1 execute unicas-control --remote --file=unicas-control.sql
wrangler d1 execute unicas-tenant --remote --command "<clear tables>"
wrangler d1 execute unicas-tenant --remote --file=unicas-tenant.sql

The 2026-09-15 App/Space cutover verified both data-only exports by rebuilding the pre-cutover schemas in temporary SQLite databases, importing the exports, and checking the expected control/data counts. Every R2 backup was also rehashed against its canonical key. Production restore remains a destructive incident action and was not executed as part of the drill.

#App OAuth issuer key rotation

App signing keys come exclusively from the active issuer's discovered jwks_uri; UniCAS has no manual or copied active-key table. Rotation happens at the authorization server: it publishes the new key alongside the old one (overlap), and UniCAS refreshes the remote JWKS after the authority cache TTL and when an unknown kid is encountered outside the fetch cooldown.

Verification behavior:

• A key removed from a successfully refreshed JWKS stops validating immediately.
• Within the cache TTL the cached JWKS keeps verifying; past the hard stale bound a failed registry refresh fails closed (registry_unavailable) — it never silently falls back to a manual key set.

Operational checks:

1. Confirm the App's OAuth issuer is active (unicas app-oauth-issuer get <appId>).
2. After the provider rotates keys with overlap, confirm traffic still verifies with the new key after the refresh interval.
3. On suspected compromise, rotate the provider signing key, then verify the provider JWKS no longer lists the compromised kid and old tokens fail after the cache TTL.

#Key compromise

1. Rotate the authorization server's signing key so the compromised key stops being advertised (publish the replacement with overlap first).
2. Wait for UniCAS to refresh the remote JWKS and confirm tokens signed by the compromised kid are rejected.
3. Rotate any secrets that may share the compromise (audit reader key, session encryption keys) and review cas_control_audit_events for the affected window.

#Incident checklist

1. Confirm service /health; confirm /v1/cas + /admin/apps probes.
2. wrangler deployments list for unicas — recent deploy? Rollback first, diagnose later.
3. Query cas_app_authorization for fail_closed / unknown_issuer — registry reachability vs key/issuer config.
4. Check the service Worker's D1 bindings (CAS_CONTROL_DB, unicas-control) and wrangler d1 execute ... SELECT reachability.
5. After resolution, run the smoke twice (70 s apart) to confirm both the happy path and the cache-refresh path.

#Pending ops items

• Scheduled backup job (Workers Cron or external) with alerting on failure.
• Throwaway-D1 restore drill (destructive restore not yet executed).
• Alert delivery integration (Cloudflare alerting webhooks or an external monitor like Better Stack / Grafana) wired to the rules above.
• Cloudflare analytics/logpush consumption for the 5-min roll-ups.