From 12e59ad88a5e7d1ca105c2329b435c687c425a7e Mon Sep 17 00:00:00 2001 From: Ben Stull Date: Fri, 12 Jun 2026 01:34:39 -0700 Subject: [PATCH] update engineering/0023/SESSION-0023.0-TRANSCRIPT-2026-06-12T01-19--INPROGRESS.md --- ...TRANSCRIPT-2026-06-12T01-19--INPROGRESS.md | 42 ++++++++++++++----- 1 file changed, 32 insertions(+), 10 deletions(-) diff --git a/engineering/0023/SESSION-0023.0-TRANSCRIPT-2026-06-12T01-19--INPROGRESS.md b/engineering/0023/SESSION-0023.0-TRANSCRIPT-2026-06-12T01-19--INPROGRESS.md index a722cda..44f2e19 100644 --- a/engineering/0023/SESSION-0023.0-TRANSCRIPT-2026-06-12T01-19--INPROGRESS.md +++ b/engineering/0023/SESSION-0023.0-TRANSCRIPT-2026-06-12T01-19--INPROGRESS.md @@ -42,16 +42,30 @@ create), freshness keyed off newest-object age. 4. Re-install on the live forge VM; run the service manually; verify a fresh marker lands and a 2nd run refreshes (the create-not-overwrite proof). -**Increment 2 — SLICE-5 freshness checker (PR B):** -1. Checker (Cloud Run job/Function, project `wiggleverse`, config `gitea`): list - newest `gitea-dump-*.zip`, compute age, emit signal. -2. Alert delivery = Cloud Monitoring-native (no new secrets): custom metric + - alert policy → email notification channel; off-VM dead-man's-switch. -3. Provision by hand (bootstrap infra, D-6 — NOT flotilla): Scheduler + checker + - notification channel + alert policy. -4. T-3 green against scratch bucket (stale → fires; fresh → quiet) before live. -5. Finalize RB-3/RB-4 in `infra/backup-restore-gitea.md`; DOC-2 update; spec - SLICE-5 done + changelog. +**Increment 1 — DONE & SHIPPED.** Marker fix verified live (job exits 0, fresh +per-run marker lands). PR #59 merged → main `36acc77`. Live VM script == repo. + +**Increment 2 — SLICE-5 freshness checker (PR B) — design (session 0023 calls):** +- **Runtime:** Cloud Function gen2 (Python), HTTP-triggered, Cloud Scheduler every + 30 min. Project `wiggleverse`, config `gitea`. By-hand bootstrap infra (D-6 — NOT + flotilla). +- **Signal:** off-VM checker lists the bucket → age of newest `gitea-dump-*.zip` + AND newest `markers/last-success-*.json`; STALE if either > 6h (interval 4h × + 1.5, ALR-1). Marker age catches post-dump failures snapshot age misses. +- **Alert delivery = log-based** (no new secret, fast to verify): function emits a + structured `severity=ERROR` `GITEA_BACKUP_STALE` log when stale → log-based alert + policy → email notification channel (ben@wiggleverse.org). Off-VM path fires even + if the VM is gone (INV-4). Dedicated **read-only** SA (storage.objectViewer) — + distinct from the VM's create-only SA. +- **ALR-2 (job failure, Medium):** on-VM systemd `OnFailure=` drop-in writes a + `markers/last-failure-.json` (a create — works under objectCreator); checker + alerts if a failure marker is newer than the newest success marker → prompt + single-failure detection within least-privilege. +- **T-3:** invoke checker against the scratch bucket — stale seed → ERROR log + + STALE verdict; fresh seed → no ERROR log + FRESH verdict (verifiable in seconds + via Cloud Logging). Then enable against the live bucket. +- Finalize RB-3/RB-4 in `infra/backup-restore-gitea.md`; DOC-2 update; spec SLICE-5 + done + changelog. ## Deferred decisions @@ -65,3 +79,11 @@ finalize. Empty if none._ keying only off newest-dump age. Per-run markers keep the spec's TEL-1 marker artifact refreshing within least-privilege. Small spec data-model wording update follows; flagging in case the operator prefers a fixed canonical pointer. +- **SLICE-5 checker implementation (medium confidence):** chose Cloud Function gen2 + + Cloud Scheduler + **log-based** alert (vs custom-metric alert, vs a Cloud Run + job). Log-based is lowest-IAM and fastest to verify end-to-end; spec D-4 only + pins "Cloud Scheduler → Cloud Run/Function," leaving the rest open. Flag if the + operator prefers a metric/dashboard view of backup age over a log-based alert. +- **ALR-2 via on-VM OnFailure failure-marker:** adds a small systemd drop-in on the + forge VM. Reversible/additive. Flag if the operator prefers ALR-2 left as + covered-by-staleness only (no VM-side change).