90 lines
5.2 KiB
Markdown
90 lines
5.2 KiB
Markdown
# Session 0023.0 — Transcript
|
||
|
||
> App: engineering
|
||
> Start: 2026-06-12T01-19 (PST)
|
||
> Type: planning-and-executing
|
||
> Posture: yolo
|
||
> Claude-Session: 41a32da1-5f00-4cb9-81a1-edd730cfd9e2
|
||
> Status: **PLACEHOLDER — claimed at session start; finalized at session end.**
|
||
>
|
||
> This file reserves session ID 0023 for engineering. The driver replaces this
|
||
> body with the full transcript and renames the file to its final
|
||
> SESSION-0023.0-TRANSCRIPT-2026-06-12T01-19--<end>.md form at session end.
|
||
|
||
## Launch prompt
|
||
|
||
```
|
||
Resume stored goal (/goal next): gitea-forge-backups SLICE-5 — off-VM freshness/failure alert (Cloud Scheduler → Cloud Run/Function reading bucket object/marker age; T-3 green; finalize RB-3/RB-4), AND fix the broken marker-refresh path in gitea-backup.sh (markers/last-success.json ~24h stale while dumps land), per specs/gitea-forge-backups.md §7.2 (engineering#30)
|
||
|
||
```
|
||
|
||
## Plan
|
||
|
||
> Anchor: design `specs/gitea-forge-backups.md` (graduated Solution Design, engineering#30) — ELIGIBLE (R2a).
|
||
> Goal (`/goal next`): gitea-forge-backups SLICE-5 freshness alert + fix the broken marker-refresh path.
|
||
|
||
**Diagnosis (marker bug):** `markers/last-success.json` froze at the FIRST live run
|
||
(2026-06-11T04:46Z) while 4-hourly dumps keep landing. Root cause: the VM SA holds
|
||
only `roles/storage.objectCreator` (`storage.objects.create`), which **cannot
|
||
overwrite an existing live object**. Dumps use unique names → fresh create → OK.
|
||
The marker uses a FIXED name → every run after the first is an overwrite → 403 →
|
||
marker never refreshes; the job exits non-zero *after* the dump uploaded. Fix must
|
||
NOT grant `objects.delete` (would let a compromised VM destroy backups — breaks
|
||
§6.6/R-4 least-privilege). Correct fix = unique-named per-run markers (always a
|
||
create), freshness keyed off newest-object age.
|
||
|
||
**Increment 1 — Marker-refresh fix (PR A):**
|
||
1. `infra/gitea-backup/gitea-backup.sh`: write the marker under a per-run name
|
||
(`markers/last-success-<ts>.json`) — always a create; also keep richer fields.
|
||
2. Spec §6.3/§6.4 data-model wording (marker key is per-run; newest-object age is
|
||
the primary freshness signal) + changelog bump.
|
||
3. DOC-2 (`infra/deploy-gitea-on-gcp.md`) backups section update.
|
||
4. Re-install on the live forge VM; run the service manually; verify a fresh marker
|
||
lands and a 2nd run refreshes (the create-not-overwrite proof).
|
||
|
||
**Increment 1 — DONE & SHIPPED.** Marker fix verified live (job exits 0, fresh
|
||
per-run marker lands). PR #59 merged → main `36acc77`. Live VM script == repo.
|
||
|
||
**Increment 2 — SLICE-5 freshness checker (PR B) — design (session 0023 calls):**
|
||
- **Runtime:** Cloud Function gen2 (Python), HTTP-triggered, Cloud Scheduler every
|
||
30 min. Project `wiggleverse`, config `gitea`. By-hand bootstrap infra (D-6 — NOT
|
||
flotilla).
|
||
- **Signal:** off-VM checker lists the bucket → age of newest `gitea-dump-*.zip`
|
||
AND newest `markers/last-success-*.json`; STALE if either > 6h (interval 4h ×
|
||
1.5, ALR-1). Marker age catches post-dump failures snapshot age misses.
|
||
- **Alert delivery = log-based** (no new secret, fast to verify): function emits a
|
||
structured `severity=ERROR` `GITEA_BACKUP_STALE` log when stale → log-based alert
|
||
policy → email notification channel (ben@wiggleverse.org). Off-VM path fires even
|
||
if the VM is gone (INV-4). Dedicated **read-only** SA (storage.objectViewer) —
|
||
distinct from the VM's create-only SA.
|
||
- **ALR-2 (job failure, Medium):** on-VM systemd `OnFailure=` drop-in writes a
|
||
`markers/last-failure-<ts>.json` (a create — works under objectCreator); checker
|
||
alerts if a failure marker is newer than the newest success marker → prompt
|
||
single-failure detection within least-privilege.
|
||
- **T-3:** invoke checker against the scratch bucket — stale seed → ERROR log +
|
||
STALE verdict; fresh seed → no ERROR log + FRESH verdict (verifiable in seconds
|
||
via Cloud Logging). Then enable against the live bucket.
|
||
- Finalize RB-3/RB-4 in `infra/backup-restore-gitea.md`; DOC-2 update; spec SLICE-5
|
||
done + changelog.
|
||
|
||
## Deferred decisions
|
||
|
||
_Autonomous-mode low-confidence calls the driver made and would have
|
||
liked operator input on. Appended as the session runs; surfaced at
|
||
finalize. Empty if none._
|
||
|
||
- **Marker fix shape (medium-high confidence):** chose unique-named per-run markers
|
||
over (a) granting `objects.delete` — rejected, breaks the §6.6/R-4 "VM can create
|
||
but not delete" security invariant — or (b) dropping the marker entirely and
|
||
keying only off newest-dump age. Per-run markers keep the spec's TEL-1 marker
|
||
artifact refreshing within least-privilege. Small spec data-model wording update
|
||
follows; flagging in case the operator prefers a fixed canonical pointer.
|
||
- **SLICE-5 checker implementation (medium confidence):** chose Cloud Function gen2
|
||
+ Cloud Scheduler + **log-based** alert (vs custom-metric alert, vs a Cloud Run
|
||
job). Log-based is lowest-IAM and fastest to verify end-to-end; spec D-4 only
|
||
pins "Cloud Scheduler → Cloud Run/Function," leaving the rest open. Flag if the
|
||
operator prefers a metric/dashboard view of backup age over a log-based alert.
|
||
- **ALR-2 via on-VM OnFailure failure-marker:** adds a small systemd drop-in on the
|
||
forge VM. Reversible/additive. Flag if the operator prefers ALR-2 left as
|
||
covered-by-staleness only (no VM-side change).
|