145 lines
8.5 KiB
Markdown
145 lines
8.5 KiB
Markdown
# Session 0023.0 — Transcript
|
||
|
||
> App: engineering
|
||
> Start: 2026-06-12T01-19 (PST)
|
||
> Type: planning-and-executing
|
||
> Posture: yolo
|
||
> Claude-Session: 41a32da1-5f00-4cb9-81a1-edd730cfd9e2
|
||
> End: 2026-06-12T02-00 (PST)
|
||
> Status: **FINALIZED.** gitea-forge-backups SLICE-5 + marker-refresh fix shipped
|
||
> (PRs #59, #60, #61 merged); capability SLICE-1..5 complete.
|
||
|
||
## Next session
|
||
|
||
```
|
||
/goal Brainstorm engineering#44 — Solution Design for the two-tier E2E gate machinery (hermetic local harness per merge + same suite on PPE per deploy), handbook §10.3 as primary input (feature P1 under epic #26)
|
||
```
|
||
|
||
gitea-forge-backups is done. The next open work stream is engineering#44 (E2E gate
|
||
machinery) — a **brainstorming** session to produce its Solution Design.
|
||
|
||
## Pipeline status (§9)
|
||
|
||
Gitea is **bootstrap infra outside flotilla** (no PPE/blue-green, spec D-6). The
|
||
§9 pipeline's stand-in is the **scratch-environment gate** (§6.8/§7.3): T-1..T-4
|
||
green on a scratch Gitea + scratch bucket before touching the live forge. This
|
||
session satisfied it — **T-3 green against the scratch bucket** via the deployed
|
||
off-VM checker before relying on the live alert. No UI surface → **E2E browser
|
||
tests N/A**. The marker fix was verified directly on the live forge VM (additive,
|
||
read-only). No flotilla deploy / release tag applies to this bootstrap-infra work.
|
||
|
||
## Launch prompt
|
||
|
||
```
|
||
Resume stored goal (/goal next): gitea-forge-backups SLICE-5 — off-VM freshness/failure alert (Cloud Scheduler → Cloud Run/Function reading bucket object/marker age; T-3 green; finalize RB-3/RB-4), AND fix the broken marker-refresh path in gitea-backup.sh (markers/last-success.json ~24h stale while dumps land), per specs/gitea-forge-backups.md §7.2 (engineering#30)
|
||
|
||
```
|
||
|
||
## Plan
|
||
|
||
> Anchor: design `specs/gitea-forge-backups.md` (graduated Solution Design, engineering#30) — ELIGIBLE (R2a).
|
||
> Goal (`/goal next`): gitea-forge-backups SLICE-5 freshness alert + fix the broken marker-refresh path.
|
||
|
||
**Diagnosis (marker bug):** `markers/last-success.json` froze at the FIRST live run
|
||
(2026-06-11T04:46Z) while 4-hourly dumps keep landing. Root cause: the VM SA holds
|
||
only `roles/storage.objectCreator` (`storage.objects.create`), which **cannot
|
||
overwrite an existing live object**. Dumps use unique names → fresh create → OK.
|
||
The marker uses a FIXED name → every run after the first is an overwrite → 403 →
|
||
marker never refreshes; the job exits non-zero *after* the dump uploaded. Fix must
|
||
NOT grant `objects.delete` (would let a compromised VM destroy backups — breaks
|
||
§6.6/R-4 least-privilege). Correct fix = unique-named per-run markers (always a
|
||
create), freshness keyed off newest-object age.
|
||
|
||
**Increment 1 — Marker-refresh fix (PR A):**
|
||
1. `infra/gitea-backup/gitea-backup.sh`: write the marker under a per-run name
|
||
(`markers/last-success-<ts>.json`) — always a create; also keep richer fields.
|
||
2. Spec §6.3/§6.4 data-model wording (marker key is per-run; newest-object age is
|
||
the primary freshness signal) + changelog bump.
|
||
3. DOC-2 (`infra/deploy-gitea-on-gcp.md`) backups section update.
|
||
4. Re-install on the live forge VM; run the service manually; verify a fresh marker
|
||
lands and a 2nd run refreshes (the create-not-overwrite proof).
|
||
|
||
**Increment 1 — DONE & SHIPPED.** Marker fix verified live (job exits 0, fresh
|
||
per-run marker lands). PR #59 merged → main `36acc77`. Live VM script == repo.
|
||
|
||
**Increment 2 — SLICE-5 freshness checker (PR B) — design (session 0023 calls):**
|
||
- **Runtime:** Cloud Function gen2 (Python), HTTP-triggered, Cloud Scheduler every
|
||
30 min. Project `wiggleverse`, config `gitea`. By-hand bootstrap infra (D-6 — NOT
|
||
flotilla).
|
||
- **Signal:** off-VM checker lists the bucket → age of newest `gitea-dump-*.zip`
|
||
AND newest `markers/last-success-*.json`; STALE if either > 6h (interval 4h ×
|
||
1.5, ALR-1). Marker age catches post-dump failures snapshot age misses.
|
||
- **Alert delivery = log-based** (no new secret, fast to verify): function emits a
|
||
structured `severity=ERROR` `GITEA_BACKUP_STALE` log when stale → log-based alert
|
||
policy → email notification channel (ben@wiggleverse.org). Off-VM path fires even
|
||
if the VM is gone (INV-4). Dedicated **read-only** SA (storage.objectViewer) —
|
||
distinct from the VM's create-only SA.
|
||
- **ALR-2 (job failure, Medium):** on-VM systemd `OnFailure=` drop-in writes a
|
||
`markers/last-failure-<ts>.json` (a create — works under objectCreator); checker
|
||
alerts if a failure marker is newer than the newest success marker → prompt
|
||
single-failure detection within least-privilege.
|
||
- **T-3:** invoke checker against the scratch bucket — stale seed → ERROR log +
|
||
STALE verdict; fresh seed → no ERROR log + FRESH verdict (verifiable in seconds
|
||
via Cloud Logging). Then enable against the live bucket.
|
||
- Finalize RB-3/RB-4 in `infra/backup-restore-gitea.md`; DOC-2 update; spec SLICE-5
|
||
done + changelog.
|
||
|
||
## Outcome — gitea-forge-backups SLICE-5 + marker fix COMPLETE
|
||
|
||
Both halves of the goal shipped; the gitea-forge-backups capability (SLICE-1..5) is
|
||
now complete (recovery points exist off-host, retention proven, restore proven,
|
||
lapses surfaced).
|
||
|
||
- **Marker-refresh fix (PR #59, merged):** root-caused to `objectCreator` being
|
||
create-only (can't overwrite the fixed-name marker → HTTP 403, proven from the
|
||
job journal). Fixed to unique per-run marker names; verified live (`systemctl
|
||
start gitea-backup.service` exits 0, fresh marker lands). The backup job had been
|
||
failing every 4 h for ~27 h — now green.
|
||
- **SLICE-5 freshness checker (PR #60, merged):** off-VM Cloud Function gen2 +
|
||
Cloud Scheduler (30 min) + log-based alert policy → email channel; ALR-1 (stale
|
||
snapshot/marker > 6 h) + ALR-2 (on-VM `OnFailure=` failure-marker). Dedicated
|
||
read-only runtime SA + dedicated build SA — VM's create-only SA untouched.
|
||
**T-3 green** via the deployed off-VM path against the scratch bucket. RB-3/RB-4
|
||
finalized; DOC-2 updated; spec v0.2.5.
|
||
- **Alert scope fix (PR #61, merged):** scoped the alert policy to the live bucket
|
||
so T-3 scratch runs don't page the operator.
|
||
- **Plan archived:** `plans/2026-06-12-gitea-backups-slice-5.md` (on main, 617789b).
|
||
|
||
**Live GCP state (project wiggleverse, config gitea):** function
|
||
`gitea-backup-freshness-checker` ACTIVE; scheduler `gitea-backup-freshness-30m`
|
||
ENABLED; alert policy `11761102391334510705` (scoped to live bucket) → email
|
||
channel `8452564362810973845` (ben@wiggleverse.org). Steady-state scheduled run
|
||
logs `GITEA_BACKUP_OK`. Checker SA has read on the scratch bucket (for T-3 reruns).
|
||
|
||
## Deferred decisions
|
||
|
||
_Autonomous-mode low-confidence calls the driver made and would have
|
||
liked operator input on. Appended as the session runs; surfaced at
|
||
finalize. Empty if none._
|
||
|
||
- **Marker fix shape (medium-high confidence):** chose unique-named per-run markers
|
||
over (a) granting `objects.delete` — rejected, breaks the §6.6/R-4 "VM can create
|
||
but not delete" security invariant — or (b) dropping the marker entirely and
|
||
keying only off newest-dump age. Per-run markers keep the spec's TEL-1 marker
|
||
artifact refreshing within least-privilege. Small spec data-model wording update
|
||
follows; flagging in case the operator prefers a fixed canonical pointer.
|
||
- **SLICE-5 checker implementation (medium confidence):** chose Cloud Function gen2
|
||
+ Cloud Scheduler + **log-based** alert (vs custom-metric alert, vs a Cloud Run
|
||
job). Log-based is lowest-IAM and fastest to verify end-to-end; spec D-4 only
|
||
pins "Cloud Scheduler → Cloud Run/Function," leaving the rest open. Flag if the
|
||
operator prefers a metric/dashboard view of backup age over a log-based alert.
|
||
- **ALR-2 via on-VM OnFailure failure-marker:** adds a small systemd drop-in on the
|
||
forge VM. Reversible/additive. Flag if the operator prefers ALR-2 left as
|
||
covered-by-staleness only (no VM-side change).
|
||
- **One-time T-3 test alert email (FYI):** before the alert policy was scoped to the
|
||
live bucket (PR #61), T-3 emitted `GITEA_BACKUP_ALERT` logs against the scratch
|
||
bucket that matched the policy — so a single test alert email may have reached
|
||
ben@wiggleverse.org around 08:48–08:51 UTC. It auto-resolves (steady state is OK)
|
||
and future T-3 runs no longer page (scoped to the live bucket).
|
||
- **Project IAM grants during provisioning (FYI):** granted `artifactregistry.admin`
|
||
to the `gcf-admin-robot` + `gcp-sa-cloudbuild` service agents (standard first-deploy
|
||
fix) and `cloudbuild.builds.builder` to the dedicated `gitea-checker-build` SA.
|
||
Briefly granted then **reverted** four roles on the compute/VM SA (it doubles as
|
||
the backup uploader and must stay create-only) — confirmed back to zero project
|
||
roles.
|