# Session 0023.0 — Transcript

> App: engineering
> Start: 2026-06-12T01-19 (PST)
> Type: planning-and-executing
> Posture: yolo
> Claude-Session: 41a32da1-5f00-4cb9-81a1-edd730cfd9e2
> End: 2026-06-12T02-00 (PST)
> Status: **FINALIZED.** gitea-forge-backups SLICE-5 + marker-refresh fix shipped
> (PRs #59, #60, #61 merged); capability SLICE-1..5 complete.

## Next session

```
/goal Brainstorm engineering#44 — Solution Design for the two-tier E2E gate machinery (hermetic local harness per merge + same suite on PPE per deploy), handbook §10.3 as primary input (feature P1 under epic #26)
```

gitea-forge-backups is done. The next open work stream is engineering#44 (E2E gate
machinery) — a **brainstorming** session to produce its Solution Design.

## Pipeline status (§9)

Gitea is **bootstrap infra outside flotilla** (no PPE/blue-green, spec D-6). The
§9 pipeline's stand-in is the **scratch-environment gate** (§6.8/§7.3): T-1..T-4
green on a scratch Gitea + scratch bucket before touching the live forge. This
session satisfied it — **T-3 green against the scratch bucket** via the deployed
off-VM checker before relying on the live alert. No UI surface → **E2E browser
tests N/A**. The marker fix was verified directly on the live forge VM (additive,
read-only). No flotilla deploy / release tag applies to this bootstrap-infra work.

## Launch prompt

```
Resume stored goal (/goal next): gitea-forge-backups SLICE-5 — off-VM freshness/failure alert (Cloud Scheduler → Cloud Run/Function reading bucket object/marker age; T-3 green; finalize RB-3/RB-4), AND fix the broken marker-refresh path in gitea-backup.sh (markers/last-success.json ~24h stale while dumps land), per specs/gitea-forge-backups.md §7.2 (engineering#30)

```

## Plan

> Anchor: design `specs/gitea-forge-backups.md` (graduated Solution Design, engineering#30) — ELIGIBLE (R2a).
> Goal (`/goal next`): gitea-forge-backups SLICE-5 freshness alert + fix the broken marker-refresh path.

**Diagnosis (marker bug):** `markers/last-success.json` froze at the FIRST live run
(2026-06-11T04:46Z) while 4-hourly dumps keep landing. Root cause: the VM SA holds
only `roles/storage.objectCreator` (`storage.objects.create`), which **cannot
overwrite an existing live object**. Dumps use unique names → fresh create → OK.
The marker uses a FIXED name → every run after the first is an overwrite → 403 →
marker never refreshes; the job exits non-zero *after* the dump uploaded. Fix must
NOT grant `objects.delete` (would let a compromised VM destroy backups — breaks
§6.6/R-4 least-privilege). Correct fix = unique-named per-run markers (always a
create), freshness keyed off newest-object age.

**Increment 1 — Marker-refresh fix (PR A):**
1. `infra/gitea-backup/gitea-backup.sh`: write the marker under a per-run name
   (`markers/last-success-<ts>.json`) — always a create; also keep richer fields.
2. Spec §6.3/§6.4 data-model wording (marker key is per-run; newest-object age is
   the primary freshness signal) + changelog bump.
3. DOC-2 (`infra/deploy-gitea-on-gcp.md`) backups section update.
4. Re-install on the live forge VM; run the service manually; verify a fresh marker
   lands and a 2nd run refreshes (the create-not-overwrite proof).

**Increment 1 — DONE & SHIPPED.** Marker fix verified live (job exits 0, fresh
per-run marker lands). PR #59 merged → main `36acc77`. Live VM script == repo.

**Increment 2 — SLICE-5 freshness checker (PR B) — design (session 0023 calls):**
- **Runtime:** Cloud Function gen2 (Python), HTTP-triggered, Cloud Scheduler every
  30 min. Project `wiggleverse`, config `gitea`. By-hand bootstrap infra (D-6 — NOT
  flotilla).
- **Signal:** off-VM checker lists the bucket → age of newest `gitea-dump-*.zip`
  AND newest `markers/last-success-*.json`; STALE if either > 6h (interval 4h ×
  1.5, ALR-1). Marker age catches post-dump failures snapshot age misses.
- **Alert delivery = log-based** (no new secret, fast to verify): function emits a
  structured `severity=ERROR` `GITEA_BACKUP_STALE` log when stale → log-based alert
  policy → email notification channel (ben@wiggleverse.org). Off-VM path fires even
  if the VM is gone (INV-4). Dedicated **read-only** SA (storage.objectViewer) —
  distinct from the VM's create-only SA.
- **ALR-2 (job failure, Medium):** on-VM systemd `OnFailure=` drop-in writes a
  `markers/last-failure-<ts>.json` (a create — works under objectCreator); checker
  alerts if a failure marker is newer than the newest success marker → prompt
  single-failure detection within least-privilege.
- **T-3:** invoke checker against the scratch bucket — stale seed → ERROR log +
  STALE verdict; fresh seed → no ERROR log + FRESH verdict (verifiable in seconds
  via Cloud Logging). Then enable against the live bucket.
- Finalize RB-3/RB-4 in `infra/backup-restore-gitea.md`; DOC-2 update; spec SLICE-5
  done + changelog.

## Outcome — gitea-forge-backups SLICE-5 + marker fix COMPLETE

Both halves of the goal shipped; the gitea-forge-backups capability (SLICE-1..5) is
now complete (recovery points exist off-host, retention proven, restore proven,
lapses surfaced).

- **Marker-refresh fix (PR #59, merged):** root-caused to `objectCreator` being
  create-only (can't overwrite the fixed-name marker → HTTP 403, proven from the
  job journal). Fixed to unique per-run marker names; verified live (`systemctl
  start gitea-backup.service` exits 0, fresh marker lands). The backup job had been
  failing every 4 h for ~27 h — now green.
- **SLICE-5 freshness checker (PR #60, merged):** off-VM Cloud Function gen2 +
  Cloud Scheduler (30 min) + log-based alert policy → email channel; ALR-1 (stale
  snapshot/marker > 6 h) + ALR-2 (on-VM `OnFailure=` failure-marker). Dedicated
  read-only runtime SA + dedicated build SA — VM's create-only SA untouched.
  **T-3 green** via the deployed off-VM path against the scratch bucket. RB-3/RB-4
  finalized; DOC-2 updated; spec v0.2.5.
- **Alert scope fix (PR #61, merged):** scoped the alert policy to the live bucket
  so T-3 scratch runs don't page the operator.
- **Plan archived:** `plans/2026-06-12-gitea-backups-slice-5.md` (on main, 617789b).

**Live GCP state (project wiggleverse, config gitea):** function
`gitea-backup-freshness-checker` ACTIVE; scheduler `gitea-backup-freshness-30m`
ENABLED; alert policy `11761102391334510705` (scoped to live bucket) → email
channel `8452564362810973845` (ben@wiggleverse.org). Steady-state scheduled run
logs `GITEA_BACKUP_OK`. Checker SA has read on the scratch bucket (for T-3 reruns).

## Deferred decisions

_Autonomous-mode low-confidence calls the driver made and would have
liked operator input on. Appended as the session runs; surfaced at
finalize. Empty if none._

- **Marker fix shape (medium-high confidence):** chose unique-named per-run markers
  over (a) granting `objects.delete` — rejected, breaks the §6.6/R-4 "VM can create
  but not delete" security invariant — or (b) dropping the marker entirely and
  keying only off newest-dump age. Per-run markers keep the spec's TEL-1 marker
  artifact refreshing within least-privilege. Small spec data-model wording update
  follows; flagging in case the operator prefers a fixed canonical pointer.
- **SLICE-5 checker implementation (medium confidence):** chose Cloud Function gen2
  + Cloud Scheduler + **log-based** alert (vs custom-metric alert, vs a Cloud Run
  job). Log-based is lowest-IAM and fastest to verify end-to-end; spec D-4 only
  pins "Cloud Scheduler → Cloud Run/Function," leaving the rest open. Flag if the
  operator prefers a metric/dashboard view of backup age over a log-based alert.
- **ALR-2 via on-VM OnFailure failure-marker:** adds a small systemd drop-in on the
  forge VM. Reversible/additive. Flag if the operator prefers ALR-2 left as
  covered-by-staleness only (no VM-side change).
- **One-time T-3 test alert email (FYI):** before the alert policy was scoped to the
  live bucket (PR #61), T-3 emitted `GITEA_BACKUP_ALERT` logs against the scratch
  bucket that matched the policy — so a single test alert email may have reached
  ben@wiggleverse.org around 08:48–08:51 UTC. It auto-resolves (steady state is OK)
  and future T-3 runs no longer page (scoped to the live bucket).
- **Project IAM grants during provisioning (FYI):** granted `artifactregistry.admin`
  to the `gcf-admin-robot` + `gcp-sa-cloudbuild` service agents (standard first-deploy
  fix) and `cloudbuild.builds.builder` to the dedicated `gitea-checker-build` SA.
  Briefly granted then **reverted** four roles on the compute/VM SA (it doubles as
  the backup uploader and must stay create-only) — confirmed back to zero project
  roles.