# Session 0073.0 — Transcript — PPE / progressive-delivery design, engineering handbook, ROADMAP Phase F

> **Renumber note (2026-06-05):** this session was originally drafted as
> `0046.0`, but that ordinal was already taken by a different, published
> session (the two-tier-testing-refinement driver). On migration to
> `wiggleverse/session-history` it was reassigned the next free number,
> `0073`. Body text below is verbatim and may still refer to "0046".

- **Start:** 2026-05-29T21-30 *(approximate — the harness could not
  recover the exact session-open time; the prior session's placeholder
  0045 is stamped 21-26, and this session's first git action was the
  engineering-handbook seed commit at 2026-05-29 22:44 -0400)*
- **End:** 2026-05-30T06-19
- **Operator:** Ben Stull
- **Driver:** Claude (claude-opus-4-8 via Claude Code)
- **Repos touched:** `wiggleverse/engineering` (created + seeded + §10.3
  update), `ben.stull/ohm-rfc` (ROADMAP Phase F via PR #3; two-tier
  refinement via PRs #4 and #5), plus read-only clones of `rfc-app`,
  `ohm-content`
- **Pin moves:** none — no rfc-app release, no `flotilla deploy`
- **TL;DR:** What began as "create a pre-prod env so we can run full
  automated browser tests without screwing up production data" grew, in
  conversation, into a full progressive-delivery design (PPE → automated
  test gate → blue/green slots → weighted sticky canary → bake monitoring
  → auto-rollback, all on the one existing VM with a shared forward-only
  SQLite). That design was written up as a new org-level engineering
  handbook (`wiggleverse/engineering`), and sliced into the OHM roadmap as
  **Phase F / Track Δ, items #37–#42**. A late operator refinement —
  two-tier testing (local Docker in the dev loop, then PPE on the VM) —
  was folded into both. No code shipped; PPE itself is not built.

---

## How the scope grew

The opening ask was narrow: a pre-prod environment to run automated
browser tests without endangering production data. Over several exchanges
the operator widened it deliberately:

1. First to a real environment with isolation guarantees.
2. Then: "plan for deployment to deploy to PPE, then run automated tests
   (including browser automation for core scenarios), then do a blue/green
   deployment and monitor real user behavior and roll back if there's any
   sign of trouble."
3. Then a key architectural question: "Can we run all three instances
   (ppe, blue, green) on a single VM? PPE would have its own database.
   blue/green wouldn't but databases are roll-forward only. A portion of
   users are sent to the newer version and the cohort increases as
   confidence grows."
4. Then the insight that anchors the safety model: "Schema changes should
   be backwards compatible with the last version of code though, right?
   We should be testing that when we deploy to PPE."
5. "We'll also need infra for e2e tests in ppe. Specifically, email
   accounts for invites, login, notifications."
6. And, late: "Could automated tests run against a local docker container
   first during development and then on PPE on the VM when we're actually
   deploying?"

The driver's job became: pressure-test the design honestly, fill the
gaps, and write it down.

## The design that settled (handbook §10)

Verified against the real repos (after cloning `rfc-app`, which was not
initially on the machine): the OHM stack is a single GCE VM, **nginx**
reverse proxy, **systemd** units, a **Vite** frontend built on the VM,
and a single-file **SQLite** database (WAL). rfc-app migrations are
numbered and **forward-only** (`backend/migrations/001…024`, no
down-migrations) — exactly what makes the proposed canary safe.

- **Three instances on one VM.** PPE (own DB, seedable/wipeable) + prod
  **blue** and **green** code slots that *share* the one prod SQLite file
  (WAL: many readers, one serialized writer). Isolation is by install dir
  / systemd unit / DB path, not new hardware. Per-unit
  `MemoryMax`/`CPUQuota` so a runaway slot (esp. the on-VM frontend build)
  can't OOM the box.
- **The safety invariant** is stronger than "roll-forward only": it's
  **expand/contract** (parallel-change). Because both slots hit the same
  schema during a ramp, every migration must keep the *previous* code
  version working (N schema must not break N−1 code). Additive changes are
  safe; rename/drop/retype ship as expand-now / contract-next-release.
- **The gate tests the combination a normal deploy skips.** A vanilla
  "deploy N and test" only exercises N-code-on-N-schema; the canary runs
  N−1-code-on-N-schema for the whole ramp. So the gate seeds the N−1
  baseline → applies N's migrations → runs the suite against N−1 code on
  the migrated DB (compat) *and* N code (forward), exercising writes both
  directions. Fail compat ⇒ not canary-safe ⇒ block the ramp.
- **Canary, not binary flip.** nginx `split_clients` on a stable per-user
  cookie → consistent slot assignment; ramp = edit % + reload. Rollback is
  DB-free (weight bad slot to 0 + reload; schema only moved forward
  compatibly).
- **Monitoring/rollback** trips on a *fast technical* signal (5xx /
  health / latency), not product analytics. The genuine open fork: an
  always-on watcher next to prod vs. a bounded bake during the deploy
  session — in tension with flotilla's laptop-only commitment. Left as an
  explicit OPEN decision, not defaulted.
- **Email infra for e2e.** A self-hosted **mail sink** (Mailpit). The
  e2e suite reads OTC codes / invite links back via the sink's API.
  Catches mail to any address, so multi-user invite scenarios need no
  provisioned mailboxes. Deliverability (SPF/DKIM/DMARC) is kept separate
  and non-gating (already roadmap #20).
- **Two tiers, one suite** (operator's final refinement). The e2e suite
  is **environment-agnostic** — one `BASE_URL` (+ mail-sink URL).
  **Tier 1**: a local `docker compose` stack (backend + built frontend +
  fresh SQLite + Mailpit) run in the dev loop and in CI on every PR —
  fast, offline, and **not dependent on #37**, so the suite can be green
  locally before PPE on the VM exists. **Tier 2**: the *same* suite vs
  PPE as the deploy gate (prod-like confirmation). The N−1/N compat gate
  runs cheaply in Tier 1 (old image → migrate → test → new image), with
  PPE as confirmation. The discipline: the suite is the unit of value;
  each environment is just another `BASE_URL`.

## Artifacts produced

1. **`wiggleverse/engineering`** — a new org-level repo holding a
   project-independent handbook, "How We Build Software at Wiggleverse"
   (the repo's `README.md`, ~610 lines). Covers: vibe-code prototypes
   first → write the first SPEC from the prototype → spec-driven
   development → git/repo conventions → sessions & transcripts (linked to
   the spec) → roadmap + parallel-agent sessions → the standard OHM stack
   and how flotilla deploys it → three infra stages (Stage 1 pre-v1
   single-prod; Stage 2 v1+users PPE/blue-green/canary/rollback; Stage 3
   deferred, name-the-trigger). The progressive-delivery design is its
   §10; the two-tier testing refinement is §10.3 (commit `0bbf759` on
   `main`). Repo created empty by the operator; seeded by pushing the
   guide straight to `main` (an empty repo has no base branch to PR
   against), then default branch set to `main` and the leftover seed
   branch deleted via the Gitea API.

2. **`ohm-rfc/ROADMAP.md` Phase F (#37–#42)** — the design sliced into
   roadmap items (item bodies + version-targets table + a new **Track Δ**
   in the parallelization plan):
   - #37 PPE pre-prod env (`ppe.ohm.wiggleverse.org`) — incl. the mail sink
   - #38 Automated + browser E2E suite (rfc-app) + PPE test gate
   - #39 Blue/green prod slots on the VM (slot model)
   - #40 Forward-only migration phase + N−1/N compatibility gate
   - #41 Weighted, sticky canary routing + promote/rollback
   - #42 Bake monitoring + auto-rollback (open: watcher vs bounded bake)
   All marked TBD and **gated on v1**, with handbook §10 as the binding
   design and an explicit "proposal, not yet built" note (flotilla v1
   defers blue/green + auto-rollback per its SPEC §19.2). Landed via
   **PR #3**. The two-tier-testing refinement was then folded in across
   two follow-up PRs: **PR #4** (#40 body + #38 table row) and **PR #5**
   (#38 body) — #38 Tier 1 explicitly does NOT depend on #37. Final
   `ohm-rfc` `main` after all three: `2d6f6d6`.

## Wrong turns (left in, per the honest-transcript rule)

- **Harness output corruption — pervasive this session.** Bash stdout and
  Read repeatedly returned garbled or *fabricated* content for larger
  outputs — at one point inventing a plausible-but-fake Playwright config,
  seed scripts, and Caddyfile for rfc-app *before that repo was even
  cloned*; later injecting junk tokens and fabricated sentences into a
  `git diff`, a Read result, a grep over this very transcript, and script
  reads at finalize time. Several `Edit` calls also silently failed
  (string-not-found, because a corrupted earlier Read had given the driver
  slightly-wrong "current" text) — so a handbook §10.3 edit and a roadmap
  #38-body edit each had to be retried after re-reading. Each time the
  driver caught it, discarded the bad output, and re-verified via small
  file-based commands and the public raw URLs. This transcript was written
  wholesale via `Write` so its bytes did not depend on an unreadable
  on-disk state. **This is the single most important caveat for the next
  session: do not trust large tool dumps here; verify with minimal
  commands and authoritative remotes.**

- **"ohm-infra doesn't exist" — false, caused by the corruption.** Early
  on, the driver concluded `~/git/ohm-infra/` (and the session tooling)
  was absent on this machine, and even hand-built a throwaway
  `~/projects/wiggleverse/ohm-session-history` clone to work around it.
  That was wrong — `~/git/ohm-infra/` exists, with
  `scripts/claim-session-id.sh`, `scripts/publish-transcript.sh`, and
  `SESSION-PROTOCOL.md`, and is the canonical home for transcripts
  (0043/0044/0045 present). This transcript was written to the correct
  location and published via the real script.

- **Roadmap item-numbering.** A corrupted early read of ROADMAP.md stopped
  at item #17, so the driver first drafted the new items as #18–#23 —
  which would have collided with the real, longer backlog (items run
  through #36). The edits happened to fail (string-not-found), nothing
  landed wrong, and Phase F was renumbered to the correct **#37–#42**.

- **Wrong Gitea token.** First PR attempt used the GCP secret
  `ohm-rfc-app-gitea-bot-token`, which authenticates as `ohm-bot` — *not*
  a collaborator on `ben.stull/ohm-rfc` → 403 "user must be a
  collaborator." Correct credential: the macOS **Keychain** PAT
  (`security find-generic-password -s ohm-gitea-token`), Ben-owned,
  `write:repository`. Memory corrected.

- **Merge conflict (PR #3).** `origin/main` had advanced ~10 commits since
  the branch base (the #36 meta-only settlement + pin bumps), so the PR
  came up `mergeable:false`. Resolved by merging `origin/main` into the
  feature branch (one ROADMAP.md conflict — kept the now-struck-through
  #36 row from main, appended #37–#42), pushing, then merging.

## Git operations (all verified against the public remotes)

- `wiggleverse/engineering`: seeded to `main`; default branch set to
  `main`; leftover `seed-engineering-handbook` deleted; §10.3 two-tier
  section committed `0bbf759` and pushed (confirmed live on the public
  raw README).
- `ben.stull/ohm-rfc`: Phase F via PR #3 → merge; two-tier refinement via
  PR #4 (#40 + #38 table row) and PR #5 (#38 body); all three feature
  branches deleted on remote; local `main` fast-forwarded to `2d6f6d6`;
  tree clean.

## Secrets discipline

No secret bytes entered the conversation or any committed artifact. Tokens
used for API calls were read into `/tmp` files / shell vars, never echoed,
and the `/tmp` token files were truncated to zero at session end; only
env-var *names* and GCP secret *resource paths* were ever surfaced.

## Not done / open

- **PPE is not built.** This session only designed and recorded it. #37 is
  the next concrete build, but its live-on-VM parts are **gated on v1** —
  confirm timing with the operator. Note #38 Tier 1 (local Docker suite)
  is NOT gated on #37 and can be started independently.
- **Open design decisions** in the roadmap: #42 auto-rollback model
  (watcher vs bounded bake); #37 OAuth-per-host caveat (dissolves if #5
  email/OTC has shipped).
- **Session 0045 is unfinalized** (`SESSION-0045.0-TRANSCRIPT-…--INPROGRESS.md`
  in `~/git/ohm-infra/`) — a prior pin-bump session's loose end, not this
  session's. Flagged for cleanup; was unrelated to 0046's publish.
- **Stray read-only clone:** the driver made a
  `~/projects/wiggleverse/ohm-session-history` clone while (wrongly)
  believing the canonical tooling was absent; harmless and uncommitted,
  can be removed.
