# Session 0073.0 — Transcript — PPE / progressive-delivery design, engineering handbook, ROADMAP Phase F > **Renumber note (2026-06-05):** this session was originally drafted as > `0046.0`, but that ordinal was already taken by a different, published > session (the two-tier-testing-refinement driver). On migration to > `wiggleverse/session-history` it was reassigned the next free number, > `0073`. Body text below is verbatim and may still refer to "0046". - **Start:** 2026-05-29T21-30 *(approximate — the harness could not recover the exact session-open time; the prior session's placeholder 0045 is stamped 21-26, and this session's first git action was the engineering-handbook seed commit at 2026-05-29 22:44 -0400)* - **End:** 2026-05-30T06-19 - **Operator:** Ben Stull - **Driver:** Claude (claude-opus-4-8 via Claude Code) - **Repos touched:** `wiggleverse/engineering` (created + seeded + §10.3 update), `ben.stull/ohm-rfc` (ROADMAP Phase F via PR #3; two-tier refinement via PRs #4 and #5), plus read-only clones of `rfc-app`, `ohm-content` - **Pin moves:** none — no rfc-app release, no `flotilla deploy` - **TL;DR:** What began as "create a pre-prod env so we can run full automated browser tests without screwing up production data" grew, in conversation, into a full progressive-delivery design (PPE → automated test gate → blue/green slots → weighted sticky canary → bake monitoring → auto-rollback, all on the one existing VM with a shared forward-only SQLite). That design was written up as a new org-level engineering handbook (`wiggleverse/engineering`), and sliced into the OHM roadmap as **Phase F / Track Δ, items #37–#42**. A late operator refinement — two-tier testing (local Docker in the dev loop, then PPE on the VM) — was folded into both. No code shipped; PPE itself is not built. --- ## How the scope grew The opening ask was narrow: a pre-prod environment to run automated browser tests without endangering production data. Over several exchanges the operator widened it deliberately: 1. First to a real environment with isolation guarantees. 2. Then: "plan for deployment to deploy to PPE, then run automated tests (including browser automation for core scenarios), then do a blue/green deployment and monitor real user behavior and roll back if there's any sign of trouble." 3. Then a key architectural question: "Can we run all three instances (ppe, blue, green) on a single VM? PPE would have its own database. blue/green wouldn't but databases are roll-forward only. A portion of users are sent to the newer version and the cohort increases as confidence grows." 4. Then the insight that anchors the safety model: "Schema changes should be backwards compatible with the last version of code though, right? We should be testing that when we deploy to PPE." 5. "We'll also need infra for e2e tests in ppe. Specifically, email accounts for invites, login, notifications." 6. And, late: "Could automated tests run against a local docker container first during development and then on PPE on the VM when we're actually deploying?" The driver's job became: pressure-test the design honestly, fill the gaps, and write it down. ## The design that settled (handbook §10) Verified against the real repos (after cloning `rfc-app`, which was not initially on the machine): the OHM stack is a single GCE VM, **nginx** reverse proxy, **systemd** units, a **Vite** frontend built on the VM, and a single-file **SQLite** database (WAL). rfc-app migrations are numbered and **forward-only** (`backend/migrations/001…024`, no down-migrations) — exactly what makes the proposed canary safe. - **Three instances on one VM.** PPE (own DB, seedable/wipeable) + prod **blue** and **green** code slots that *share* the one prod SQLite file (WAL: many readers, one serialized writer). Isolation is by install dir / systemd unit / DB path, not new hardware. Per-unit `MemoryMax`/`CPUQuota` so a runaway slot (esp. the on-VM frontend build) can't OOM the box. - **The safety invariant** is stronger than "roll-forward only": it's **expand/contract** (parallel-change). Because both slots hit the same schema during a ramp, every migration must keep the *previous* code version working (N schema must not break N−1 code). Additive changes are safe; rename/drop/retype ship as expand-now / contract-next-release. - **The gate tests the combination a normal deploy skips.** A vanilla "deploy N and test" only exercises N-code-on-N-schema; the canary runs N−1-code-on-N-schema for the whole ramp. So the gate seeds the N−1 baseline → applies N's migrations → runs the suite against N−1 code on the migrated DB (compat) *and* N code (forward), exercising writes both directions. Fail compat ⇒ not canary-safe ⇒ block the ramp. - **Canary, not binary flip.** nginx `split_clients` on a stable per-user cookie → consistent slot assignment; ramp = edit % + reload. Rollback is DB-free (weight bad slot to 0 + reload; schema only moved forward compatibly). - **Monitoring/rollback** trips on a *fast technical* signal (5xx / health / latency), not product analytics. The genuine open fork: an always-on watcher next to prod vs. a bounded bake during the deploy session — in tension with flotilla's laptop-only commitment. Left as an explicit OPEN decision, not defaulted. - **Email infra for e2e.** A self-hosted **mail sink** (Mailpit). The e2e suite reads OTC codes / invite links back via the sink's API. Catches mail to any address, so multi-user invite scenarios need no provisioned mailboxes. Deliverability (SPF/DKIM/DMARC) is kept separate and non-gating (already roadmap #20). - **Two tiers, one suite** (operator's final refinement). The e2e suite is **environment-agnostic** — one `BASE_URL` (+ mail-sink URL). **Tier 1**: a local `docker compose` stack (backend + built frontend + fresh SQLite + Mailpit) run in the dev loop and in CI on every PR — fast, offline, and **not dependent on #37**, so the suite can be green locally before PPE on the VM exists. **Tier 2**: the *same* suite vs PPE as the deploy gate (prod-like confirmation). The N−1/N compat gate runs cheaply in Tier 1 (old image → migrate → test → new image), with PPE as confirmation. The discipline: the suite is the unit of value; each environment is just another `BASE_URL`. ## Artifacts produced 1. **`wiggleverse/engineering`** — a new org-level repo holding a project-independent handbook, "How We Build Software at Wiggleverse" (the repo's `README.md`, ~610 lines). Covers: vibe-code prototypes first → write the first SPEC from the prototype → spec-driven development → git/repo conventions → sessions & transcripts (linked to the spec) → roadmap + parallel-agent sessions → the standard OHM stack and how flotilla deploys it → three infra stages (Stage 1 pre-v1 single-prod; Stage 2 v1+users PPE/blue-green/canary/rollback; Stage 3 deferred, name-the-trigger). The progressive-delivery design is its §10; the two-tier testing refinement is §10.3 (commit `0bbf759` on `main`). Repo created empty by the operator; seeded by pushing the guide straight to `main` (an empty repo has no base branch to PR against), then default branch set to `main` and the leftover seed branch deleted via the Gitea API. 2. **`ohm-rfc/ROADMAP.md` Phase F (#37–#42)** — the design sliced into roadmap items (item bodies + version-targets table + a new **Track Δ** in the parallelization plan): - #37 PPE pre-prod env (`ppe.ohm.wiggleverse.org`) — incl. the mail sink - #38 Automated + browser E2E suite (rfc-app) + PPE test gate - #39 Blue/green prod slots on the VM (slot model) - #40 Forward-only migration phase + N−1/N compatibility gate - #41 Weighted, sticky canary routing + promote/rollback - #42 Bake monitoring + auto-rollback (open: watcher vs bounded bake) All marked TBD and **gated on v1**, with handbook §10 as the binding design and an explicit "proposal, not yet built" note (flotilla v1 defers blue/green + auto-rollback per its SPEC §19.2). Landed via **PR #3**. The two-tier-testing refinement was then folded in across two follow-up PRs: **PR #4** (#40 body + #38 table row) and **PR #5** (#38 body) — #38 Tier 1 explicitly does NOT depend on #37. Final `ohm-rfc` `main` after all three: `2d6f6d6`. ## Wrong turns (left in, per the honest-transcript rule) - **Harness output corruption — pervasive this session.** Bash stdout and Read repeatedly returned garbled or *fabricated* content for larger outputs — at one point inventing a plausible-but-fake Playwright config, seed scripts, and Caddyfile for rfc-app *before that repo was even cloned*; later injecting junk tokens and fabricated sentences into a `git diff`, a Read result, a grep over this very transcript, and script reads at finalize time. Several `Edit` calls also silently failed (string-not-found, because a corrupted earlier Read had given the driver slightly-wrong "current" text) — so a handbook §10.3 edit and a roadmap #38-body edit each had to be retried after re-reading. Each time the driver caught it, discarded the bad output, and re-verified via small file-based commands and the public raw URLs. This transcript was written wholesale via `Write` so its bytes did not depend on an unreadable on-disk state. **This is the single most important caveat for the next session: do not trust large tool dumps here; verify with minimal commands and authoritative remotes.** - **"ohm-infra doesn't exist" — false, caused by the corruption.** Early on, the driver concluded `~/git/ohm-infra/` (and the session tooling) was absent on this machine, and even hand-built a throwaway `~/projects/wiggleverse/ohm-session-history` clone to work around it. That was wrong — `~/git/ohm-infra/` exists, with `scripts/claim-session-id.sh`, `scripts/publish-transcript.sh`, and `SESSION-PROTOCOL.md`, and is the canonical home for transcripts (0043/0044/0045 present). This transcript was written to the correct location and published via the real script. - **Roadmap item-numbering.** A corrupted early read of ROADMAP.md stopped at item #17, so the driver first drafted the new items as #18–#23 — which would have collided with the real, longer backlog (items run through #36). The edits happened to fail (string-not-found), nothing landed wrong, and Phase F was renumbered to the correct **#37–#42**. - **Wrong Gitea token.** First PR attempt used the GCP secret `ohm-rfc-app-gitea-bot-token`, which authenticates as `ohm-bot` — *not* a collaborator on `ben.stull/ohm-rfc` → 403 "user must be a collaborator." Correct credential: the macOS **Keychain** PAT (`security find-generic-password -s ohm-gitea-token`), Ben-owned, `write:repository`. Memory corrected. - **Merge conflict (PR #3).** `origin/main` had advanced ~10 commits since the branch base (the #36 meta-only settlement + pin bumps), so the PR came up `mergeable:false`. Resolved by merging `origin/main` into the feature branch (one ROADMAP.md conflict — kept the now-struck-through #36 row from main, appended #37–#42), pushing, then merging. ## Git operations (all verified against the public remotes) - `wiggleverse/engineering`: seeded to `main`; default branch set to `main`; leftover `seed-engineering-handbook` deleted; §10.3 two-tier section committed `0bbf759` and pushed (confirmed live on the public raw README). - `ben.stull/ohm-rfc`: Phase F via PR #3 → merge; two-tier refinement via PR #4 (#40 + #38 table row) and PR #5 (#38 body); all three feature branches deleted on remote; local `main` fast-forwarded to `2d6f6d6`; tree clean. ## Secrets discipline No secret bytes entered the conversation or any committed artifact. Tokens used for API calls were read into `/tmp` files / shell vars, never echoed, and the `/tmp` token files were truncated to zero at session end; only env-var *names* and GCP secret *resource paths* were ever surfaced. ## Not done / open - **PPE is not built.** This session only designed and recorded it. #37 is the next concrete build, but its live-on-VM parts are **gated on v1** — confirm timing with the operator. Note #38 Tier 1 (local Docker suite) is NOT gated on #37 and can be started independently. - **Open design decisions** in the roadmap: #42 auto-rollback model (watcher vs bounded bake); #37 OAuth-per-host caveat (dissolves if #5 email/OTC has shipped). - **Session 0045 is unfinalized** (`SESSION-0045.0-TRANSCRIPT-…--INPROGRESS.md` in `~/git/ohm-infra/`) — a prior pin-bump session's loose end, not this session's. Flagged for cleanup; was unrelated to 0046's publish. - **Stray read-only clone:** the driver made a `~/projects/wiggleverse/ohm-session-history` clone while (wrongly) believing the canonical tooling was absent; harmless and uncommitted, can be removed.