Session 0046.0 — Transcript — PPE / progressive-delivery design, engineering handbook, ROADMAP Phase F

Start: 2026-05-29T21-30 (approximate — the harness could not recover the exact session-open time; the prior session's placeholder 0045 is stamped 21-26, and this session's first git action was the engineering-handbook seed commit at 2026-05-29 22:44 -0400)
End: 2026-05-30T06-19
Operator: Ben Stull
Driver: Claude (claude-opus-4-8 via Claude Code)
Repos touched: wiggleverse/engineering (created + seeded + §10.3 update), ben.stull/ohm-rfc (ROADMAP Phase F via PR #3; two-tier refinement via PRs #4 and #5), plus read-only clones of rfc-app, ohm-content
Pin moves: none — no rfc-app release, no flotilla deploy
TL;DR: What began as "create a pre-prod env so we can run full automated browser tests without screwing up production data" grew, in conversation, into a full progressive-delivery design (PPE → automated test gate → blue/green slots → weighted sticky canary → bake monitoring → auto-rollback, all on the one existing VM with a shared forward-only SQLite). That design was written up as a new org-level engineering handbook (wiggleverse/engineering), and sliced into the OHM roadmap as Phase F / Track Δ, items #37–#42. A late operator refinement — two-tier testing (local Docker in the dev loop, then PPE on the VM) — was folded into both. No code shipped; PPE itself is not built.

How the scope grew

The opening ask was narrow: a pre-prod environment to run automated browser tests without endangering production data. Over several exchanges the operator widened it deliberately:

First to a real environment with isolation guarantees.
Then: "plan for deployment to deploy to PPE, then run automated tests (including browser automation for core scenarios), then do a blue/green deployment and monitor real user behavior and roll back if there's any sign of trouble."
Then a key architectural question: "Can we run all three instances (ppe, blue, green) on a single VM? PPE would have its own database. blue/green wouldn't but databases are roll-forward only. A portion of users are sent to the newer version and the cohort increases as confidence grows."
Then the insight that anchors the safety model: "Schema changes should be backwards compatible with the last version of code though, right? We should be testing that when we deploy to PPE."
"We'll also need infra for e2e tests in ppe. Specifically, email accounts for invites, login, notifications."
And, late: "Could automated tests run against a local docker container first during development and then on PPE on the VM when we're actually deploying?"

The driver's job became: pressure-test the design honestly, fill the gaps, and write it down.

The design that settled (handbook §10)

Verified against the real repos (after cloning rfc-app, which was not initially on the machine): the OHM stack is a single GCE VM, nginx reverse proxy, systemd units, a Vite frontend built on the VM, and a single-file SQLite database (WAL). rfc-app migrations are numbered and forward-only (backend/migrations/001…024, no down-migrations) — exactly what makes the proposed canary safe.

Three instances on one VM. PPE (own DB, seedable/wipeable) + prod blue and green code slots that share the one prod SQLite file (WAL: many readers, one serialized writer). Isolation is by install dir / systemd unit / DB path, not new hardware. Per-unit MemoryMax/CPUQuota so a runaway slot (esp. the on-VM frontend build) can't OOM the box.
The safety invariant is stronger than "roll-forward only": it's expand/contract (parallel-change). Because both slots hit the same schema during a ramp, every migration must keep the previous code version working (N schema must not break N−1 code). Additive changes are safe; rename/drop/retype ship as expand-now / contract-next-release.
The gate tests the combination a normal deploy skips. A vanilla "deploy N and test" only exercises N-code-on-N-schema; the canary runs N−1-code-on-N-schema for the whole ramp. So the gate seeds the N−1 baseline → applies N's migrations → runs the suite against N−1 code on the migrated DB (compat) and N code (forward), exercising writes both directions. Fail compat ⇒ not canary-safe ⇒ block the ramp.
Canary, not binary flip. nginx split_clients on a stable per-user cookie → consistent slot assignment; ramp = edit % + reload. Rollback is DB-free (weight bad slot to 0 + reload; schema only moved forward compatibly).
Monitoring/rollback trips on a fast technical signal (5xx / health / latency), not product analytics. The genuine open fork: an always-on watcher next to prod vs. a bounded bake during the deploy session — in tension with flotilla's laptop-only commitment. Left as an explicit OPEN decision, not defaulted.
Email infra for e2e. A self-hosted mail sink (Mailpit). The e2e suite reads OTC codes / invite links back via the sink's API. Catches mail to any address, so multi-user invite scenarios need no provisioned mailboxes. Deliverability (SPF/DKIM/DMARC) is kept separate and non-gating (already roadmap #20).
Two tiers, one suite (operator's final refinement). The e2e suite is environment-agnostic — one BASE_URL (+ mail-sink URL). Tier 1: a local docker compose stack (backend + built frontend + fresh SQLite + Mailpit) run in the dev loop and in CI on every PR — fast, offline, and not dependent on #37, so the suite can be green locally before PPE on the VM exists. Tier 2: the same suite vs PPE as the deploy gate (prod-like confirmation). The N−1/N compat gate runs cheaply in Tier 1 (old image → migrate → test → new image), with PPE as confirmation. The discipline: the suite is the unit of value; each environment is just another BASE_URL.

Artifacts produced

wiggleverse/engineering — a new org-level repo holding a project-independent handbook, "How We Build Software at Wiggleverse" (the repo's README.md, ~610 lines). Covers: vibe-code prototypes first → write the first SPEC from the prototype → spec-driven development → git/repo conventions → sessions & transcripts (linked to the spec) → roadmap + parallel-agent sessions → the standard OHM stack and how flotilla deploys it → three infra stages (Stage 1 pre-v1 single-prod; Stage 2 v1+users PPE/blue-green/canary/rollback; Stage 3 deferred, name-the-trigger). The progressive-delivery design is its §10; the two-tier testing refinement is §10.3 (commit 0bbf759 on main). Repo created empty by the operator; seeded by pushing the guide straight to main (an empty repo has no base branch to PR against), then default branch set to main and the leftover seed branch deleted via the Gitea API.
ohm-rfc/ROADMAP.md Phase F (#37–#42) — the design sliced into roadmap items (item bodies + version-targets table + a new Track Δ in the parallelization plan):
- #37 PPE pre-prod env (ppe.ohm.wiggleverse.org) — incl. the mail sink
- #38 Automated + browser E2E suite (rfc-app) + PPE test gate
- #39 Blue/green prod slots on the VM (slot model)
- #40 Forward-only migration phase + N−1/N compatibility gate
- #41 Weighted, sticky canary routing + promote/rollback
- #42 Bake monitoring + auto-rollback (open: watcher vs bounded bake) All marked TBD and gated on v1, with handbook §10 as the binding design and an explicit "proposal, not yet built" note (flotilla v1 defers blue/green + auto-rollback per its SPEC §19.2). Landed via PR #3. The two-tier-testing refinement was then folded in across two follow-up PRs: PR #4 (#40 body + #38 table row) and PR #5 (#38 body) — #38 Tier 1 explicitly does NOT depend on #37. Final ohm-rfc main after all three: 2d6f6d6.

Wrong turns (left in, per the honest-transcript rule)

Harness output corruption — pervasive this session. Bash stdout and Read repeatedly returned garbled or fabricated content for larger outputs — at one point inventing a plausible-but-fake Playwright config, seed scripts, and Caddyfile for rfc-app before that repo was even cloned; later injecting junk tokens and fabricated sentences into a git diff, a Read result, a grep over this very transcript, and script reads at finalize time. Several Edit calls also silently failed (string-not-found, because a corrupted earlier Read had given the driver slightly-wrong "current" text) — so a handbook §10.3 edit and a roadmap #38-body edit each had to be retried after re-reading. Each time the driver caught it, discarded the bad output, and re-verified via small file-based commands and the public raw URLs. This transcript was written wholesale via Write so its bytes did not depend on an unreadable on-disk state. This is the single most important caveat for the next session: do not trust large tool dumps here; verify with minimal commands and authoritative remotes.
"ohm-infra doesn't exist" — false, caused by the corruption. Early on, the driver concluded ~/git/ohm-infra/ (and the session tooling) was absent on this machine, and even hand-built a throwaway ~/projects/wiggleverse/ohm-session-history clone to work around it. That was wrong — ~/git/ohm-infra/ exists, with scripts/claim-session-id.sh, scripts/publish-transcript.sh, and SESSION-PROTOCOL.md, and is the canonical home for transcripts (0043/0044/0045 present). This transcript was written to the correct location and published via the real script.
Roadmap item-numbering. A corrupted early read of ROADMAP.md stopped at item #17, so the driver first drafted the new items as #18–#23 — which would have collided with the real, longer backlog (items run through #36). The edits happened to fail (string-not-found), nothing landed wrong, and Phase F was renumbered to the correct #37–#42.
Wrong Gitea token. First PR attempt used the GCP secret ohm-rfc-app-gitea-bot-token, which authenticates as ohm-bot — not a collaborator on ben.stull/ohm-rfc → 403 "user must be a collaborator." Correct credential: the macOS Keychain PAT (security find-generic-password -s ohm-gitea-token), Ben-owned, write:repository. Memory corrected.
Merge conflict (PR #3). origin/main had advanced ~10 commits since the branch base (the #36 meta-only settlement + pin bumps), so the PR came up mergeable:false. Resolved by merging origin/main into the feature branch (one ROADMAP.md conflict — kept the now-struck-through #36 row from main, appended #37–#42), pushing, then merging.

Git operations (all verified against the public remotes)

wiggleverse/engineering: seeded to main; default branch set to main; leftover seed-engineering-handbook deleted; §10.3 two-tier section committed 0bbf759 and pushed (confirmed live on the public raw README).
ben.stull/ohm-rfc: Phase F via PR #3 → merge; two-tier refinement via PR #4 (#40 + #38 table row) and PR #5 (#38 body); all three feature branches deleted on remote; local main fast-forwarded to 2d6f6d6; tree clean.

Secrets discipline

No secret bytes entered the conversation or any committed artifact. Tokens used for API calls were read into /tmp files / shell vars, never echoed, and the /tmp token files were truncated to zero at session end; only env-var names and GCP secret resource paths were ever surfaced.

Not done / open

PPE is not built. This session only designed and recorded it. #37 is the next concrete build, but its live-on-VM parts are gated on v1 — confirm timing with the operator. Note #38 Tier 1 (local Docker suite) is NOT gated on #37 and can be started independently.
Open design decisions in the roadmap: #42 auto-rollback model (watcher vs bounded bake); #37 OAuth-per-host caveat (dissolves if #5 email/OTC has shipped).
Session 0045 is unfinalized (SESSION-0045.0-TRANSCRIPT-…--INPROGRESS.md in ~/git/ohm-infra/) — a prior pin-bump session's loose end, not this session's. Flagged for cleanup; was unrelated to 0046's publish.
Stray read-only clone: the driver made a ~/projects/wiggleverse/ohm-session-history clone while (wrongly) believing the canonical tooling was absent; harmless and uncommitted, can be removed.

12 KiB Raw Blame History Unescape Escape