wiggleverse/session-history
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

add 0030/SESSION-0030.0-TRANSCRIPT-2026-05-28T16-12--2026-05-28T18-41.md + replace placeholder/variant SESSION-0030.0-TRANSCRIPT-2026-05-28T16-12--2026-05-28T17-39.md

Browse Source
This commit is contained in:
Ben Stull
2026-05-28 18:42:53 -07:00
parent e4ec14379c
commit 1de67da107
2 changed files with 230 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
0030/SESSION-0030.0-TRANSCRIPT-2026-05-28T16-12--2026-05-28T17-39.md
-211
View File
@@ -1,211 +0,0 @@
# Session 0030.0 — Transcript
> Date: 2026-05-28
> Start: 2026-05-28T16-12 (PST implied)
> End: 2026-05-28T17-39 (PST implied)
> Subject: Security-audit-0026 remediation — flotilla-owned fixes + live GCP/VM hardening
> Status: **FINALIZED.**
> Sanitization note: this transcript is published to the public
> `wiggleverse/ohm-session-history`. Because several audit findings were
> still **live and unpatched** on production at session end (they belong to
> a parallel, not-yet-deployed branch — see "Coordination" below), this
> record deliberately references findings by **ID and general category
> only** and contains **no exploit sketches or step-by-step reproduction**.
> The full-detail audit (`SECURITY-AUDIT-0026-2026-05-28.md`) lives locally
> in `~/git/ohm-infra`, which is not a git repository, and was never
> published. No secret bytes appear here (§3 invariant 1).
---
## Launch prompt
> full remediation of security report at ~/git/ohm-infra
The "security report" is `SECURITY-AUDIT-0026-2026-05-28.md` (produced by
the parallel audit session 0026): 1 Critical, 1 High, 5 Medium, 9 Low,
6 Info findings spanning three surfaces — the **rfc-app** codebase, the
**flotilla** CLI (this repo), and the **live GCP/VM** deployment.
---
## What this session settled
Session 0030 took the **flotilla-owned findings** and the **live GCP/VM
findings that live in no repo**, applied + verified them, and merged +
deployed the flotilla changes. The rfc-app/nginx/systemd findings were
left to a parallel in-flight session (see Coordination). Net for OHM:
the VM is materially hardened and the operator CLI has three defense-in-
depth improvements; the application-layer findings remain pending that
other session's release.
## The "did we leak the audit?" scare (resolved: NO)
Mid-session the operator flagged that a transcript containing the audit
might have been pushed to the public repo before an internet drop. I
froze remediation and verified the **entire git history** of
`wiggleverse/ohm-session-history` (fresh clone, all branches,
`git log -S` for audit-unique strings):
- Audit session 0026 published **only its 454-byte placeholder** — its
full transcript was never pushed (the publish push is atomic; it failed
on the internet drop, leaving only the start-gesture placeholder).
- **Zero** audit-unique strings appear in any commit, ever.
- The audit file itself lives only in local `~/git/ohm-infra` (not a git
repo) — never committed anywhere.
Verdict: nothing leaked. The real containment for the audit's value (an
exploit roadmap) is shipping the fixes, which is what the session then
continued doing.
## Coordination — the rfc-app remediation is owned elsewhere
On inspecting `/Users/benstull/git/rfc-app` I found branch
`feature/v0.25.0-security-hardening` already checked out with a large
**uncommitted** remediation (a parallel live session, 0027 or 0029).
It covers the rfc-app application findings, the nginx server block, and
the systemd unit. Editing rfc-app would have clobbered another session's
live work, so — with the operator — Session 0030 explicitly scoped to:
- **flotilla repo** fixes (no overlap), and
- **live GCP/VM ops** that exist in no repo.
The operator chose this division (and chose to perform the live ops this
session, and to defer SSH-scoping to a proper IAP rollout).
## Work performed
### Flotilla repo (PR #1 — merged)
Branch `session-0030/flotilla-security-hardening` → **PR #1**, merged to
`main` (merge commit `7b32a46`). 161 tests pass.
- **L1** — the live secret redactor is threaded into `_PhaseRunner` and
now scrubs **both** the success and failure phase-`detail` paths before
they are persisted to `deploys.phases` / surfaced to the operator
(previously only the separate `error_summary` was scrubbed). Test
extended to assert no secret survives in any persisted phase detail.
(This fix was already present in the working tree at session start;
Session 0030 verified, tested, and landed it.)
- **I6** — `redact_live` now matches several decodings of the same secret
bytes (utf-8/replace, surrogateescape, latin-1) so a non-ASCII secret
that surfaced via a different decode path than the needle still
redacts. Match only ever redacts more, never less. Residual byte-
boundary limitation documented honestly (moot for ASCII/UTF-8 secrets).
- **L9** — `ssh.py` no longer leans on `--quiet`'s blanket first-connect
host-key accept; it adds `StrictHostKeyChecking=accept-new`, which still
connects to genuinely-new hosts non-interactively but **refuses a
changed key**. Full first-connect TOFU removal remains the §19.2 IAP
item.
- **I5** — `overlay set` warns (does not block) when a key name carries an
unambiguous secret word, nudging toward `secret bind`/`secret set`.
Deliberately does not match generic `*_KEY`/`*_API_KEY`, so public site
keys don't train the operator to ignore the warning.
After merge, a clean idempotent redeploy of the current pin (v0.26.0)
exercised the hardened flotilla code end-to-end: **`deploys.id=35`**, all
9 phases ok, health verified in 5.0s.
### Live GCP/VM ops (project `wiggleverse-ohm`, VM `ohm-rfc-app`)
- **M3** — the SQLite data directory was world-traversable and the DB
(plus its WAL/SHM sidecars) world-readable. Tightened the **directory to
`750`** (the robust containment — closes world access regardless of
per-file mode regression) and the files to **`640`**. Verified to
survive both a full VM reboot and a deploy-triggered service restart
(sqlite preserves the main DB's mode on its sidecars). Also discovered
and **secured** (but did not delete) a set of orphaned `ohm-app.db\r*`
files — stale copies left by the historical CRLF path bug; flagged to
the operator for deletion after confirmation.
- **L5** — deleted the dead `default-allow-rdp` firewall rule (3389 is
unused surface on Linux). SSH-from-anywhere left in place by operator
decision; the right fix is an IAP rollout (§19.2), not fragile IP
scoping.
- **L6** — created a dedicated least-privilege runtime service account
(`ohm-rfc-app-vm@…`) with logging+monitoring writer only (the app reads
its config from `.env` at deploy time, so **no** Secret Manager access
is needed on the VM), and swapped the VM off the default compute SA.
- **L7** — enabled Shielded-VM **Secure Boot** (boot disk confirmed
UEFI-compatible first; vTPM + integrity monitoring were already on).
L6 + L7 were applied in a single VM stop/start window (~15s of downtime),
after confirming no deploy was in flight. All four verified intact after
the later deploy.
### Config verification (audit action item 5)
Read the live `.env` **variable names** only (no values):
- `TURNSTILE_REQUIRED="true"` — confirmed (mitigates the OTC abuse Low).
- `GITEA_WEBHOOK_SECRET` — present (Gitea webhook authenticated).
- **`WEBHOOK_EMAIL_BOUNCE_SECRET` — absent.** This confirms M5 as a live
config gap (the bounce webhook is currently unauthenticated). Handed to
the operator with an operator-run secret gesture (never request/echo
secret bytes in-conversation — §3 conversation-layer corollary). The
operator will bind it and redeploy as a follow-up.
## Friction / dead-ends (kept honest)
- The first `chmod 640 …/data/*` failed: having just set the directory to
`750`, the non-root SSH login user could no longer list it to expand the
glob. Re-ran the file chmod under `sudo sh -c` so root expanded it.
- After merging PR #1, `git checkout main && git pull` did **not**
fast-forward (local `main` had no upstream tracking), so the working
tree briefly reverted to pre-fix code. Recovered with
`git fetch && git merge --ff-only origin/main` and set the upstream.
No work lost (the fixes were safe in the merge commit).
## Findings status at session end
| Owner | Findings | Status |
| --- | --- | --- |
| **0030 (flotilla)** | L1, I6, L9, I5 | **Done** — merged (PR #1) + deployed |
| **0030 (live VM/GCP)** | M3, L5(RDP), L6, L7 | **Done** — applied + verified |
| **Operator follow-up** | M5 (config bind), `*.db\r` cleanup, systemd `UMask=0077` | Flagged with gestures |
| **Deferred** | L5 (SSH 0.0.0.0/0 → IAP §19.2) | By operator decision |
| **Parallel v0.25.0 branch** | C1, H1, M1, M2, M4, L2, L3, L4, L8, I1, I3, I4 | **Pending** — uncommitted, **not yet deployed** (still live on prod) |
## Cross-references
- flotilla PR #1 (`wiggleverse/ohm-rfc-app-flotilla`), merge `7b32a46`.
- OHM deploy `deploys.id=35`, v0.26.0 (idempotent redeploy exercising the
hardened flotilla).
- GCP: VM `ohm-rfc-app` / `us-central1-a` / `wiggleverse-ohm`; new SA
`ohm-rfc-app-vm@wiggleverse-ohm.iam.gserviceaccount.com`; deleted
firewall rule `default-allow-rdp`.
- Source audit: `~/git/ohm-infra/SECURITY-AUDIT-0026-2026-05-28.md`
(local only — never published).
---
## Operating instructions for the next session
The application-layer security work is the open thread. Concretely:
1. **Ship v0.25.0.** The `feature/v0.25.0-security-hardening` branch in
rfc-app holds the application + nginx + systemd remediation (C1 XSS
sanitizer, the auth-verify rate-limit/lockout, the device-trust lookup
fix, security headers, cookie hardening, the bounce-webhook
`_required` guard, etc.). It is **uncommitted and undeployed**, which
means the audit's **Critical (C1)** and other findings are **still live
on production**. This is the highest priority: confirm/finish the
branch, commit, release as v0.25.0, bump `ohm-rfc/.rfc-app-version`,
and deploy via flotilla. Coordinate with whichever session owns that
branch (was 0027/0029) before touching it.
2. **M5 config** — the operator intends to bind `WEBHOOK_EMAIL_BOUNCE_SECRET`
(Secret Manager + `flotilla secret bind` + provider-side config) and
redeploy. The code-side `_required` guard rides along in v0.25.0.
3. **systemd `UMask=0077`** — add to rfc-app's `deploy/systemd/rfc-app.service`
(belt-and-suspenders on top of the `750` data dir).
4. **Orphaned `ohm-app.db\r*` files** on the VM — secured to `640`; delete
once confirmed orphaned.
5. **L5 SSH / IAP (§19.2)** — the deferred item: IAP-only SSH (firewall +
tunnel role + flip flotilla's deploy path to `--tunnel-through-iap`),
which also retires the L9 first-connect TOFU residue.
Suggested next-session launch prompt:
> Ship rfc-app v0.25.0-security-hardening (finish/commit the existing
> branch, release, bump the OHM pin, deploy) — this lands the still-live
> audit-0026 application findings incl. the Critical XSS. Coordinate with
> the session that owns the branch first. Then help the operator bind
> WEBHOOK_EMAIL_BOUNCE_SECRET (M5) and add systemd UMask=0077.
0030/SESSION-0030.0-TRANSCRIPT-2026-05-28T16-12--2026-05-28T18-41.md
+230
View File
@@ -0,0 +1,230 @@
# Session 0030.0 — Transcript
> Date: 2026-05-28
> Start: 2026-05-28T16-12 (PST implied)
> End: 2026-05-28T18-41 (PST implied)
> Subject: Security-audit-0026 FULL remediation — flotilla fixes, live VM
> hardening, and the rfc-app v0.27.0 security release (shipped + deployed)
> Status: **FINALIZED.**
> Sanitization note: this transcript is published to the public
> `wiggleverse/ohm-session-history`. It references audit findings by **ID
> and general category only** — no exploit sketches or step-by-step
> reproduction. (At the first finalize, several findings were still live
> and unpatched, which made this mandatory; by this finalize the Critical
> and the rest of the application findings are remediated + deployed, but
> the convention holds — L5 SSH remains open, and exploit recipes never
> belong in a public record.) The full-detail audit
> (`SECURITY-AUDIT-0026-2026-05-28.md`) lives only in local `~/git/ohm-infra`
> (not a git repo) and was never published. No secret bytes appear here
> (§3 invariant 1).
---
## Launch prompt
> full remediation of security report at ~/git/ohm-infra
Plus two follow-on operator directives mid-session: **"ship v0.25.0"**
(the security branch) and a request to bind `WEBHOOK_EMAIL_BOUNCE_SECRET`.
The "security report" is `SECURITY-AUDIT-0026-2026-05-28.md` (from the
parallel audit session 0026): 1 Critical, 1 High, 5 Medium, 9 Low, 6 Info
across the rfc-app codebase, the flotilla CLI, and the live GCP/VM.
---
## Outcome in one line
Every actionable audit-0026 finding is remediated and **live on
production**, via three vehicles: flotilla **PR #1** (merged + deployed),
direct **live GCP/VM ops**, and the rfc-app **v0.27.0** security release
(`deploys.id=37`, verified live). Only the SSH-from-anywhere Low (deferred
to a proper IAP rollout) and a few Info items remain.
## The "did we leak the audit?" scare (resolved: NO)
Early on the operator worried a transcript containing the audit had been
pushed to the public repo before an internet drop. Froze work and verified
the **entire git history** of `wiggleverse/ohm-session-history` (fresh
clone, all branches, `git log -S` for audit-unique strings): the audit
session 0026 had published **only its 454-byte placeholder**; **zero**
audit-unique strings in any commit ever; the audit file itself lives only
in local `~/git/ohm-infra` (not a git repo). Nothing leaked. The atomic
publish push had failed on the internet drop, leaving only the
start-gesture placeholder.
## Coordination
The rfc-app application findings were being remediated on an **uncommitted**
`feature/v0.25.0-security-hardening` branch by a parallel session. With the
operator, Session 0030 first scoped to the non-overlapping work (flotilla +
live VM). Later the operator directed this session to **take over and ship**
that branch — which it did (as v0.27.0; see below).
---
## Work performed
### 1. Flotilla repo (PR #1 — merged + deployed)
Branch `session-0030/flotilla-security-hardening` → **PR #1** → merged
(`7b32a46`). 161 tests pass.
- **L1** — secret redactor threaded into `_PhaseRunner`; both success and
failure phase-`detail` paths are scrubbed before persistence/printing.
- **I6** — `redact_live` matches multiple byte-decodings (utf-8/replace,
surrogateescape, latin-1) so a non-ASCII secret can't slip the needle.
- **L9** — ssh adds `StrictHostKeyChecking=accept-new` (refuses a *changed*
host key instead of blanket TOFU-accept).
- **I5** — `overlay set` warns on unambiguous secret-word key names (not
generic `*_KEY`, to avoid nagging on public site keys).
### 2. Live GCP/VM ops (project `wiggleverse-ohm`, VM `ohm-rfc-app`)
- **M3** — data dir → `750`, DB + WAL/SHM → `640` (survives reboot +
restart). Also secured (did not delete) orphaned `ohm-app.db\r*` files
left by the historical CRLF path bug.
- **L5** — deleted the dead `default-allow-rdp` firewall rule. SSH-from-
anywhere deferred to an IAP rollout (operator decision).
- **L6** — created dedicated least-priv SA `ohm-rfc-app-vm@…` (logging +
monitoring writer only; no Secret Manager — the app reads `.env`), swapped
the VM off the default compute SA.
- **L7** — enabled Shielded-VM Secure Boot (UEFI-verified first; vTPM +
integrity monitoring already on).
L6 + L7 applied in one stop/start window (~15s downtime).
### 3. rfc-app v0.27.0 — the security release (shipped + deployed)
The "v0.25.0" branch was based on v0.24.0, but main/live was already
**v0.26.0** (#28). Shipping it as v0.25.0 would have downgraded prod, so it
was **rebased onto main and reversioned to v0.27.0**. Then: committed,
version-bumped (`VERSION` + `package.json` + a CHANGELOG entry with upgrade
steps), tested, PR'd (**PR #1 on `ben.stull/rfc-app`**), merged (`fe044ed`),
tagged **v0.27.0**, pin bumped in `ohm-rfc/.rfc-app-version` (0.26.0→0.27.0),
deployed (`deploys.id=37`), and verified live.
Findings landed: **C1** (DOMPurify sanitize chokepoint for every
markdown→HTML sink — the Critical), **H1** (per-IP ratelimit + per-account
OTC lockout, migration 023 auto-applied), **M1** (indexed single-row
device-trust lookup), **M2** (nginx security headers — see below), **M4**
(session cookie Secure-by-default), **M5** (bounce webhook fails closed +
secret bound), **L2/L3** (throttles), **L4** (systemd sandbox), **L8/I1**
(server_tokens off, legacy TLS removed).
**The parallel branch was incomplete** — its changes broke 60 backend
tests, which this session diagnosed and fixed:
- M4 set the session cookie `Secure`, but `TestClient` talks plain http,
so the cookie was never returned and every authenticated flow failed →
the test env now sets `SESSION_COOKIE_SECURE=false` (the documented
dev opt-out).
- M5 fail-closed returned 503 in tests → test env sets
`RFC_APP_INSECURE_BOUNCE_WEBHOOK=1` (mirrors the existing
`RFC_APP_INSECURE_WEBHOOKS` pattern).
- The new per-IP rate limiter is a process-global singleton with no
per-test reset, so later tests tripped the budget → added
`ratelimit._reset_all_for_tests()` + an autouse `conftest.py` fixture.
Result: **363 backend tests green; frontend builds clean (0 vulns).**
### 4. M2 + L4 applied out-of-band (the deploy gesture does not)
The flotilla deploy does **not** install nginx/systemd files. The repo's
`deploy/` templates also carried **stale pre-rename paths** (`User=rfc-app`,
`/opt/rfc-app/…`) that would have taken the site down if copied verbatim.
So:
- **M2/L8/I1** — injected the 5 security headers + `server_tokens off` into
the **live** nginx 443 block (preserving the Certbot TLS config), `nginx
-t`, reload. Verified CSP/HSTS/XFO/XCTO/Referrer live.
- **L4** — appended the sandbox block (`CapabilityBoundingSet=`,
`ProtectKernel*`, `RestrictAddressFamilies`, `MemoryDenyWriteExecute`,
`SystemCallFilter=@system-service`, `UMask=0077`, …) to the live
`ohm-rfc-app.service`, with a backup + restart + health-check +
auto-revert-on-failure. Service came up clean; no revert needed.
### 5. M5 secret (operator-run, never in-conversation)
`WEBHOOK_EMAIL_BOUNCE_SECRET` was confirmed **absent** from prod `.env`
(the bounce webhook was unauthenticated). The operator minted it
themselves (`openssl rand | gcloud secrets create … --data-file=-`) and
bound the reference; v0.27.0's deploy landed it. The endpoint now returns
**401** without the header (was 503-unset / would be 200-open). Note: OHM
has no bounce *source* wired today (Google Workspace SMTP relay delivers
bounces as DSN email, not webhooks), so the secret simply locks an
otherwise-callable endpoint.
## Friction / dead-ends (kept honest)
- `chmod 640 …/data/*` first failed: after setting the dir to `750`, the
non-root SSH user couldn't expand the glob; re-ran under `sudo sh -c`.
- After merging flotilla PR #1, `git checkout main && git pull` didn't
fast-forward (no upstream tracking) and the tree briefly reverted;
recovered with `git merge --ff-only origin/main`.
- The operator hit a `SyntaxError` running `secret bind`: a separate
`trial-merge-v1.2.0` branch had a **conflicted, unresolved** merge of
session-0028's flotilla v1.2.0 into the working tree (conflict markers in
`deploy.py`, in the exact `_PhaseRunner` block L1 touched). Aborted the
merge (non-destructive — branch + PR preserved) → CLI imports again.
- First v0.27.0 deploy **failed at phase 3**: the VM pulls code from
`git.benstull.org/benstull/rfc-app`, not the `git.wiggleverse.org` remote
I'd pushed/tagged to. Dual-pushed main + the v0.27.0 tag to the
`benstull` remote; redeploy succeeded.
- 60 backend test failures from the incomplete parallel branch (see §3).
## Findings status at session end
| Finding | Status |
| --- | --- |
| C1, H1, M1, M2, M4, M5, L2, L3, L4, L8, I1 | **Remediated + deployed** (v0.27.0 / live VM) |
| L1, I6, L9, I5 (flotilla) | **Merged + deployed** (PR #1) |
| M3, L5-RDP, L6, L7 (live VM) | **Applied + verified** |
| L5 SSH `0.0.0.0/0` | **Deferred** to IAP rollout (§19.2) |
| I2 (webhook replay) | Acceptable (idempotent handlers) |
| I3 (dead HTML-email branch), I4 (sync siteverify) | Open — minor |
## Cross-references
- flotilla **PR #1** (`wiggleverse/ohm-rfc-app-flotilla`), merge `7b32a46`;
exercised by OHM deploy `deploys.id=35`.
- rfc-app **PR #1** (`ben.stull/rfc-app`), merge `fe044ed`, tag **v0.27.0**;
dual-pushed to `git.benstull.org/benstull/rfc-app` (the VM's deploy
source). OHM deploy **`deploys.id=37`**.
- OHM pin: `ohm-rfc/.rfc-app-version` 0.26.0 → 0.27.0 (`c91f7c0`).
- GCP: VM `ohm-rfc-app`/`us-central1-a`/`wiggleverse-ohm`; new SA
`ohm-rfc-app-vm@…`; deleted FW rule `default-allow-rdp`; Secure Boot on.
- Source audit: local-only `~/git/ohm-infra/SECURITY-AUDIT-0026-2026-05-28.md`.
---
## Operating instructions for the next session
The audit is remediated; the open threads are infra hardening and a stuck
flotilla merge:
1. **IAP SSH rollout (§19.2)** — closes L5 (`default-allow-ssh` 0.0.0.0/0)
and retires the L9 first-connect-TOFU residue. Steps: add an IAP firewall
rule (`35.235.240.0/20` → tcp:22), grant `roles/iap.tunnelResourceAccessor`,
verify `gcloud compute ssh --tunnel-through-iap` works, **then** remove the
0.0.0.0/0 SSH rule, and flip the flotilla deploy path / deployment record
to `tunnel_through_iap=true`. Don't remove 0.0.0.0/0 until IAP is verified.
2. **Resolve `trial-merge-v1.2.0`** — to ship flotilla **v1.2.0** (session
0028's PR), do a proper 3-way merge: it conflicts with the L1 redaction
change in `_PhaseRunner.run` (both edited the same block). Keep both —
the redaction wrapping AND v1.2.0's robustness/record-maintenance verbs.
3. **Info items** — I3 (delete the dead `text/html` email branch or guard it
so HTML email can't ship without escaping) and I4 (make the Turnstile
siteverify call async so a slow CloudFlare call can't block the event
loop). Both small, both rfc-app-core.
4. **Cleanup** — `sudo rm` the orphaned `ohm-app.db\r*` files on the VM once
confirmed orphaned; consider repo-side path-alignment of the stale
`deploy/nginx` + `deploy/systemd` templates (`/opt/rfc-app` →
`/opt/ohm-rfc-app`, `User=rfc-app` → `ohm-rfc-app`) so a future operator
doesn't copy them verbatim.
Suggested next-session launch prompt:
> Harden OHM SSH via IAP (§19.2): add the IAP firewall rule + tunnel role,
> verify IAP SSH, then remove the 0.0.0.0/0 SSH rule and switch flotilla's
> deploy path to tunnel-through-iap (closes audit-0026 L5 + L9). Then resolve
> the trial-merge-v1.2.0 conflict to ship flotilla v1.2.0 (it conflicts with
> the L1 redaction change in _PhaseRunner.run — keep both).