reshape: move OHM transcripts under ohm/ for org-wide multi-app layout; add org-wide root README
This commit is contained in:
@@ -0,0 +1,230 @@
|
||||
# Session 0030.0 — Transcript
|
||||
|
||||
> Date: 2026-05-28
|
||||
> Start: 2026-05-28T16-12 (PST implied)
|
||||
> End: 2026-05-28T18-41 (PST implied)
|
||||
> Subject: Security-audit-0026 FULL remediation — flotilla fixes, live VM
|
||||
> hardening, and the rfc-app v0.27.0 security release (shipped + deployed)
|
||||
> Status: **FINALIZED.**
|
||||
|
||||
> Sanitization note: this transcript is published to the public
|
||||
> `wiggleverse/ohm-session-history`. It references audit findings by **ID
|
||||
> and general category only** — no exploit sketches or step-by-step
|
||||
> reproduction. (At the first finalize, several findings were still live
|
||||
> and unpatched, which made this mandatory; by this finalize the Critical
|
||||
> and the rest of the application findings are remediated + deployed, but
|
||||
> the convention holds — L5 SSH remains open, and exploit recipes never
|
||||
> belong in a public record.) The full-detail audit
|
||||
> (`SECURITY-AUDIT-0026-2026-05-28.md`) lives only in local `~/git/ohm-infra`
|
||||
> (not a git repo) and was never published. No secret bytes appear here
|
||||
> (§3 invariant 1).
|
||||
|
||||
---
|
||||
|
||||
## Launch prompt
|
||||
|
||||
> full remediation of security report at ~/git/ohm-infra
|
||||
|
||||
Plus two follow-on operator directives mid-session: **"ship v0.25.0"**
|
||||
(the security branch) and a request to bind `WEBHOOK_EMAIL_BOUNCE_SECRET`.
|
||||
|
||||
The "security report" is `SECURITY-AUDIT-0026-2026-05-28.md` (from the
|
||||
parallel audit session 0026): 1 Critical, 1 High, 5 Medium, 9 Low, 6 Info
|
||||
across the rfc-app codebase, the flotilla CLI, and the live GCP/VM.
|
||||
|
||||
---
|
||||
|
||||
## Outcome in one line
|
||||
|
||||
Every actionable audit-0026 finding is remediated and **live on
|
||||
production**, via three vehicles: flotilla **PR #1** (merged + deployed),
|
||||
direct **live GCP/VM ops**, and the rfc-app **v0.27.0** security release
|
||||
(`deploys.id=37`, verified live). Only the SSH-from-anywhere Low (deferred
|
||||
to a proper IAP rollout) and a few Info items remain.
|
||||
|
||||
## The "did we leak the audit?" scare (resolved: NO)
|
||||
|
||||
Early on the operator worried a transcript containing the audit had been
|
||||
pushed to the public repo before an internet drop. Froze work and verified
|
||||
the **entire git history** of `wiggleverse/ohm-session-history` (fresh
|
||||
clone, all branches, `git log -S` for audit-unique strings): the audit
|
||||
session 0026 had published **only its 454-byte placeholder**; **zero**
|
||||
audit-unique strings in any commit ever; the audit file itself lives only
|
||||
in local `~/git/ohm-infra` (not a git repo). Nothing leaked. The atomic
|
||||
publish push had failed on the internet drop, leaving only the
|
||||
start-gesture placeholder.
|
||||
|
||||
## Coordination
|
||||
|
||||
The rfc-app application findings were being remediated on an **uncommitted**
|
||||
`feature/v0.25.0-security-hardening` branch by a parallel session. With the
|
||||
operator, Session 0030 first scoped to the non-overlapping work (flotilla +
|
||||
live VM). Later the operator directed this session to **take over and ship**
|
||||
that branch — which it did (as v0.27.0; see below).
|
||||
|
||||
---
|
||||
|
||||
## Work performed
|
||||
|
||||
### 1. Flotilla repo (PR #1 — merged + deployed)
|
||||
|
||||
Branch `session-0030/flotilla-security-hardening` → **PR #1** → merged
|
||||
(`7b32a46`). 161 tests pass.
|
||||
|
||||
- **L1** — secret redactor threaded into `_PhaseRunner`; both success and
|
||||
failure phase-`detail` paths are scrubbed before persistence/printing.
|
||||
- **I6** — `redact_live` matches multiple byte-decodings (utf-8/replace,
|
||||
surrogateescape, latin-1) so a non-ASCII secret can't slip the needle.
|
||||
- **L9** — ssh adds `StrictHostKeyChecking=accept-new` (refuses a *changed*
|
||||
host key instead of blanket TOFU-accept).
|
||||
- **I5** — `overlay set` warns on unambiguous secret-word key names (not
|
||||
generic `*_KEY`, to avoid nagging on public site keys).
|
||||
|
||||
### 2. Live GCP/VM ops (project `wiggleverse-ohm`, VM `ohm-rfc-app`)
|
||||
|
||||
- **M3** — data dir → `750`, DB + WAL/SHM → `640` (survives reboot +
|
||||
restart). Also secured (did not delete) orphaned `ohm-app.db\r*` files
|
||||
left by the historical CRLF path bug.
|
||||
- **L5** — deleted the dead `default-allow-rdp` firewall rule. SSH-from-
|
||||
anywhere deferred to an IAP rollout (operator decision).
|
||||
- **L6** — created dedicated least-priv SA `ohm-rfc-app-vm@…` (logging +
|
||||
monitoring writer only; no Secret Manager — the app reads `.env`), swapped
|
||||
the VM off the default compute SA.
|
||||
- **L7** — enabled Shielded-VM Secure Boot (UEFI-verified first; vTPM +
|
||||
integrity monitoring already on).
|
||||
|
||||
L6 + L7 applied in one stop/start window (~15s downtime).
|
||||
|
||||
### 3. rfc-app v0.27.0 — the security release (shipped + deployed)
|
||||
|
||||
The "v0.25.0" branch was based on v0.24.0, but main/live was already
|
||||
**v0.26.0** (#28). Shipping it as v0.25.0 would have downgraded prod, so it
|
||||
was **rebased onto main and reversioned to v0.27.0**. Then: committed,
|
||||
version-bumped (`VERSION` + `package.json` + a CHANGELOG entry with upgrade
|
||||
steps), tested, PR'd (**PR #1 on `ben.stull/rfc-app`**), merged (`fe044ed`),
|
||||
tagged **v0.27.0**, pin bumped in `ohm-rfc/.rfc-app-version` (0.26.0→0.27.0),
|
||||
deployed (`deploys.id=37`), and verified live.
|
||||
|
||||
Findings landed: **C1** (DOMPurify sanitize chokepoint for every
|
||||
markdown→HTML sink — the Critical), **H1** (per-IP ratelimit + per-account
|
||||
OTC lockout, migration 023 auto-applied), **M1** (indexed single-row
|
||||
device-trust lookup), **M2** (nginx security headers — see below), **M4**
|
||||
(session cookie Secure-by-default), **M5** (bounce webhook fails closed +
|
||||
secret bound), **L2/L3** (throttles), **L4** (systemd sandbox), **L8/I1**
|
||||
(server_tokens off, legacy TLS removed).
|
||||
|
||||
**The parallel branch was incomplete** — its changes broke 60 backend
|
||||
tests, which this session diagnosed and fixed:
|
||||
- M4 set the session cookie `Secure`, but `TestClient` talks plain http,
|
||||
so the cookie was never returned and every authenticated flow failed →
|
||||
the test env now sets `SESSION_COOKIE_SECURE=false` (the documented
|
||||
dev opt-out).
|
||||
- M5 fail-closed returned 503 in tests → test env sets
|
||||
`RFC_APP_INSECURE_BOUNCE_WEBHOOK=1` (mirrors the existing
|
||||
`RFC_APP_INSECURE_WEBHOOKS` pattern).
|
||||
- The new per-IP rate limiter is a process-global singleton with no
|
||||
per-test reset, so later tests tripped the budget → added
|
||||
`ratelimit._reset_all_for_tests()` + an autouse `conftest.py` fixture.
|
||||
Result: **363 backend tests green; frontend builds clean (0 vulns).**
|
||||
|
||||
### 4. M2 + L4 applied out-of-band (the deploy gesture does not)
|
||||
|
||||
The flotilla deploy does **not** install nginx/systemd files. The repo's
|
||||
`deploy/` templates also carried **stale pre-rename paths** (`User=rfc-app`,
|
||||
`/opt/rfc-app/…`) that would have taken the site down if copied verbatim.
|
||||
So:
|
||||
- **M2/L8/I1** — injected the 5 security headers + `server_tokens off` into
|
||||
the **live** nginx 443 block (preserving the Certbot TLS config), `nginx
|
||||
-t`, reload. Verified CSP/HSTS/XFO/XCTO/Referrer live.
|
||||
- **L4** — appended the sandbox block (`CapabilityBoundingSet=`,
|
||||
`ProtectKernel*`, `RestrictAddressFamilies`, `MemoryDenyWriteExecute`,
|
||||
`SystemCallFilter=@system-service`, `UMask=0077`, …) to the live
|
||||
`ohm-rfc-app.service`, with a backup + restart + health-check +
|
||||
auto-revert-on-failure. Service came up clean; no revert needed.
|
||||
|
||||
### 5. M5 secret (operator-run, never in-conversation)
|
||||
|
||||
`WEBHOOK_EMAIL_BOUNCE_SECRET` was confirmed **absent** from prod `.env`
|
||||
(the bounce webhook was unauthenticated). The operator minted it
|
||||
themselves (`openssl rand | gcloud secrets create … --data-file=-`) and
|
||||
bound the reference; v0.27.0's deploy landed it. The endpoint now returns
|
||||
**401** without the header (was 503-unset / would be 200-open). Note: OHM
|
||||
has no bounce *source* wired today (Google Workspace SMTP relay delivers
|
||||
bounces as DSN email, not webhooks), so the secret simply locks an
|
||||
otherwise-callable endpoint.
|
||||
|
||||
## Friction / dead-ends (kept honest)
|
||||
|
||||
- `chmod 640 …/data/*` first failed: after setting the dir to `750`, the
|
||||
non-root SSH user couldn't expand the glob; re-ran under `sudo sh -c`.
|
||||
- After merging flotilla PR #1, `git checkout main && git pull` didn't
|
||||
fast-forward (no upstream tracking) and the tree briefly reverted;
|
||||
recovered with `git merge --ff-only origin/main`.
|
||||
- The operator hit a `SyntaxError` running `secret bind`: a separate
|
||||
`trial-merge-v1.2.0` branch had a **conflicted, unresolved** merge of
|
||||
session-0028's flotilla v1.2.0 into the working tree (conflict markers in
|
||||
`deploy.py`, in the exact `_PhaseRunner` block L1 touched). Aborted the
|
||||
merge (non-destructive — branch + PR preserved) → CLI imports again.
|
||||
- First v0.27.0 deploy **failed at phase 3**: the VM pulls code from
|
||||
`git.benstull.org/benstull/rfc-app`, not the `git.wiggleverse.org` remote
|
||||
I'd pushed/tagged to. Dual-pushed main + the v0.27.0 tag to the
|
||||
`benstull` remote; redeploy succeeded.
|
||||
- 60 backend test failures from the incomplete parallel branch (see §3).
|
||||
|
||||
## Findings status at session end
|
||||
|
||||
| Finding | Status |
|
||||
| --- | --- |
|
||||
| C1, H1, M1, M2, M4, M5, L2, L3, L4, L8, I1 | **Remediated + deployed** (v0.27.0 / live VM) |
|
||||
| L1, I6, L9, I5 (flotilla) | **Merged + deployed** (PR #1) |
|
||||
| M3, L5-RDP, L6, L7 (live VM) | **Applied + verified** |
|
||||
| L5 SSH `0.0.0.0/0` | **Deferred** to IAP rollout (§19.2) |
|
||||
| I2 (webhook replay) | Acceptable (idempotent handlers) |
|
||||
| I3 (dead HTML-email branch), I4 (sync siteverify) | Open — minor |
|
||||
|
||||
## Cross-references
|
||||
|
||||
- flotilla **PR #1** (`wiggleverse/ohm-rfc-app-flotilla`), merge `7b32a46`;
|
||||
exercised by OHM deploy `deploys.id=35`.
|
||||
- rfc-app **PR #1** (`ben.stull/rfc-app`), merge `fe044ed`, tag **v0.27.0**;
|
||||
dual-pushed to `git.benstull.org/benstull/rfc-app` (the VM's deploy
|
||||
source). OHM deploy **`deploys.id=37`**.
|
||||
- OHM pin: `ohm-rfc/.rfc-app-version` 0.26.0 → 0.27.0 (`c91f7c0`).
|
||||
- GCP: VM `ohm-rfc-app`/`us-central1-a`/`wiggleverse-ohm`; new SA
|
||||
`ohm-rfc-app-vm@…`; deleted FW rule `default-allow-rdp`; Secure Boot on.
|
||||
- Source audit: local-only `~/git/ohm-infra/SECURITY-AUDIT-0026-2026-05-28.md`.
|
||||
|
||||
---
|
||||
|
||||
## Operating instructions for the next session
|
||||
|
||||
The audit is remediated; the open threads are infra hardening and a stuck
|
||||
flotilla merge:
|
||||
|
||||
1. **IAP SSH rollout (§19.2)** — closes L5 (`default-allow-ssh` 0.0.0.0/0)
|
||||
and retires the L9 first-connect-TOFU residue. Steps: add an IAP firewall
|
||||
rule (`35.235.240.0/20` → tcp:22), grant `roles/iap.tunnelResourceAccessor`,
|
||||
verify `gcloud compute ssh --tunnel-through-iap` works, **then** remove the
|
||||
0.0.0.0/0 SSH rule, and flip the flotilla deploy path / deployment record
|
||||
to `tunnel_through_iap=true`. Don't remove 0.0.0.0/0 until IAP is verified.
|
||||
2. **Resolve `trial-merge-v1.2.0`** — to ship flotilla **v1.2.0** (session
|
||||
0028's PR), do a proper 3-way merge: it conflicts with the L1 redaction
|
||||
change in `_PhaseRunner.run` (both edited the same block). Keep both —
|
||||
the redaction wrapping AND v1.2.0's robustness/record-maintenance verbs.
|
||||
3. **Info items** — I3 (delete the dead `text/html` email branch or guard it
|
||||
so HTML email can't ship without escaping) and I4 (make the Turnstile
|
||||
siteverify call async so a slow CloudFlare call can't block the event
|
||||
loop). Both small, both rfc-app-core.
|
||||
4. **Cleanup** — `sudo rm` the orphaned `ohm-app.db\r*` files on the VM once
|
||||
confirmed orphaned; consider repo-side path-alignment of the stale
|
||||
`deploy/nginx` + `deploy/systemd` templates (`/opt/rfc-app` →
|
||||
`/opt/ohm-rfc-app`, `User=rfc-app` → `ohm-rfc-app`) so a future operator
|
||||
doesn't copy them verbatim.
|
||||
|
||||
Suggested next-session launch prompt:
|
||||
|
||||
> Harden OHM SSH via IAP (§19.2): add the IAP firewall rule + tunnel role,
|
||||
> verify IAP SSH, then remove the 0.0.0.0/0 SSH rule and switch flotilla's
|
||||
> deploy path to tunnel-through-iap (closes audit-0026 L5 + L9). Then resolve
|
||||
> the trial-merge-v1.2.0 conflict to ship flotilla v1.2.0 (it conflicts with
|
||||
> the L1 redaction change in _PhaseRunner.run — keep both).
|
||||
Reference in New Issue
Block a user