231 lines
11 KiB
Markdown
231 lines
11 KiB
Markdown
# Session 0030.0 — Transcript
|
|
|
|
> Date: 2026-05-28
|
|
> Start: 2026-05-28T16-12 (PST implied)
|
|
> End: 2026-05-28T18-41 (PST implied)
|
|
> Subject: Security-audit-0026 FULL remediation — flotilla fixes, live VM
|
|
> hardening, and the rfc-app v0.27.0 security release (shipped + deployed)
|
|
> Status: **FINALIZED.**
|
|
|
|
> Sanitization note: this transcript is published to the public
|
|
> `wiggleverse/ohm-session-history`. It references audit findings by **ID
|
|
> and general category only** — no exploit sketches or step-by-step
|
|
> reproduction. (At the first finalize, several findings were still live
|
|
> and unpatched, which made this mandatory; by this finalize the Critical
|
|
> and the rest of the application findings are remediated + deployed, but
|
|
> the convention holds — L5 SSH remains open, and exploit recipes never
|
|
> belong in a public record.) The full-detail audit
|
|
> (`SECURITY-AUDIT-0026-2026-05-28.md`) lives only in local `~/git/ohm-infra`
|
|
> (not a git repo) and was never published. No secret bytes appear here
|
|
> (§3 invariant 1).
|
|
|
|
---
|
|
|
|
## Launch prompt
|
|
|
|
> full remediation of security report at ~/git/ohm-infra
|
|
|
|
Plus two follow-on operator directives mid-session: **"ship v0.25.0"**
|
|
(the security branch) and a request to bind `WEBHOOK_EMAIL_BOUNCE_SECRET`.
|
|
|
|
The "security report" is `SECURITY-AUDIT-0026-2026-05-28.md` (from the
|
|
parallel audit session 0026): 1 Critical, 1 High, 5 Medium, 9 Low, 6 Info
|
|
across the rfc-app codebase, the flotilla CLI, and the live GCP/VM.
|
|
|
|
---
|
|
|
|
## Outcome in one line
|
|
|
|
Every actionable audit-0026 finding is remediated and **live on
|
|
production**, via three vehicles: flotilla **PR #1** (merged + deployed),
|
|
direct **live GCP/VM ops**, and the rfc-app **v0.27.0** security release
|
|
(`deploys.id=37`, verified live). Only the SSH-from-anywhere Low (deferred
|
|
to a proper IAP rollout) and a few Info items remain.
|
|
|
|
## The "did we leak the audit?" scare (resolved: NO)
|
|
|
|
Early on the operator worried a transcript containing the audit had been
|
|
pushed to the public repo before an internet drop. Froze work and verified
|
|
the **entire git history** of `wiggleverse/ohm-session-history` (fresh
|
|
clone, all branches, `git log -S` for audit-unique strings): the audit
|
|
session 0026 had published **only its 454-byte placeholder**; **zero**
|
|
audit-unique strings in any commit ever; the audit file itself lives only
|
|
in local `~/git/ohm-infra` (not a git repo). Nothing leaked. The atomic
|
|
publish push had failed on the internet drop, leaving only the
|
|
start-gesture placeholder.
|
|
|
|
## Coordination
|
|
|
|
The rfc-app application findings were being remediated on an **uncommitted**
|
|
`feature/v0.25.0-security-hardening` branch by a parallel session. With the
|
|
operator, Session 0030 first scoped to the non-overlapping work (flotilla +
|
|
live VM). Later the operator directed this session to **take over and ship**
|
|
that branch — which it did (as v0.27.0; see below).
|
|
|
|
---
|
|
|
|
## Work performed
|
|
|
|
### 1. Flotilla repo (PR #1 — merged + deployed)
|
|
|
|
Branch `session-0030/flotilla-security-hardening` → **PR #1** → merged
|
|
(`7b32a46`). 161 tests pass.
|
|
|
|
- **L1** — secret redactor threaded into `_PhaseRunner`; both success and
|
|
failure phase-`detail` paths are scrubbed before persistence/printing.
|
|
- **I6** — `redact_live` matches multiple byte-decodings (utf-8/replace,
|
|
surrogateescape, latin-1) so a non-ASCII secret can't slip the needle.
|
|
- **L9** — ssh adds `StrictHostKeyChecking=accept-new` (refuses a *changed*
|
|
host key instead of blanket TOFU-accept).
|
|
- **I5** — `overlay set` warns on unambiguous secret-word key names (not
|
|
generic `*_KEY`, to avoid nagging on public site keys).
|
|
|
|
### 2. Live GCP/VM ops (project `wiggleverse-ohm`, VM `ohm-rfc-app`)
|
|
|
|
- **M3** — data dir → `750`, DB + WAL/SHM → `640` (survives reboot +
|
|
restart). Also secured (did not delete) orphaned `ohm-app.db\r*` files
|
|
left by the historical CRLF path bug.
|
|
- **L5** — deleted the dead `default-allow-rdp` firewall rule. SSH-from-
|
|
anywhere deferred to an IAP rollout (operator decision).
|
|
- **L6** — created dedicated least-priv SA `ohm-rfc-app-vm@…` (logging +
|
|
monitoring writer only; no Secret Manager — the app reads `.env`), swapped
|
|
the VM off the default compute SA.
|
|
- **L7** — enabled Shielded-VM Secure Boot (UEFI-verified first; vTPM +
|
|
integrity monitoring already on).
|
|
|
|
L6 + L7 applied in one stop/start window (~15s downtime).
|
|
|
|
### 3. rfc-app v0.27.0 — the security release (shipped + deployed)
|
|
|
|
The "v0.25.0" branch was based on v0.24.0, but main/live was already
|
|
**v0.26.0** (#28). Shipping it as v0.25.0 would have downgraded prod, so it
|
|
was **rebased onto main and reversioned to v0.27.0**. Then: committed,
|
|
version-bumped (`VERSION` + `package.json` + a CHANGELOG entry with upgrade
|
|
steps), tested, PR'd (**PR #1 on `ben.stull/rfc-app`**), merged (`fe044ed`),
|
|
tagged **v0.27.0**, pin bumped in `ohm-rfc/.rfc-app-version` (0.26.0→0.27.0),
|
|
deployed (`deploys.id=37`), and verified live.
|
|
|
|
Findings landed: **C1** (DOMPurify sanitize chokepoint for every
|
|
markdown→HTML sink — the Critical), **H1** (per-IP ratelimit + per-account
|
|
OTC lockout, migration 023 auto-applied), **M1** (indexed single-row
|
|
device-trust lookup), **M2** (nginx security headers — see below), **M4**
|
|
(session cookie Secure-by-default), **M5** (bounce webhook fails closed +
|
|
secret bound), **L2/L3** (throttles), **L4** (systemd sandbox), **L8/I1**
|
|
(server_tokens off, legacy TLS removed).
|
|
|
|
**The parallel branch was incomplete** — its changes broke 60 backend
|
|
tests, which this session diagnosed and fixed:
|
|
- M4 set the session cookie `Secure`, but `TestClient` talks plain http,
|
|
so the cookie was never returned and every authenticated flow failed →
|
|
the test env now sets `SESSION_COOKIE_SECURE=false` (the documented
|
|
dev opt-out).
|
|
- M5 fail-closed returned 503 in tests → test env sets
|
|
`RFC_APP_INSECURE_BOUNCE_WEBHOOK=1` (mirrors the existing
|
|
`RFC_APP_INSECURE_WEBHOOKS` pattern).
|
|
- The new per-IP rate limiter is a process-global singleton with no
|
|
per-test reset, so later tests tripped the budget → added
|
|
`ratelimit._reset_all_for_tests()` + an autouse `conftest.py` fixture.
|
|
Result: **363 backend tests green; frontend builds clean (0 vulns).**
|
|
|
|
### 4. M2 + L4 applied out-of-band (the deploy gesture does not)
|
|
|
|
The flotilla deploy does **not** install nginx/systemd files. The repo's
|
|
`deploy/` templates also carried **stale pre-rename paths** (`User=rfc-app`,
|
|
`/opt/rfc-app/…`) that would have taken the site down if copied verbatim.
|
|
So:
|
|
- **M2/L8/I1** — injected the 5 security headers + `server_tokens off` into
|
|
the **live** nginx 443 block (preserving the Certbot TLS config), `nginx
|
|
-t`, reload. Verified CSP/HSTS/XFO/XCTO/Referrer live.
|
|
- **L4** — appended the sandbox block (`CapabilityBoundingSet=`,
|
|
`ProtectKernel*`, `RestrictAddressFamilies`, `MemoryDenyWriteExecute`,
|
|
`SystemCallFilter=@system-service`, `UMask=0077`, …) to the live
|
|
`ohm-rfc-app.service`, with a backup + restart + health-check +
|
|
auto-revert-on-failure. Service came up clean; no revert needed.
|
|
|
|
### 5. M5 secret (operator-run, never in-conversation)
|
|
|
|
`WEBHOOK_EMAIL_BOUNCE_SECRET` was confirmed **absent** from prod `.env`
|
|
(the bounce webhook was unauthenticated). The operator minted it
|
|
themselves (`openssl rand | gcloud secrets create … --data-file=-`) and
|
|
bound the reference; v0.27.0's deploy landed it. The endpoint now returns
|
|
**401** without the header (was 503-unset / would be 200-open). Note: OHM
|
|
has no bounce *source* wired today (Google Workspace SMTP relay delivers
|
|
bounces as DSN email, not webhooks), so the secret simply locks an
|
|
otherwise-callable endpoint.
|
|
|
|
## Friction / dead-ends (kept honest)
|
|
|
|
- `chmod 640 …/data/*` first failed: after setting the dir to `750`, the
|
|
non-root SSH user couldn't expand the glob; re-ran under `sudo sh -c`.
|
|
- After merging flotilla PR #1, `git checkout main && git pull` didn't
|
|
fast-forward (no upstream tracking) and the tree briefly reverted;
|
|
recovered with `git merge --ff-only origin/main`.
|
|
- The operator hit a `SyntaxError` running `secret bind`: a separate
|
|
`trial-merge-v1.2.0` branch had a **conflicted, unresolved** merge of
|
|
session-0028's flotilla v1.2.0 into the working tree (conflict markers in
|
|
`deploy.py`, in the exact `_PhaseRunner` block L1 touched). Aborted the
|
|
merge (non-destructive — branch + PR preserved) → CLI imports again.
|
|
- First v0.27.0 deploy **failed at phase 3**: the VM pulls code from
|
|
`git.benstull.org/benstull/rfc-app`, not the `git.wiggleverse.org` remote
|
|
I'd pushed/tagged to. Dual-pushed main + the v0.27.0 tag to the
|
|
`benstull` remote; redeploy succeeded.
|
|
- 60 backend test failures from the incomplete parallel branch (see §3).
|
|
|
|
## Findings status at session end
|
|
|
|
| Finding | Status |
|
|
| --- | --- |
|
|
| C1, H1, M1, M2, M4, M5, L2, L3, L4, L8, I1 | **Remediated + deployed** (v0.27.0 / live VM) |
|
|
| L1, I6, L9, I5 (flotilla) | **Merged + deployed** (PR #1) |
|
|
| M3, L5-RDP, L6, L7 (live VM) | **Applied + verified** |
|
|
| L5 SSH `0.0.0.0/0` | **Deferred** to IAP rollout (§19.2) |
|
|
| I2 (webhook replay) | Acceptable (idempotent handlers) |
|
|
| I3 (dead HTML-email branch), I4 (sync siteverify) | Open — minor |
|
|
|
|
## Cross-references
|
|
|
|
- flotilla **PR #1** (`wiggleverse/ohm-rfc-app-flotilla`), merge `7b32a46`;
|
|
exercised by OHM deploy `deploys.id=35`.
|
|
- rfc-app **PR #1** (`ben.stull/rfc-app`), merge `fe044ed`, tag **v0.27.0**;
|
|
dual-pushed to `git.benstull.org/benstull/rfc-app` (the VM's deploy
|
|
source). OHM deploy **`deploys.id=37`**.
|
|
- OHM pin: `ohm-rfc/.rfc-app-version` 0.26.0 → 0.27.0 (`c91f7c0`).
|
|
- GCP: VM `ohm-rfc-app`/`us-central1-a`/`wiggleverse-ohm`; new SA
|
|
`ohm-rfc-app-vm@…`; deleted FW rule `default-allow-rdp`; Secure Boot on.
|
|
- Source audit: local-only `~/git/ohm-infra/SECURITY-AUDIT-0026-2026-05-28.md`.
|
|
|
|
---
|
|
|
|
## Operating instructions for the next session
|
|
|
|
The audit is remediated; the open threads are infra hardening and a stuck
|
|
flotilla merge:
|
|
|
|
1. **IAP SSH rollout (§19.2)** — closes L5 (`default-allow-ssh` 0.0.0.0/0)
|
|
and retires the L9 first-connect-TOFU residue. Steps: add an IAP firewall
|
|
rule (`35.235.240.0/20` → tcp:22), grant `roles/iap.tunnelResourceAccessor`,
|
|
verify `gcloud compute ssh --tunnel-through-iap` works, **then** remove the
|
|
0.0.0.0/0 SSH rule, and flip the flotilla deploy path / deployment record
|
|
to `tunnel_through_iap=true`. Don't remove 0.0.0.0/0 until IAP is verified.
|
|
2. **Resolve `trial-merge-v1.2.0`** — to ship flotilla **v1.2.0** (session
|
|
0028's PR), do a proper 3-way merge: it conflicts with the L1 redaction
|
|
change in `_PhaseRunner.run` (both edited the same block). Keep both —
|
|
the redaction wrapping AND v1.2.0's robustness/record-maintenance verbs.
|
|
3. **Info items** — I3 (delete the dead `text/html` email branch or guard it
|
|
so HTML email can't ship without escaping) and I4 (make the Turnstile
|
|
siteverify call async so a slow CloudFlare call can't block the event
|
|
loop). Both small, both rfc-app-core.
|
|
4. **Cleanup** — `sudo rm` the orphaned `ohm-app.db\r*` files on the VM once
|
|
confirmed orphaned; consider repo-side path-alignment of the stale
|
|
`deploy/nginx` + `deploy/systemd` templates (`/opt/rfc-app` →
|
|
`/opt/ohm-rfc-app`, `User=rfc-app` → `ohm-rfc-app`) so a future operator
|
|
doesn't copy them verbatim.
|
|
|
|
Suggested next-session launch prompt:
|
|
|
|
> Harden OHM SSH via IAP (§19.2): add the IAP firewall rule + tunnel role,
|
|
> verify IAP SSH, then remove the 0.0.0.0/0 SSH rule and switch flotilla's
|
|
> deploy path to tunnel-through-iap (closes audit-0026 L5 + L9). Then resolve
|
|
> the trial-merge-v1.2.0 conflict to ship flotilla v1.2.0 (it conflicts with
|
|
> the L1 redaction change in _PhaseRunner.run — keep both).
|