From d06a127e5b1cf7fdebc44e18b092d8722baba908 Mon Sep 17 00:00:00 2001 From: Ben Stull Date: Sun, 7 Jun 2026 06:49:14 -0700 Subject: [PATCH] add flotilla-core/0001/SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--2026-06-07T06-47.md + replace placeholder/variant SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--INPROGRESS.md --- ...RIPT-2026-06-06T22-53--2026-06-07T06-47.md | 182 ++++++++++++++++++ ...TRANSCRIPT-2026-06-06T22-53--INPROGRESS.md | 20 -- 2 files changed, 182 insertions(+), 20 deletions(-) create mode 100644 flotilla-core/0001/SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--2026-06-07T06-47.md delete mode 100644 flotilla-core/0001/SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--INPROGRESS.md diff --git a/flotilla-core/0001/SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--2026-06-07T06-47.md b/flotilla-core/0001/SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--2026-06-07T06-47.md new file mode 100644 index 0000000..7df99df --- /dev/null +++ b/flotilla-core/0001/SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--2026-06-07T06-47.md @@ -0,0 +1,182 @@ +# Session 0001.0 — Transcript + +> App: flotilla-core +> Type: coding +> Start: 2026-06-06T22-53 (PST) · End: 2026-06-07T06-47 (PST) +> Status: **FINALIZED.** + +## Launch prompt + +> git.wiggleverse.org is down. To deploy rfc-app ppe, I followed these +> instructions. Taking TLS from flexible to full allowed +> rfc-app-ppe.wiggleverse.org to receive connections, but git.wiggleverse.org +> went down. What should we do across all deployments to ensure they all work? + +(Also: flotilla-core had no `app.json` — created one this session to claim ID 0001.) + +## Deferred decisions + +_Autonomous-mode low-confidence calls the driver made and would have +liked operator input on. Appended as the session runs; surfaced at +finalize. Empty if none._ + +## Plan + +**Incident:** Operator flipped the Cloudflare `wiggleverse.org` zone SSL/TLS mode +from **Flexible → Full** to get `rfc-app-ppe.wiggleverse.org` serving. That made +ppe work but knocked `git.wiggleverse.org` "down". Question: what to do across +ALL deployments so they all work. + +**Root-cause hypothesis:** Cloudflare SSL/TLS encryption mode is a **zone-level** +setting — every proxied host under wiggleverse.org shares it. Flexible (CF→origin +over plain HTTP:80) vs Full/Full-strict (CF→origin over HTTPS:443). Hosts whose +origin nginx only serves :80 break under Full; hosts that redirect 80→443 or only +serve :443 break under Flexible. The fix can't be per-host via the mode toggle — +it must be: every origin serves HTTPS:443 with the CF Origin-CA wildcard cert, +then set the zone to Full (strict). This is exactly flotilla's +wildcard-tls-cutover design (docs/wildcard-tls-cutover-runbook.md). + +**Steps:** +1. Probe every public host under wiggleverse.org → which are green / which throw a + CF origin error (521/522/525/526) under the *current* mode. +2. Determine the current zone SSL/TLS mode (CF API or operator). +3. For each origin, confirm whether nginx serves :443 with the wildcard Origin-CA + cert (the flotilla secret-file delivery). +4. Bring every lagging origin up to HTTPS:443 + wildcard cert; set zone Full(strict). +5. Encode the invariant in flotilla docs/spec so a future deploy can't break siblings. + +## Live cutover progress (2026-06-06) + +**Diagnosis (evidence):** Cloudflare SSL/TLS mode is zone-wide. Operator has a +per-host Configuration Rule `rfc*.wiggleverse.org → Full`; zone default = Flexible. +DNS (CF API): proxied web origins = git, rfc, rfc-ppe, wiggle-snip-ppe; git-ssh is +grey (direct :2222). Mapped origins to VMs: +- rfc.wiggleverse.org = ohm-rfc-app (wiggleverse-ohm), 136.116.40.66 +- rfc-ppe = rfc-app-ppe (rfc-app-ppe proj), 35.253.21.170 +- wiggle-snip-ppe = wiggle-snip-ppe (wiggle-snip proj), 35.255.99.91 +- git = 34.55.46.221 (Gitea infra, NOT in my gcloud projects — out of reach) + +**Origin :443 state probed:** +- rfc (ohm): Origin-CA cert ✓, :80→:443 redirect ✓, but :443 was WORLD-OPEN. +- rfc-ppe: firewalled CF-only ✓ (presumed Origin-CA). +- wiggle-snip-ppe: serves :443 but with a SELF-SIGNED cert + (CN=ppe.wiggle-snip.wiggleverse.org) at /etc/ssl/wiggle-snip-ppe/origin.crt — + works only because served via Flexible (:80). NOT strict-ready. +- git: :80 only, :443 closed. + +**DONE — firewall lockdown of ohm-rfc-app (the one reachable non-compliant origin):** +- Created `ohm-rfc-app-cf-web` (allow tcp:80,443 from the 15 Cloudflare IPv4 + ranges, target tags http-server,https-server) in wiggleverse-ohm. +- Deleted `default-allow-http` + `default-allow-https` (0.0.0.0/0). +- Verified: rfc.wiggleverse.org + /api/health stay 200 via CF; direct :80/:443 to + 136.116.40.66 now TIME OUT. IAP-SSH (:22) untouched. +- rfc-app-ppe + wiggle-snip already had the CF-only `*-cf-web` pattern. + +**Cannot do (handoff): ** zone flip to Full(strict) + delete rfc* rule (CF token is +DNS-scoped); git origin :443 + Origin-CA + firewall (no access to Gitea VM); +wiggle-snip-ppe Origin-CA cert delivery (needed before strict flip). + +## Deferred decisions +- Implementing flotilla enforcement as VERIFY-AND-FAIL at deploy + enforce at + provisioning (launch-app/provision-vm), per SPEC invariant 3 (flotilla is the + deploy actuator, not the provisioner). Confirming mechanism with operator. + +## Enforcement shipped (flotilla-core v0.10.0, PR #4) + +flotilla deploy now has a 10th phase "verify edge" (finalize -> 10): after +/api/health, SSH to origin and deterministically check nginx :443 + Cloudflare +Origin-CA cert (not self-signed) + :80->:443 redirect. Any violation FAILS the +deploy. New flotilla_core/edge.py (pure evaluate_edge + parse + gather, mirrors +health.py). --skip-edge-check break-glass. SPEC §8.1/§3-inv7 + policy doc updated. +461 tests pass. This is the verify-and-fail half (operator chose verify-and-fail + +provision-default). + +## Outstanding handoffs +1. Zone flip: Cloudflare wiggleverse.org SSL/TLS -> Full (strict) zone-wide, then + delete the rfc* Configuration Rule. (My CF token is DNS-scoped; needs dashboard + or a Zone-Settings:Edit token.) DO NOT flip until git + wiggle-snip-ppe are + strict-ready, else they break. +2. git.wiggleverse.org origin (34.55.46.221, Gitea infra, not in my gcloud + projects): add :443 + Origin-CA cert + :80->:443; firewall :80,443 to CF ranges + (keep git-ssh :2222 grey/direct). +3. wiggle-snip-ppe: replace self-signed /etc/ssl/wiggle-snip-ppe/origin.crt with + the Cloudflare Origin-CA *.wiggleverse.org cert (via flotilla wildcard-tls). +4. Provisioning half: launch-app/provision-vm must always deliver Origin-CA cert + + CF-only firewall + nginx :443/redirect (separate repo). Until then a new + non-compliant origin fails its first deploy (by design). + +## wiggle-snip-ppe cert fix (done) + git note +- Copied wildcard cert into wiggle-snip project, bound secret-files, delivered on + VM, repointed nginx vhost (/etc/ssl/certs/wiggleverse-wildcard.crt), reloaded. +- :443 now serves Cloudflare Origin-CA (valid 2041); still 200 through CF. +- :80->:443 redirect deliberately NOT added yet (would loop under Flexible); + add at zone-flip time. +- NOTE: full `flotilla deploy wiggle-snip` is blocked — pin read returns HTTP 521 + from git.wiggleverse.org (git origin unreachable from CF again; SSH/2222 fine). +- git.wiggleverse.org is 521 (down on the CF HTTP/HTTPS path) as of ~23:xx PST. + +## Correction (v0.10.1, PR #5) + git outage insight +- The wiggle-snip vhost (and launch-app provision-vm / session 0061) document the + REAL convention: nginx serves :80+:443 with NO origin redirect; Cloudflare + "Always Use HTTPS" does the http->https upgrade (a hard origin redirect loops + under Flexible). My 0.10.0 phase-9 check wrongly REQUIRED an origin redirect -> + would have failed correctly-configured hosts. Fixed: evaluate_edge now verifies + TLS only (serves :443 with Origin-CA cert). SPEC + policy doc corrected. +- Did NOT add a :80->:443 redirect to wiggle-snip (would loop under Flexible). +- GIT OUTAGE ROOT CAUSE confirmed: git is HTTP-only on :80, so ANY move of the + zone toward Full 521s it; reverting to Flexible restores it. This is exactly why + git must get :443 + Origin-CA BEFORE the zone can go Full(strict). Until then, + keep zone=Flexible + the rfc* Full rule. +- Cert/key live at wiggleverse/wiggleverse-wildcard-{fullchain,privkey}. + +## git strict-readiness DONE (the last blocker) +- git = `gitea` VM in project `wiggleverse` (Docker: gitea + nginx:alpine containers). +- nginx container was :80-only. Added :443: delivered wildcard Origin-CA cert to + /opt/gitea/certs (VM SA lacked Secret Manager OAuth scope, so streamed from + Secret Manager via operator creds over the SSH pipe — no local disk, no echo), + appended a :443 server block to /opt/gitea/nginx.conf (serves :80+:443, NO + redirect per session-0061), added 443:443 + cert mount to docker-compose.yml, + validated nginx -t in a throwaway container, recreated nginx only (gitea + container untouched, up 6 days). +- Firewall: created gitea-cf-web (:80,443 from CF ranges, tag gitea), deleted + world-open allow-web. Kept allow-gitea-ssh (:2222 grey). Direct :80/:443 now + time out; git via CF = 200; SSH :2222 OK. +- ALL FOUR origins now strict-ready + firewall-locked: rfc(ohm), rfc-ppe, + wiggle-snip-ppe, git. Only the operator Cloudflare flip remains. + +## Final state (session close, 2026-06-07) + +**Estate cutover to Cloudflare Full (strict) — COMPLETE.** The operator flipped the +`wiggleverse.org` zone to Full (strict) + Always Use HTTPS once all origins were +ready. Verified end-to-end: all 4 hosts `https=200` (not 525/526 → strict +validates each origin cert), `http→301→https` (Always Use HTTPS), direct origin +access times out, git SSH `:2222` OK, app `/api/health` + git API `200`. + +| Origin | :443 Origin-CA | Direct access disabled | +| --- | --- | --- | +| rfc (ohm-rfc-app) | ✓ | ✓ | +| rfc-ppe | ✓ | ✓ | +| wiggle-snip-ppe | ✓ (this session) | ✓ | +| git (gitea) | ✓ (this session) | ✓ | + +**Shipped (flotilla-core, all merged to main):** v0.9.0 edge-tls policy doc + SPEC +inv 7 (PR#3) → v0.10.0 enforce as deploy phase 9 (PR#4) → v0.10.1 TLS-only fix +(PR#5). 460 tests green. Plus `app.json` + `CLAUDE.md` registration. +**Shipped (engineering):** `infra/deploy-gitea-on-gcp.md` runbook (PR#16). + +**Deferred-decisions report:** the one logged call — flotilla enforcement as +*verify-and-fail at deploy + provision-default* (vs active-repair) — was surfaced +and **confirmed by the operator** via AskUserQuestion (chose verify-and-fail + +provision-default). No unresolved low-confidence calls remain. + +**Open follow-ups (next session):** +1. Provisioning-side enforcement — launch-app/provision-vm always creates CF-only + firewall + Origin-CA cert + nginx :80+:443 (the provision-default half). +2. gitea VM lacks the Secret Manager OAuth scope → cert can't self-heal on + rotation; re-stream via operator creds (runbook) or stop/start with + `--scopes=cloud-platform`. + +## Next /goal +``` +/goal Implement the provisioning-side edge enforcement: make launch-app/provision-vm always create the CF-only firewall + deliver the Origin-CA *.wiggleverse.org cert + nginx serving :80+:443 (no redirect), so new deployments are born edge-compliant (provision-default half of the verify-and-fail approach already enforced by flotilla-core phase 9). Also fix the gitea VM Secret Manager OAuth scope for cert rotation. +``` diff --git a/flotilla-core/0001/SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--INPROGRESS.md b/flotilla-core/0001/SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--INPROGRESS.md deleted file mode 100644 index c19f596..0000000 --- a/flotilla-core/0001/SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--INPROGRESS.md +++ /dev/null @@ -1,20 +0,0 @@ -# Session 0001.0 — Transcript - -> App: flotilla-core -> Start: 2026-06-06T22-53 (PST) -> Type: coding -> Status: **PLACEHOLDER — claimed at session start; finalized at session end.** -> -> This file reserves session ID 0001 for flotilla-core. The driver replaces this -> body with the full transcript and renames the file to its final -> SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--.md form at session end. - -## Launch prompt - -_(launch prompt not captured at claim time)_ - -## Deferred decisions - -_Autonomous-mode low-confidence calls the driver made and would have -liked operator input on. Appended as the session runs; surfaced at -finalize. Empty if none._