session-history/flotilla-core/0001/SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--2026-06-07T06-47.md

Session 0001.0 — Transcript

App: flotilla-core Type: coding Start: 2026-06-06T22-53 (PST) · End: 2026-06-07T06-47 (PST) Status: FINALIZED.

Launch prompt

git.wiggleverse.org is down. To deploy rfc-app ppe, I followed these instructions. Taking TLS from flexible to full allowed rfc-app-ppe.wiggleverse.org to receive connections, but git.wiggleverse.org went down. What should we do across all deployments to ensure they all work?

(Also: flotilla-core had no app.json — created one this session to claim ID 0001.)

Deferred decisions

Autonomous-mode low-confidence calls the driver made and would have liked operator input on. Appended as the session runs; surfaced at finalize. Empty if none.

Plan

Incident: Operator flipped the Cloudflare wiggleverse.org zone SSL/TLS mode from Flexible → Full to get rfc-app-ppe.wiggleverse.org serving. That made ppe work but knocked git.wiggleverse.org "down". Question: what to do across ALL deployments so they all work.

Root-cause hypothesis: Cloudflare SSL/TLS encryption mode is a zone-level setting — every proxied host under wiggleverse.org shares it. Flexible (CF→origin over plain HTTP:80) vs Full/Full-strict (CF→origin over HTTPS:443). Hosts whose origin nginx only serves :80 break under Full; hosts that redirect 80→443 or only serve :443 break under Flexible. The fix can't be per-host via the mode toggle — it must be: every origin serves HTTPS:443 with the CF Origin-CA wildcard cert, then set the zone to Full (strict). This is exactly flotilla's wildcard-tls-cutover design (docs/wildcard-tls-cutover-runbook.md).

Steps:

1. Probe every public host under wiggleverse.org → which are green / which throw a CF origin error (521/522/525/526) under the current mode.
2. Determine the current zone SSL/TLS mode (CF API or operator).
3. For each origin, confirm whether nginx serves :443 with the wildcard Origin-CA cert (the flotilla secret-file delivery).
4. Bring every lagging origin up to HTTPS:443 + wildcard cert; set zone Full(strict).
5. Encode the invariant in flotilla docs/spec so a future deploy can't break siblings.

Live cutover progress (2026-06-06)

Diagnosis (evidence): Cloudflare SSL/TLS mode is zone-wide. Operator has a per-host Configuration Rule rfc*.wiggleverse.org → Full; zone default = Flexible. DNS (CF API): proxied web origins = git, rfc, rfc-ppe, wiggle-snip-ppe; git-ssh is grey (direct :2222). Mapped origins to VMs:

• rfc.wiggleverse.org = ohm-rfc-app (wiggleverse-ohm), 136.116.40.66
• rfc-ppe = rfc-app-ppe (rfc-app-ppe proj), 35.253.21.170
• wiggle-snip-ppe = wiggle-snip-ppe (wiggle-snip proj), 35.255.99.91
• git = 34.55.46.221 (Gitea infra, NOT in my gcloud projects — out of reach)

Origin :443 state probed:

• rfc (ohm): Origin-CA cert ✓, :80→:443 redirect ✓, but :443 was WORLD-OPEN.
• rfc-ppe: firewalled CF-only ✓ (presumed Origin-CA).
• wiggle-snip-ppe: serves :443 but with a SELF-SIGNED cert (CN=ppe.wiggle-snip.wiggleverse.org) at /etc/ssl/wiggle-snip-ppe/origin.crt — works only because served via Flexible (:80). NOT strict-ready.
• git: :80 only, :443 closed.

DONE — firewall lockdown of ohm-rfc-app (the one reachable non-compliant origin):

• Created ohm-rfc-app-cf-web (allow tcp:80,443 from the 15 Cloudflare IPv4 ranges, target tags http-server,https-server) in wiggleverse-ohm.
• Deleted default-allow-http + default-allow-https (0.0.0.0/0).
• Verified: rfc.wiggleverse.org + /api/health stay 200 via CF; direct :80/:443 to 136.116.40.66 now TIME OUT. IAP-SSH (:22) untouched.
• rfc-app-ppe + wiggle-snip already had the CF-only *-cf-web pattern.

*Cannot do (handoff): ** zone flip to Full(strict) + delete rfc rule (CF token is DNS-scoped); git origin :443 + Origin-CA + firewall (no access to Gitea VM); wiggle-snip-ppe Origin-CA cert delivery (needed before strict flip).

Deferred decisions

• Implementing flotilla enforcement as VERIFY-AND-FAIL at deploy + enforce at provisioning (launch-app/provision-vm), per SPEC invariant 3 (flotilla is the deploy actuator, not the provisioner). Confirming mechanism with operator.

Enforcement shipped (flotilla-core v0.10.0, PR #4)

flotilla deploy now has a 10th phase "verify edge" (finalize -> 10): after /api/health, SSH to origin and deterministically check nginx :443 + Cloudflare Origin-CA cert (not self-signed) + :80->:443 redirect. Any violation FAILS the deploy. New flotilla_core/edge.py (pure evaluate_edge + parse + gather, mirrors health.py). --skip-edge-check break-glass. SPEC §8.1/§3-inv7 + policy doc updated. 461 tests pass. This is the verify-and-fail half (operator chose verify-and-fail + provision-default).

Outstanding handoffs

1. Zone flip: Cloudflare wiggleverse.org SSL/TLS -> Full (strict) zone-wide, then delete the rfc* Configuration Rule. (My CF token is DNS-scoped; needs dashboard or a Zone-Settings:Edit token.) DO NOT flip until git + wiggle-snip-ppe are strict-ready, else they break.
2. git.wiggleverse.org origin (34.55.46.221, Gitea infra, not in my gcloud projects): add :443 + Origin-CA cert + :80->:443; firewall :80,443 to CF ranges (keep git-ssh :2222 grey/direct).
3. wiggle-snip-ppe: replace self-signed /etc/ssl/wiggle-snip-ppe/origin.crt with the Cloudflare Origin-CA *.wiggleverse.org cert (via flotilla wildcard-tls).
4. Provisioning half: launch-app/provision-vm must always deliver Origin-CA cert + CF-only firewall + nginx :443/redirect (separate repo). Until then a new non-compliant origin fails its first deploy (by design).

wiggle-snip-ppe cert fix (done) + git note

• Copied wildcard cert into wiggle-snip project, bound secret-files, delivered on VM, repointed nginx vhost (/etc/ssl/certs/wiggleverse-wildcard.crt), reloaded.
• :443 now serves Cloudflare Origin-CA (valid 2041); still 200 through CF.
• :80->:443 redirect deliberately NOT added yet (would loop under Flexible); add at zone-flip time.
• NOTE: full flotilla deploy wiggle-snip is blocked — pin read returns HTTP 521 from git.wiggleverse.org (git origin unreachable from CF again; SSH/2222 fine).
• git.wiggleverse.org is 521 (down on the CF HTTP/HTTPS path) as of ~23:xx PST.

Correction (v0.10.1, PR #5) + git outage insight

• The wiggle-snip vhost (and launch-app provision-vm / session 0061) document the REAL convention: nginx serves :80+:443 with NO origin redirect; Cloudflare "Always Use HTTPS" does the http->https upgrade (a hard origin redirect loops under Flexible). My 0.10.0 phase-9 check wrongly REQUIRED an origin redirect -> would have failed correctly-configured hosts. Fixed: evaluate_edge now verifies TLS only (serves :443 with Origin-CA cert). SPEC + policy doc corrected.
• Did NOT add a :80->:443 redirect to wiggle-snip (would loop under Flexible).
• GIT OUTAGE ROOT CAUSE confirmed: git is HTTP-only on :80, so ANY move of the zone toward Full 521s it; reverting to Flexible restores it. This is exactly why git must get :443 + Origin-CA BEFORE the zone can go Full(strict). Until then, keep zone=Flexible + the rfc* Full rule.
• Cert/key live at wiggleverse/wiggleverse-wildcard-{fullchain,privkey}.

git strict-readiness DONE (the last blocker)

• git = gitea VM in project wiggleverse (Docker: gitea + nginx:alpine containers).
• nginx container was :80-only. Added :443: delivered wildcard Origin-CA cert to /opt/gitea/certs (VM SA lacked Secret Manager OAuth scope, so streamed from Secret Manager via operator creds over the SSH pipe — no local disk, no echo), appended a :443 server block to /opt/gitea/nginx.conf (serves :80+:443, NO redirect per session-0061), added 443:443 + cert mount to docker-compose.yml, validated nginx -t in a throwaway container, recreated nginx only (gitea container untouched, up 6 days).
• Firewall: created gitea-cf-web (:80,443 from CF ranges, tag gitea), deleted world-open allow-web. Kept allow-gitea-ssh (:2222 grey). Direct :80/:443 now time out; git via CF = 200; SSH :2222 OK.
• ALL FOUR origins now strict-ready + firewall-locked: rfc(ohm), rfc-ppe, wiggle-snip-ppe, git. Only the operator Cloudflare flip remains.

Final state (session close, 2026-06-07)

Estate cutover to Cloudflare Full (strict) — COMPLETE. The operator flipped the wiggleverse.org zone to Full (strict) + Always Use HTTPS once all origins were ready. Verified end-to-end: all 4 hosts https=200 (not 525/526 → strict validates each origin cert), http→301→https (Always Use HTTPS), direct origin access times out, git SSH :2222 OK, app /api/health + git API 200.

Origin	:443 Origin-CA	Direct access disabled
rfc (ohm-rfc-app)	✓	✓
rfc-ppe	✓	✓
wiggle-snip-ppe	✓ (this session)	✓
git (gitea)	✓ (this session)	✓

Shipped (flotilla-core, all merged to main): v0.9.0 edge-tls policy doc + SPEC inv 7 (PR#3) → v0.10.0 enforce as deploy phase 9 (PR#4) → v0.10.1 TLS-only fix (PR#5). 460 tests green. Plus app.json + CLAUDE.md registration. Shipped (engineering): infra/deploy-gitea-on-gcp.md runbook (PR#16).

Deferred-decisions report: the one logged call — flotilla enforcement as verify-and-fail at deploy + provision-default (vs active-repair) — was surfaced and confirmed by the operator via AskUserQuestion (chose verify-and-fail + provision-default). No unresolved low-confidence calls remain.

Open follow-ups (next session):

1. Provisioning-side enforcement — launch-app/provision-vm always creates CF-only firewall + Origin-CA cert + nginx :80+:443 (the provision-default half).
2. gitea VM lacks the Secret Manager OAuth scope → cert can't self-heal on rotation; re-stream via operator creds (runbook) or stop/start with --scopes=cloud-platform.

Next /goal

/goal Implement the provisioning-side edge enforcement: make launch-app/provision-vm always create the CF-only firewall + deliver the Origin-CA *.wiggleverse.org cert + nginx serving :80+:443 (no redirect), so new deployments are born edge-compliant (provision-default half of the verify-and-fail approach already enforced by flotilla-core phase 9). Also fix the gitea VM Secret Manager OAuth scope for cert rotation.