session-history/flotilla-core/0001/SESSION-0001.0-TRANSCRIPT-2026-06-06T22-53--2026-06-07T06-47.md

# Session 0001.0 — Transcript

> App: flotilla-core
> Type: coding
> Start: 2026-06-06T22-53 (PST) · End: 2026-06-07T06-47 (PST)
> Status: **FINALIZED.**

## Launch prompt

> git.wiggleverse.org is down. To deploy rfc-app ppe, I followed these
> instructions. Taking TLS from flexible to full allowed
> rfc-app-ppe.wiggleverse.org to receive connections, but git.wiggleverse.org
> went down. What should we do across all deployments to ensure they all work?

(Also: flotilla-core had no `app.json` — created one this session to claim ID 0001.)

## Deferred decisions

_Autonomous-mode low-confidence calls the driver made and would have
liked operator input on. Appended as the session runs; surfaced at
finalize. Empty if none._

## Plan

**Incident:** Operator flipped the Cloudflare `wiggleverse.org` zone SSL/TLS mode
from **Flexible → Full** to get `rfc-app-ppe.wiggleverse.org` serving. That made
ppe work but knocked `git.wiggleverse.org` "down". Question: what to do across
ALL deployments so they all work.

**Root-cause hypothesis:** Cloudflare SSL/TLS encryption mode is a **zone-level**
setting — every proxied host under wiggleverse.org shares it. Flexible (CF→origin
over plain HTTP:80) vs Full/Full-strict (CF→origin over HTTPS:443). Hosts whose
origin nginx only serves :80 break under Full; hosts that redirect 80→443 or only
serve :443 break under Flexible. The fix can't be per-host via the mode toggle —
it must be: every origin serves HTTPS:443 with the CF Origin-CA wildcard cert,
then set the zone to Full (strict). This is exactly flotilla's
wildcard-tls-cutover design (docs/wildcard-tls-cutover-runbook.md).

**Steps:**
1. Probe every public host under wiggleverse.org → which are green / which throw a
   CF origin error (521/522/525/526) under the *current* mode.
2. Determine the current zone SSL/TLS mode (CF API or operator).
3. For each origin, confirm whether nginx serves :443 with the wildcard Origin-CA
   cert (the flotilla secret-file delivery).
4. Bring every lagging origin up to HTTPS:443 + wildcard cert; set zone Full(strict).
5. Encode the invariant in flotilla docs/spec so a future deploy can't break siblings.

## Live cutover progress (2026-06-06)

**Diagnosis (evidence):** Cloudflare SSL/TLS mode is zone-wide. Operator has a
per-host Configuration Rule `rfc*.wiggleverse.org → Full`; zone default = Flexible.
DNS (CF API): proxied web origins = git, rfc, rfc-ppe, wiggle-snip-ppe; git-ssh is
grey (direct :2222). Mapped origins to VMs:
- rfc.wiggleverse.org = ohm-rfc-app (wiggleverse-ohm), 136.116.40.66
- rfc-ppe = rfc-app-ppe (rfc-app-ppe proj), 35.253.21.170
- wiggle-snip-ppe = wiggle-snip-ppe (wiggle-snip proj), 35.255.99.91
- git = 34.55.46.221 (Gitea infra, NOT in my gcloud projects — out of reach)

**Origin :443 state probed:**
- rfc (ohm): Origin-CA cert ✓, :80→:443 redirect ✓, but :443 was WORLD-OPEN.
- rfc-ppe: firewalled CF-only ✓ (presumed Origin-CA).
- wiggle-snip-ppe: serves :443 but with a SELF-SIGNED cert
  (CN=ppe.wiggle-snip.wiggleverse.org) at /etc/ssl/wiggle-snip-ppe/origin.crt —
  works only because served via Flexible (:80). NOT strict-ready.
- git: :80 only, :443 closed.

**DONE — firewall lockdown of ohm-rfc-app (the one reachable non-compliant origin):**
- Created `ohm-rfc-app-cf-web` (allow tcp:80,443 from the 15 Cloudflare IPv4
  ranges, target tags http-server,https-server) in wiggleverse-ohm.
- Deleted `default-allow-http` + `default-allow-https` (0.0.0.0/0).
- Verified: rfc.wiggleverse.org + /api/health stay 200 via CF; direct :80/:443 to
  136.116.40.66 now TIME OUT. IAP-SSH (:22) untouched.
- rfc-app-ppe + wiggle-snip already had the CF-only `*-cf-web` pattern.

**Cannot do (handoff): ** zone flip to Full(strict) + delete rfc* rule (CF token is
DNS-scoped); git origin :443 + Origin-CA + firewall (no access to Gitea VM);
wiggle-snip-ppe Origin-CA cert delivery (needed before strict flip).

## Deferred decisions
- Implementing flotilla enforcement as VERIFY-AND-FAIL at deploy + enforce at
  provisioning (launch-app/provision-vm), per SPEC invariant 3 (flotilla is the
  deploy actuator, not the provisioner). Confirming mechanism with operator.

## Enforcement shipped (flotilla-core v0.10.0, PR #4)

flotilla deploy now has a 10th phase "verify edge" (finalize -> 10): after
/api/health, SSH to origin and deterministically check nginx :443 + Cloudflare
Origin-CA cert (not self-signed) + :80->:443 redirect. Any violation FAILS the
deploy. New flotilla_core/edge.py (pure evaluate_edge + parse + gather, mirrors
health.py). --skip-edge-check break-glass. SPEC §8.1/§3-inv7 + policy doc updated.
461 tests pass. This is the verify-and-fail half (operator chose verify-and-fail +
provision-default).

## Outstanding handoffs
1. Zone flip: Cloudflare wiggleverse.org SSL/TLS -> Full (strict) zone-wide, then
   delete the rfc* Configuration Rule. (My CF token is DNS-scoped; needs dashboard
   or a Zone-Settings:Edit token.) DO NOT flip until git + wiggle-snip-ppe are
   strict-ready, else they break.
2. git.wiggleverse.org origin (34.55.46.221, Gitea infra, not in my gcloud
   projects): add :443 + Origin-CA cert + :80->:443; firewall :80,443 to CF ranges
   (keep git-ssh :2222 grey/direct).
3. wiggle-snip-ppe: replace self-signed /etc/ssl/wiggle-snip-ppe/origin.crt with
   the Cloudflare Origin-CA *.wiggleverse.org cert (via flotilla wildcard-tls).
4. Provisioning half: launch-app/provision-vm must always deliver Origin-CA cert +
   CF-only firewall + nginx :443/redirect (separate repo). Until then a new
   non-compliant origin fails its first deploy (by design).

## wiggle-snip-ppe cert fix (done) + git note
- Copied wildcard cert into wiggle-snip project, bound secret-files, delivered on
  VM, repointed nginx vhost (/etc/ssl/certs/wiggleverse-wildcard.crt), reloaded.
- :443 now serves Cloudflare Origin-CA (valid 2041); still 200 through CF.
- :80->:443 redirect deliberately NOT added yet (would loop under Flexible);
  add at zone-flip time.
- NOTE: full `flotilla deploy wiggle-snip` is blocked — pin read returns HTTP 521
  from git.wiggleverse.org (git origin unreachable from CF again; SSH/2222 fine).
- git.wiggleverse.org is 521 (down on the CF HTTP/HTTPS path) as of ~23:xx PST.

## Correction (v0.10.1, PR #5) + git outage insight
- The wiggle-snip vhost (and launch-app provision-vm / session 0061) document the
  REAL convention: nginx serves :80+:443 with NO origin redirect; Cloudflare
  "Always Use HTTPS" does the http->https upgrade (a hard origin redirect loops
  under Flexible). My 0.10.0 phase-9 check wrongly REQUIRED an origin redirect ->
  would have failed correctly-configured hosts. Fixed: evaluate_edge now verifies
  TLS only (serves :443 with Origin-CA cert). SPEC + policy doc corrected.
- Did NOT add a :80->:443 redirect to wiggle-snip (would loop under Flexible).
- GIT OUTAGE ROOT CAUSE confirmed: git is HTTP-only on :80, so ANY move of the
  zone toward Full 521s it; reverting to Flexible restores it. This is exactly why
  git must get :443 + Origin-CA BEFORE the zone can go Full(strict). Until then,
  keep zone=Flexible + the rfc* Full rule.
- Cert/key live at wiggleverse/wiggleverse-wildcard-{fullchain,privkey}.

## git strict-readiness DONE (the last blocker)
- git = `gitea` VM in project `wiggleverse` (Docker: gitea + nginx:alpine containers).
- nginx container was :80-only. Added :443: delivered wildcard Origin-CA cert to
  /opt/gitea/certs (VM SA lacked Secret Manager OAuth scope, so streamed from
  Secret Manager via operator creds over the SSH pipe — no local disk, no echo),
  appended a :443 server block to /opt/gitea/nginx.conf (serves :80+:443, NO
  redirect per session-0061), added 443:443 + cert mount to docker-compose.yml,
  validated nginx -t in a throwaway container, recreated nginx only (gitea
  container untouched, up 6 days).
- Firewall: created gitea-cf-web (:80,443 from CF ranges, tag gitea), deleted
  world-open allow-web. Kept allow-gitea-ssh (:2222 grey). Direct :80/:443 now
  time out; git via CF = 200; SSH :2222 OK.
- ALL FOUR origins now strict-ready + firewall-locked: rfc(ohm), rfc-ppe,
  wiggle-snip-ppe, git. Only the operator Cloudflare flip remains.

## Final state (session close, 2026-06-07)

**Estate cutover to Cloudflare Full (strict) — COMPLETE.** The operator flipped the
`wiggleverse.org` zone to Full (strict) + Always Use HTTPS once all origins were
ready. Verified end-to-end: all 4 hosts `https=200` (not 525/526 → strict
validates each origin cert), `http→301→https` (Always Use HTTPS), direct origin
access times out, git SSH `:2222` OK, app `/api/health` + git API `200`.

| Origin | :443 Origin-CA | Direct access disabled |
| --- | --- | --- |
| rfc (ohm-rfc-app) | ✓ | ✓ |
| rfc-ppe | ✓ | ✓ |
| wiggle-snip-ppe | ✓ (this session) | ✓ |
| git (gitea) | ✓ (this session) | ✓ |

**Shipped (flotilla-core, all merged to main):** v0.9.0 edge-tls policy doc + SPEC
inv 7 (PR#3) → v0.10.0 enforce as deploy phase 9 (PR#4) → v0.10.1 TLS-only fix
(PR#5). 460 tests green. Plus `app.json` + `CLAUDE.md` registration.
**Shipped (engineering):** `infra/deploy-gitea-on-gcp.md` runbook (PR#16).

**Deferred-decisions report:** the one logged call — flotilla enforcement as
*verify-and-fail at deploy + provision-default* (vs active-repair) — was surfaced
and **confirmed by the operator** via AskUserQuestion (chose verify-and-fail +
provision-default). No unresolved low-confidence calls remain.

**Open follow-ups (next session):**
1. Provisioning-side enforcement — launch-app/provision-vm always creates CF-only
   firewall + Origin-CA cert + nginx :80+:443 (the provision-default half).
2. gitea VM lacks the Secret Manager OAuth scope → cert can't self-heal on
   rotation; re-stream via operator creds (runbook) or stop/start with
   `--scopes=cloud-platform`.

## Next /goal
```
/goal Implement the provisioning-side edge enforcement: make launch-app/provision-vm always create the CF-only firewall + deliver the Origin-CA *.wiggleverse.org cert + nginx serving :80+:443 (no redirect), so new deployments are born edge-compliant (provision-default half of the verify-and-fail approach already enforced by flotilla-core phase 9). Also fix the gitea VM Secret Manager OAuth scope for cert rotation.
```