From 757412ef2af2e7955700291ea0dec70d85cb8945 Mon Sep 17 00:00:00 2001 From: Ben Stull Date: Fri, 12 Jun 2026 00:48:44 -0700 Subject: [PATCH] docs(products): image pipeline ops + seams (DOC-1/DOC-4); overlay bucket config; v0.7.0 Co-Authored-By: Claude Fable 5 --- VERSION | 2 +- deployment.toml | 2 + docs/OPERATIONS.md | 94 ++++++++++++++++++++++++++++++++++++++ docs/products-domain.md | 94 ++++++++++++++++++++++++++++++++++---- frontend/package-lock.json | 4 +- frontend/package.json | 2 +- 6 files changed, 184 insertions(+), 14 deletions(-) diff --git a/VERSION b/VERSION index a918a2a..faef31a 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.6.0 +0.7.0 diff --git a/deployment.toml b/deployment.toml index 33a4939..865f27c 100644 --- a/deployment.toml +++ b/deployment.toml @@ -35,6 +35,8 @@ ECOMM_SMTP_HOST = "smtp.gmail.com" ECOMM_SMTP_PORT = "587" ECOMM_SMTP_USER = "ben.stull@wiggleverse.org" ECOMM_SMTP_FROM = "ecomm " +ECOMM_OBJECTSTORE_KIND = "gcs" +ECOMM_OBJECTSTORE_BUCKET = "wiggleverse-ecomm-ppe-media" [secrets] # REFERENCES only — never bytes (§8.3) ECOMM_SESSION_SECRET = "wiggleverse-ecomm/ecomm-ppe-session-secret" diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index 34b09bb..aa1f414 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -43,6 +43,8 @@ never file names, URLs, catalog content, or secret bytes. | TEL-2 `import_run_completed` | apply transaction commits | `run_id, storefront_id, added, updated, errored, duration_ms` | | TEL-3 `catalog_exported` | export stream completes | `storefront_id, status_filter, product_count, duration_ms` | | TEL-6 `import_apply_failed` | apply transaction aborts unexpectedly | `draft_id, storefront_id, error_class` | +| TEL-4 `image_phase_completed` | run's fetch task finishes | `run_id, fetched, rejected, failed, duration_ms` | +| TEL-5 `image_phase_recovered` | startup recovery resumes a run | `run_id, pending_resumed` | ### Export (PUC-9, SLICE-6) @@ -56,6 +58,98 @@ re-importing an unmodified export previews as all-unchanged with the import action disabled (PUC-10). TEL-3 (`catalog_exported`) is emitted once the stream completes — counts and duration only, never catalog content. +### Images (SLICE-7) + +After a merchant confirms an import, the app transitions the run to +`fetching_images` and processes image URLs via a bounded thread pool +(4 workers). Each image goes through the SSRF guard, is decoded, and up to +four WebP renditions (`original`, `thumb`, `card`, `detail`) are written to +the objectstore under `product-images/`. Progress is tracked per run in +`image_progress` / `image_counts` / `image_outcomes` and visible in the +import history detail panel (PUC-8). TEL-4 (`image_phase_completed`) fires +when the phase finishes; startup recovery emits TEL-5 (`image_phase_recovered`) +when it resumes a stuck run. + +#### `provision-bucket` — one-time gesture per environment (SLICE-7 PPE deploy) + +Before the first SLICE-7 deploy, run the `provision-bucket` skill from the +engineering `launch-app` suite. This gesture: + +1. Creates the GCS bucket `wiggleverse-ecomm-ppe-media` in the + `wiggleverse-ecomm` GCP project. +2. Grants the PPE VM's service account `roles/storage.objectAdmin` on the + bucket. +3. Sets an `import-drafts/` lifecycle rule (forward-compat for the draft-blob + migration deferred past SLICE-7). + +After provisioning, `deployment.toml`'s `[overlay]` already carries: + +``` +ECOMM_OBJECTSTORE_KIND = "gcs" +ECOMM_OBJECTSTORE_BUCKET = "wiggleverse-ecomm-ppe-media" +``` + +so the next `flotilla deploy` picks them up automatically. GCS auth is the +VM service-account ADC — no credential bytes needed. + +### RB-3 — image phase stuck + +Triggered by ALR-3 (run in `fetching_images` > 2 h, or the same run recovered +≥ 3×). + +1. **Identify the stuck run.** The import history (PUC-8, `GET + /api/products/imports/runs`) lists runs by status; look for a run with + `status="fetching_images"` that hasn't advanced. + + ``` + journalctl -u ecomm.service | grep image_phase + ``` + +2. **Restart to trigger recovery.** A process restart causes the startup + recovery scan to re-enqueue pending images for any run still in + `fetching_images` (TEL-5 confirms the resumption). Restarts are safe — + per-image claims are idempotent; already-fetched images are skipped. + + ``` + systemctl restart ecomm.service + ``` + +3. **If a specific image URL is the cause.** The `image_outcomes` field on + the run records per-image rejection reasons (SSRF block, resolution + rejection, network error, etc.). The merchant can update or remove the + offending URL and re-import. + +4. **File a bug** on `wiggleverse/wiggleverse-ecomm` with the `run_id` and + the `image_outcomes` content from the log. + +### ALR-3 — the log-based alert for stuck image phases + +Run once per environment, at the SLICE-7 PPE deploy. Mirrors ALR-2's +approach — a log metric over the stuck/recovered signal, the existing email +channel, and an alert policy in project `wiggleverse-ecomm`. + +``` +# Select the deployment's gcloud config for this one process (handbook §8.4). +export CLOUDSDK_ACTIVE_CONFIG_NAME=wiggleverse-ecomm + +# Log-based metric counting image-phase events (TEL-4/TEL-5 — stuck/recovered). +gcloud logging metrics create ecomm_image_phase_events \ + --description="ecomm TEL-4/TEL-5 image phase completed/recovered (SD-0002 ALR-3)" \ + --log-filter='resource.type="gce_instance" AND (jsonPayload.message:"image_phase_completed" OR jsonPayload.message:"image_phase_recovered" OR textPayload:"image_phase_completed" OR textPayload:"image_phase_recovered")' +``` + +Then attach an alert policy — **operator email channel, threshold any event > +0 within a 2-hour window, severity notify-only**. List channels first: + +``` +# Find the operator email channel's id for the policy. +gcloud beta monitoring channels list +``` + +Create the alert policy (Cloud Console or `gcloud alpha monitoring policies +create`); condition: `fetching_images` duration > 2 h **or** the same +`run_id` appears in TEL-5 ≥ 3 times. + ### RB-2 — import apply failed Triggered by ALR-2 (any TEL-6 event). The apply raised mid-transaction and diff --git a/docs/products-domain.md b/docs/products-domain.md index 81c5185..8a38a09 100644 --- a/docs/products-domain.md +++ b/docs/products-domain.md @@ -112,20 +112,94 @@ same through the UI (export download → re-upload → all-unchanged preview, im disabled). Numeric/`Decimal` fields — the property test's deliberate blind spot (string-form vs value-identity) — get explicit round-trip unit tests. +## Image pipeline (SLICE-7) + +### `platform/objectstore` + +A two-adapter port (`backend/app/platform/objectstore/`): + +- `local.py` — writes blobs under a configurable local directory; used in + tests and local dev (`ECOMM_OBJECTSTORE_KIND=local`). +- `gcs.py` — wraps `google-cloud-storage`; bucket name comes from + `ECOMM_OBJECTSTORE_BUCKET` (`ECOMM_OBJECTSTORE_KIND=gcs`). Auth is the + VM's service-account ADC — no credential bytes in the overlay or secrets. + +Object keys for product images follow the prefix `product-images/` (the +`provision-bucket` gesture also sets a lifecycle rule on `import-drafts/` for +forward-compat, even though the draft blob remains BYTEA this slice — see +seams below). + +### `platform/images` + +`backend/app/platform/images.py` decodes an uploaded or fetched image byte +string and produces up to four renditions stored in the objectstore: + +- **`original`** — stored verbatim after the resolution bar passes. +- **`thumb`** — 150 px on the shorter side, WebP. +- **`card`** — 400 px on the shorter side, WebP. +- **`detail`** — 800 px on the shorter side, WebP. + +**Resolution bar (`MIN_IMAGE_SHORT_SIDE = 500`):** images whose shorter side +is below 500 px are rejected (Q-3). This is the only hard gate; oversized +images are scaled down without rejection. All renditions share the same +objectstore key prefix (`product-images//`). + +### `domains/products/imagefetch.py` — the fetch phase + +After `confirm_draft` commits, the run enters `fetching_images` (if it has +any pending image URLs) and a `ThreadPoolExecutor(max_workers=4)` processes +them concurrently: + +- **SSRF guard (INV-18, extended):** only `http`/`https` schemes are + permitted; the resolved IP must be public (RFC-1918, loopback, link-local, + and multicast ranges are blocked); redirects are followed only within the + same guard — a redirect to a private IP is rejected mid-chain. +- **Bounds:** ≤ 20 MB response body, ≤ 30 s per fetch. +- **Per-image idempotent claim:** each `CatalogImage` row is claimed before + fetch; a restart that finds a row already `claimed` skips it — making the + phase resumable. +- **Startup recovery scan:** on process start the app scans for runs stuck in + `fetching_images` and re-enqueues their pending images (emits TEL-5). This + covers crash/restart scenarios without a separate scheduler. +- **Progress tracking:** the run row carries `image_progress`, + `image_counts`, and `image_outcomes` (updated per image); these fields feed + the run-detail UI panel. + +On completion the phase sets the run status to `complete` and emits TEL-4. + +### Image-serving route + +`GET /api/products/images/{id}/{rendition}` — storefront-authorized, +streams the objectstore blob. The response sets +`Cache-Control: public, immutable, max-age=31536000` (the object key encodes +the image id, so the URL is stable for the lifetime of the image). INV-16: +an image belongs to exactly one storefront; the route enforces storefront +scope before reading from the objectstore. + +### Hosted-URL recognition and INV-12 over images + +`domains/products/hosted.py` provides a predicate that recognises whether an +image URL is already served by this app (i.e., a `GET /api/products/images/…` +URL at the current `APP_URL`). The diff pre-pass `_resolve_hosted_images` +uses it to convert hosted URLs back to their `CatalogImage` FK before hashing +the diff fingerprint — this closes the INV-12 round-trip guarantee over +image columns (a re-imported export previews as all-unchanged even when +images are hosted here). + ## Named seams (what later slices replace) -- **`import_draft.file_bytes` → objectstore key (SLICE-7).** The upload - currently lives as BYTEA on the draft row (`0002_products.sql`); SLICE-7 - moves the bytes to object storage and stores a key. +- **`import_draft.file_bytes` remains BYTEA (deferred past SLICE-7).** Moving + the draft blob to object storage was scoped out of SLICE-7. The media + bucket holds only `product-images/` objects this slice; the `provision-bucket` + script sets the `import-drafts/` lifecycle rule for forward-compat so the + migration will be clean when it lands. - **`codec.detect_dialect` → Shopify (SLICE-8).** Today it always returns `"canonical"`; SLICE-8 recognizes Shopify's exact header set here (INV-17). -- **Run-status complete shortcut → `fetching_images` (SLICE-7).** - `confirm_draft` inserts the run with `status="complete"` directly; SLICE-7 - inserts it as `fetching_images` and hands off to the image-fetch task. - Relatedly, image rows are created with the schema default - `status='pending'` **today and stay pending** — placeholder behavior until - SLICE-7's fetch phase (the run-detail payload already carries the stable - `image_progress` / `image_outcomes` shape, zeroed). +- **Run-status complete shortcut → shipped in SLICE-7.** + `confirm_draft` now inserts the run as `applying`, then transitions to + `fetching_images` (if the run has pending image URLs) or directly to + `complete`. The fetch phase fills `image_progress` / `image_counts` / + `image_outcomes` and sets `complete` when done. ## Test map diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 4a10ad6..cb4c6d9 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "wiggleverse-ecomm-frontend", - "version": "0.6.0", + "version": "0.7.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "wiggleverse-ecomm-frontend", - "version": "0.6.0", + "version": "0.7.0", "dependencies": { "react": "^18.3.1", "react-dom": "^18.3.1" diff --git a/frontend/package.json b/frontend/package.json index a351174..25ee4f9 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,7 +1,7 @@ { "name": "wiggleverse-ecomm-frontend", "private": true, - "version": "0.6.0", + "version": "0.7.0", "type": "module", "scripts": { "dev": "vite",