docs(products): image pipeline ops + seams (DOC-1/DOC-4); overlay bucket config; v0.7.0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -35,6 +35,8 @@ ECOMM_SMTP_HOST = "smtp.gmail.com"
|
||||
ECOMM_SMTP_PORT = "587"
|
||||
ECOMM_SMTP_USER = "ben.stull@wiggleverse.org"
|
||||
ECOMM_SMTP_FROM = "ecomm <ben.stull@wiggleverse.org>"
|
||||
ECOMM_OBJECTSTORE_KIND = "gcs"
|
||||
ECOMM_OBJECTSTORE_BUCKET = "wiggleverse-ecomm-ppe-media"
|
||||
|
||||
[secrets] # REFERENCES only — never bytes (§8.3)
|
||||
ECOMM_SESSION_SECRET = "wiggleverse-ecomm/ecomm-ppe-session-secret"
|
||||
|
||||
@@ -43,6 +43,8 @@ never file names, URLs, catalog content, or secret bytes.
|
||||
| TEL-2 `import_run_completed` | apply transaction commits | `run_id, storefront_id, added, updated, errored, duration_ms` |
|
||||
| TEL-3 `catalog_exported` | export stream completes | `storefront_id, status_filter, product_count, duration_ms` |
|
||||
| TEL-6 `import_apply_failed` | apply transaction aborts unexpectedly | `draft_id, storefront_id, error_class` |
|
||||
| TEL-4 `image_phase_completed` | run's fetch task finishes | `run_id, fetched, rejected, failed, duration_ms` |
|
||||
| TEL-5 `image_phase_recovered` | startup recovery resumes a run | `run_id, pending_resumed` |
|
||||
|
||||
### Export (PUC-9, SLICE-6)
|
||||
|
||||
@@ -56,6 +58,98 @@ re-importing an unmodified export previews as all-unchanged with the import
|
||||
action disabled (PUC-10). TEL-3 (`catalog_exported`) is emitted once the stream
|
||||
completes — counts and duration only, never catalog content.
|
||||
|
||||
### Images (SLICE-7)
|
||||
|
||||
After a merchant confirms an import, the app transitions the run to
|
||||
`fetching_images` and processes image URLs via a bounded thread pool
|
||||
(4 workers). Each image goes through the SSRF guard, is decoded, and up to
|
||||
four WebP renditions (`original`, `thumb`, `card`, `detail`) are written to
|
||||
the objectstore under `product-images/`. Progress is tracked per run in
|
||||
`image_progress` / `image_counts` / `image_outcomes` and visible in the
|
||||
import history detail panel (PUC-8). TEL-4 (`image_phase_completed`) fires
|
||||
when the phase finishes; startup recovery emits TEL-5 (`image_phase_recovered`)
|
||||
when it resumes a stuck run.
|
||||
|
||||
#### `provision-bucket` — one-time gesture per environment (SLICE-7 PPE deploy)
|
||||
|
||||
Before the first SLICE-7 deploy, run the `provision-bucket` skill from the
|
||||
engineering `launch-app` suite. This gesture:
|
||||
|
||||
1. Creates the GCS bucket `wiggleverse-ecomm-ppe-media` in the
|
||||
`wiggleverse-ecomm` GCP project.
|
||||
2. Grants the PPE VM's service account `roles/storage.objectAdmin` on the
|
||||
bucket.
|
||||
3. Sets an `import-drafts/` lifecycle rule (forward-compat for the draft-blob
|
||||
migration deferred past SLICE-7).
|
||||
|
||||
After provisioning, `deployment.toml`'s `[overlay]` already carries:
|
||||
|
||||
```
|
||||
ECOMM_OBJECTSTORE_KIND = "gcs"
|
||||
ECOMM_OBJECTSTORE_BUCKET = "wiggleverse-ecomm-ppe-media"
|
||||
```
|
||||
|
||||
so the next `flotilla deploy` picks them up automatically. GCS auth is the
|
||||
VM service-account ADC — no credential bytes needed.
|
||||
|
||||
### RB-3 — image phase stuck
|
||||
|
||||
Triggered by ALR-3 (run in `fetching_images` > 2 h, or the same run recovered
|
||||
≥ 3×).
|
||||
|
||||
1. **Identify the stuck run.** The import history (PUC-8, `GET
|
||||
/api/products/imports/runs`) lists runs by status; look for a run with
|
||||
`status="fetching_images"` that hasn't advanced.
|
||||
|
||||
```
|
||||
journalctl -u ecomm.service | grep image_phase
|
||||
```
|
||||
|
||||
2. **Restart to trigger recovery.** A process restart causes the startup
|
||||
recovery scan to re-enqueue pending images for any run still in
|
||||
`fetching_images` (TEL-5 confirms the resumption). Restarts are safe —
|
||||
per-image claims are idempotent; already-fetched images are skipped.
|
||||
|
||||
```
|
||||
systemctl restart ecomm.service
|
||||
```
|
||||
|
||||
3. **If a specific image URL is the cause.** The `image_outcomes` field on
|
||||
the run records per-image rejection reasons (SSRF block, resolution
|
||||
rejection, network error, etc.). The merchant can update or remove the
|
||||
offending URL and re-import.
|
||||
|
||||
4. **File a bug** on `wiggleverse/wiggleverse-ecomm` with the `run_id` and
|
||||
the `image_outcomes` content from the log.
|
||||
|
||||
### ALR-3 — the log-based alert for stuck image phases
|
||||
|
||||
Run once per environment, at the SLICE-7 PPE deploy. Mirrors ALR-2's
|
||||
approach — a log metric over the stuck/recovered signal, the existing email
|
||||
channel, and an alert policy in project `wiggleverse-ecomm`.
|
||||
|
||||
```
|
||||
# Select the deployment's gcloud config for this one process (handbook §8.4).
|
||||
export CLOUDSDK_ACTIVE_CONFIG_NAME=wiggleverse-ecomm
|
||||
|
||||
# Log-based metric counting image-phase events (TEL-4/TEL-5 — stuck/recovered).
|
||||
gcloud logging metrics create ecomm_image_phase_events \
|
||||
--description="ecomm TEL-4/TEL-5 image phase completed/recovered (SD-0002 ALR-3)" \
|
||||
--log-filter='resource.type="gce_instance" AND (jsonPayload.message:"image_phase_completed" OR jsonPayload.message:"image_phase_recovered" OR textPayload:"image_phase_completed" OR textPayload:"image_phase_recovered")'
|
||||
```
|
||||
|
||||
Then attach an alert policy — **operator email channel, threshold any event >
|
||||
0 within a 2-hour window, severity notify-only**. List channels first:
|
||||
|
||||
```
|
||||
# Find the operator email channel's id for the policy.
|
||||
gcloud beta monitoring channels list
|
||||
```
|
||||
|
||||
Create the alert policy (Cloud Console or `gcloud alpha monitoring policies
|
||||
create`); condition: `fetching_images` duration > 2 h **or** the same
|
||||
`run_id` appears in TEL-5 ≥ 3 times.
|
||||
|
||||
### RB-2 — import apply failed
|
||||
|
||||
Triggered by ALR-2 (any TEL-6 event). The apply raised mid-transaction and
|
||||
|
||||
+84
-10
@@ -112,20 +112,94 @@ same through the UI (export download → re-upload → all-unchanged preview, im
|
||||
disabled). Numeric/`Decimal` fields — the property test's deliberate blind spot
|
||||
(string-form vs value-identity) — get explicit round-trip unit tests.
|
||||
|
||||
## Image pipeline (SLICE-7)
|
||||
|
||||
### `platform/objectstore`
|
||||
|
||||
A two-adapter port (`backend/app/platform/objectstore/`):
|
||||
|
||||
- `local.py` — writes blobs under a configurable local directory; used in
|
||||
tests and local dev (`ECOMM_OBJECTSTORE_KIND=local`).
|
||||
- `gcs.py` — wraps `google-cloud-storage`; bucket name comes from
|
||||
`ECOMM_OBJECTSTORE_BUCKET` (`ECOMM_OBJECTSTORE_KIND=gcs`). Auth is the
|
||||
VM's service-account ADC — no credential bytes in the overlay or secrets.
|
||||
|
||||
Object keys for product images follow the prefix `product-images/` (the
|
||||
`provision-bucket` gesture also sets a lifecycle rule on `import-drafts/` for
|
||||
forward-compat, even though the draft blob remains BYTEA this slice — see
|
||||
seams below).
|
||||
|
||||
### `platform/images`
|
||||
|
||||
`backend/app/platform/images.py` decodes an uploaded or fetched image byte
|
||||
string and produces up to four renditions stored in the objectstore:
|
||||
|
||||
- **`original`** — stored verbatim after the resolution bar passes.
|
||||
- **`thumb`** — 150 px on the shorter side, WebP.
|
||||
- **`card`** — 400 px on the shorter side, WebP.
|
||||
- **`detail`** — 800 px on the shorter side, WebP.
|
||||
|
||||
**Resolution bar (`MIN_IMAGE_SHORT_SIDE = 500`):** images whose shorter side
|
||||
is below 500 px are rejected (Q-3). This is the only hard gate; oversized
|
||||
images are scaled down without rejection. All renditions share the same
|
||||
objectstore key prefix (`product-images/<image_id>/`).
|
||||
|
||||
### `domains/products/imagefetch.py` — the fetch phase
|
||||
|
||||
After `confirm_draft` commits, the run enters `fetching_images` (if it has
|
||||
any pending image URLs) and a `ThreadPoolExecutor(max_workers=4)` processes
|
||||
them concurrently:
|
||||
|
||||
- **SSRF guard (INV-18, extended):** only `http`/`https` schemes are
|
||||
permitted; the resolved IP must be public (RFC-1918, loopback, link-local,
|
||||
and multicast ranges are blocked); redirects are followed only within the
|
||||
same guard — a redirect to a private IP is rejected mid-chain.
|
||||
- **Bounds:** ≤ 20 MB response body, ≤ 30 s per fetch.
|
||||
- **Per-image idempotent claim:** each `CatalogImage` row is claimed before
|
||||
fetch; a restart that finds a row already `claimed` skips it — making the
|
||||
phase resumable.
|
||||
- **Startup recovery scan:** on process start the app scans for runs stuck in
|
||||
`fetching_images` and re-enqueues their pending images (emits TEL-5). This
|
||||
covers crash/restart scenarios without a separate scheduler.
|
||||
- **Progress tracking:** the run row carries `image_progress`,
|
||||
`image_counts`, and `image_outcomes` (updated per image); these fields feed
|
||||
the run-detail UI panel.
|
||||
|
||||
On completion the phase sets the run status to `complete` and emits TEL-4.
|
||||
|
||||
### Image-serving route
|
||||
|
||||
`GET /api/products/images/{id}/{rendition}` — storefront-authorized,
|
||||
streams the objectstore blob. The response sets
|
||||
`Cache-Control: public, immutable, max-age=31536000` (the object key encodes
|
||||
the image id, so the URL is stable for the lifetime of the image). INV-16:
|
||||
an image belongs to exactly one storefront; the route enforces storefront
|
||||
scope before reading from the objectstore.
|
||||
|
||||
### Hosted-URL recognition and INV-12 over images
|
||||
|
||||
`domains/products/hosted.py` provides a predicate that recognises whether an
|
||||
image URL is already served by this app (i.e., a `GET /api/products/images/…`
|
||||
URL at the current `APP_URL`). The diff pre-pass `_resolve_hosted_images`
|
||||
uses it to convert hosted URLs back to their `CatalogImage` FK before hashing
|
||||
the diff fingerprint — this closes the INV-12 round-trip guarantee over
|
||||
image columns (a re-imported export previews as all-unchanged even when
|
||||
images are hosted here).
|
||||
|
||||
## Named seams (what later slices replace)
|
||||
|
||||
- **`import_draft.file_bytes` → objectstore key (SLICE-7).** The upload
|
||||
currently lives as BYTEA on the draft row (`0002_products.sql`); SLICE-7
|
||||
moves the bytes to object storage and stores a key.
|
||||
- **`import_draft.file_bytes` remains BYTEA (deferred past SLICE-7).** Moving
|
||||
the draft blob to object storage was scoped out of SLICE-7. The media
|
||||
bucket holds only `product-images/` objects this slice; the `provision-bucket`
|
||||
script sets the `import-drafts/` lifecycle rule for forward-compat so the
|
||||
migration will be clean when it lands.
|
||||
- **`codec.detect_dialect` → Shopify (SLICE-8).** Today it always returns
|
||||
`"canonical"`; SLICE-8 recognizes Shopify's exact header set here (INV-17).
|
||||
- **Run-status complete shortcut → `fetching_images` (SLICE-7).**
|
||||
`confirm_draft` inserts the run with `status="complete"` directly; SLICE-7
|
||||
inserts it as `fetching_images` and hands off to the image-fetch task.
|
||||
Relatedly, image rows are created with the schema default
|
||||
`status='pending'` **today and stay pending** — placeholder behavior until
|
||||
SLICE-7's fetch phase (the run-detail payload already carries the stable
|
||||
`image_progress` / `image_outcomes` shape, zeroed).
|
||||
- **Run-status complete shortcut → shipped in SLICE-7.**
|
||||
`confirm_draft` now inserts the run as `applying`, then transitions to
|
||||
`fetching_images` (if the run has pending image URLs) or directly to
|
||||
`complete`. The fetch phase fills `image_progress` / `image_counts` /
|
||||
`image_outcomes` and sets `complete` when done.
|
||||
|
||||
## Test map
|
||||
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "wiggleverse-ecomm-frontend",
|
||||
"version": "0.6.0",
|
||||
"version": "0.7.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "wiggleverse-ecomm-frontend",
|
||||
"version": "0.6.0",
|
||||
"version": "0.7.0",
|
||||
"dependencies": {
|
||||
"react": "^18.3.1",
|
||||
"react-dom": "^18.3.1"
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "wiggleverse-ecomm-frontend",
|
||||
"private": true,
|
||||
"version": "0.6.0",
|
||||
"version": "0.7.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
|
||||
Reference in New Issue
Block a user