From 2606fbf826b787d8743637527fb4257b71ea149e Mon Sep 17 00:00:00 2001 From: Ben Stull Date: Thu, 11 Jun 2026 17:00:13 -0700 Subject: [PATCH] docs(products): operator guide (DOC-1) + domain notes (DOC-4); version 0.5.0 Co-Authored-By: Claude Fable 5 --- VERSION | 2 +- docs/OPERATIONS.md | 111 ++++++++++++++++++++++++++++++++++++++ docs/products-domain.md | 116 ++++++++++++++++++++++++++++++++++++++++ frontend/package.json | 2 +- 4 files changed, 229 insertions(+), 2 deletions(-) create mode 100644 docs/OPERATIONS.md create mode 100644 docs/products-domain.md diff --git a/VERSION b/VERSION index 1d0ba9e..8f0916f 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.4.0 +0.5.0 diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md new file mode 100644 index 0000000..ca711c4 --- /dev/null +++ b/docs/OPERATIONS.md @@ -0,0 +1,111 @@ +# Operating ecomm + +The framework-repo operator guide (SD-0002 DOC-1), started at SLICE-5. It covers +the app's operational surface — telemetry, runbooks, alert gestures, the E2E +gate. Per-deployment mechanics (deploy, secrets, VM access) live in the +deployment's flotilla docs and `deployment.toml`; environment bring-up is in +[`BOOTSTRAP.md`](./BOOTSTRAP.md). + +## Products import/export ops (SD-0002, SLICE-5) + +The §6.4 surface: a merchant uploads a catalog CSV (`POST +/api/products/imports`), the app validates it and stores an **import draft** +with a full preview (adds / updates / unchanged / errors), the merchant +confirms or cancels at the preview gate, and a confirm applies the previewed +diff as one **import run** recorded in the history (`/api/products/imports/runs`). + +Caps and behavior to know (all enforced in code, not config): + +- **Caps (INV-18):** ≤ 5,000 data rows and ≤ 10 MB per file — + `MAX_DATA_ROWS` / `MAX_FILE_BYTES` in `backend/app/domains/products/models.py`, + enforced in `backend/app/domains/products/codec.py` (file-level rejection) + plus a 413 `file_too_large` guard in the BFF (`backend/app/main.py`). +- **Draft expiry:** ~1 hour (`expires_at = now() + interval '1 hour'`). Cleanup + is a lazy sweep — expired drafts are deleted on the next upload and on any + access to an expired draft; there is no background job to babysit. +- **Upsert-only (INV-10):** an import adds and updates, never deletes. Catalog + products/variants/images absent from the file are untouched. +- **One-transaction apply (INV-11):** a confirm applies the whole previewed + diff in a single DB transaction — it lands completely or not at all. + +### Telemetry + +Structured JSON events on the `ecomm.telemetry` logger +(`backend/app/platform/telemetry.py`), one JSON object per line, emitted from +`backend/app/domains/products/service.py`. The app's `ecomm.*` log handler +writes to the process's stderr, which journald captures on the VM (and Cloud +Logging where the agent ships it). Events carry counts and durations only — +never file names, URLs, catalog content, or secret bytes. + +| Event | Trigger | Payload fields | +| --- | --- | --- | +| TEL-1 `import_draft_created` | validation completes, draft stored | `storefront_id, dialect, row_count, adds, updates, unchanged, errors, unknown_columns_count, duration_ms` | +| TEL-2 `import_run_completed` | apply transaction commits | `run_id, storefront_id, added, updated, errored, duration_ms` | +| TEL-6 `import_apply_failed` | apply transaction aborts unexpectedly | `draft_id, storefront_id, error_class` | + +### RB-2 — import apply failed + +Triggered by ALR-2 (any TEL-6 event). The apply raised mid-transaction and +rolled back. + +1. **Locate the failure.** On the VM, filter the journal for the event and note + the `draft_id`, `storefront_id`, and `error_class`: + + ``` + journalctl -u ecomm.service | grep import_apply_failed + ``` + +2. **Confirm the rollback held (INV-11).** The apply is one transaction, so a + failure leaves the catalog exactly as it was: the storefront's runs history + (`GET /api/products/imports/runs`) shows **no new run**, and the products + summary (`GET /api/products/summary`) shows an unchanged `product_count`. +3. **The merchant's draft is intact.** A failed apply does not consume the + draft — the merchant can retry confirm, or re-upload if the draft has since + expired (~1 h). Advise accordingly. +4. **File a bug** on `wiggleverse/wiggleverse-ecomm` with the `error_class` + and the surrounding log context (the traceback is in the app log next to + the event). + +### ALR-2 — the log-based alert (one gesture per environment) + +Run once per environment, at this slice's PPE deploy. This is an ad hoc op on +the existing GCP project (`wiggleverse-ecomm`), not a provisioning gesture. + +``` +# Select the deployment's gcloud config for this one process (handbook §8.4). +export CLOUDSDK_ACTIVE_CONFIG_NAME=wiggleverse-ecomm + +# Log-based metric counting import-apply failures (TEL-6). +gcloud logging metrics create ecomm_import_apply_failed \ + --description="ecomm TEL-6 import_apply_failed events (SD-0002 ALR-2)" \ + --log-filter='resource.type="gce_instance" AND jsonPayload.message:"import_apply_failed" OR textPayload:"import_apply_failed"' +``` + +Then attach an alert policy to the metric — **operator email channel, threshold +any event > 0 in 5 minutes, severity notify-only** (pre-v1: no paging). The +policy is created in the Cloud Console or with +`gcloud alpha monitoring policies create`; the exact command depends on the +notification-channel id, so list channels first: + +``` +# Find the operator email channel's id for the policy. +gcloud beta monitoring channels list +``` + +### E2E browser suite + +- Lives at `e2e/` — Playwright, Chromium, four SLICE-5 scenarios + (preview/confirm happy path, actionable errors, file rejection, cancel). +- Run with `bash scripts/e2e.sh`. The harness boots a **fresh `ecomm_e2e` + database** against the local compose Postgres and serves the built SPA from + the backend on **:8765** (the deployed topology), so it needs the dev + Postgres up (`scripts/dev.sh`). +- **Not in `scripts/check.sh` / CI yet** — the Gitea runner has no browsers + (the §10.6 machinery gap). Run it locally before merge, and against PPE per + the §9 pipeline (the PPE browser run is still manual this slice). + +## Cross-references + +- SLO and alert definitions: SD-0002 §9–§10 (content repo, + `wiggleverse-ecomm-content/specs/SD-0002-products-bulk-csv-import-export.md`). +- Import/diff engine internals: [`products-domain.md`](./products-domain.md). diff --git a/docs/products-domain.md b/docs/products-domain.md new file mode 100644 index 0000000..5210708 --- /dev/null +++ b/docs/products-domain.md @@ -0,0 +1,116 @@ +# products domain — developer notes + +DOC-4 (SD-0002 §11): the import/export spine as built in SLICE-5, extended by +SLICE-6–8. Operator-facing material is in [`OPERATIONS.md`](./OPERATIONS.md); +the spec is SD-0002 in the content repo. + +## Layout and pipeline + +`backend/app/domains/products/` is layered like the rest of the app +(main → domains → platform, enforced by import-linter): + +- `models.py` — canonical row model + the column registry. +- `codec.py` — bytes → `ParsedFile`; file-level gates only. +- `validate.py` — rows → `CanonicalProduct` blocks + per-row errors. +- `diff.py` — catalog × canonical products → apply plan + preview records. +- `repo.py` — SQL only: catalog snapshot, draft/run CRUD, apply primitives. + Never commits or rolls back. +- `service.py` — use-case orchestration; owns every transaction boundary and + emits the TEL events via `backend/app/platform/telemetry.py`. + +```mermaid +flowchart LR + subgraph validate_path [import_validate] + A[parse_csv] --> B[build_products] --> D[compute_diff] --> E[(import_draft)] + C[load_catalog] --> D + end + subgraph confirm_path [confirm_draft] + E --> F[re-derive: parse → build → diff] --> G{fingerprint match?} + G -- yes --> H[one-transaction apply
run + products + delete draft] + G -- no --> I[PreviewStale
draft kept] + end +``` + +Confirm re-derives everything from the draft's stored `file_bytes` against the +live catalog, then checks the fingerprint — so what lands is exactly what the +preview showed, or the confirm refuses (`preview_stale`). A confirm with no +adds and no updates refuses with `nothing_to_apply`. + +## Canonical model and blank-vs-absent + +`models.py` is the one model every dialect maps to (INV-17). `KNOWN_COLUMNS` +is the registry header detection, unknown-column warnings, and validation all +read; `CLEAR_DEFAULTS` holds the reset values for clearable fields +(`status`, `published`, `product_type`, `tags`). + +The §6.5.1 cell semantics, as implemented: + +- **Absent column** → the field never enters `fields{}` → untouched by diff + and apply (never a change). +- **Present-but-empty cell** → `fields[name] = None` (an explicit clear) → + resolved at diff time to its `CLEAR_DEFAULTS` entry, or NULL where none + exists. +- **The position exception:** a cleared `Variant Position` has no + `CLEAR_DEFAULTS` entry — `diff.resolved_variant_fields` resolves it to the + variant's 1-based file order within its product, never to NULL. + +Option names live both in `CanonicalProduct.option_names` (the values) and in +`fields{}` (the file-presence marker the diff needs for the absent-vs-clear +distinction). + +## Error granularity + +`validate.py` never raises on a row problem: every violation is recorded as a +merchant-language `RowError` and **poisons its whole product block** — the +product previews as `kind="error"` and is excluded from apply, while parsing +continues so one pass yields a complete accounting (BUC-1a). On apply, error +rows are recorded per line in `import_run_error`; `rows_errored` on the run is +the **error-row count**, while the preview's errors tile counts error +**products** — the two numbers legitimately differ. + +File-level problems (`not_csv`, `missing_required_column`, `too_many_rows`, +`file_too_large`) raise `FileRejected` in `codec.py` instead: no draft is +created. The BFF adds an early 413 for oversized uploads (`main.py`). + +## INV-11 mechanics + +`compute_diff` makes **one walk** that produces two views of the same +computation: the typed apply `plan` (resolved natives — `Decimal`, `bool`, +lists) that `confirm_draft` executes, and the JSON-safe preview `records` +stored as draft JSONB and served verbatim to the SPA. Because both derive from +the same walk they cannot diverge. The `fingerprint` is +`sha256(json.dumps(records, sort_keys=True, separators=(",", ":")))`; a +mismatch at confirm means the catalog drifted since preview → `PreviewStale` +(409 `preview_stale`, draft kept for re-validation). The whole apply — run row, +product/variant/image writes, error rows, draft delete — is one transaction; +any exception rolls it back and emits TEL-6. + +## Named seams (what later slices replace) + +- **`import_draft.file_bytes` → objectstore key (SLICE-7).** The upload + currently lives as BYTEA on the draft row (`0002_products.sql`); SLICE-7 + moves the bytes to object storage and stores a key. +- **`codec.detect_dialect` → Shopify (SLICE-8).** Today it always returns + `"canonical"`; SLICE-8 recognizes Shopify's exact header set here (INV-17). +- **Run-status complete shortcut → `fetching_images` (SLICE-7).** + `confirm_draft` inserts the run with `status="complete"` directly; SLICE-7 + inserts it as `fetching_images` and hands off to the image-fetch task. + Relatedly, image rows are created with the schema default + `status='pending'` **today and stay pending** — placeholder behavior until + SLICE-7's fetch phase (the run-detail payload already carries the stable + `image_progress` / `image_outcomes` shape, zeroed). + +## Test map + +| File | Covers | +| --- | --- | +| `backend/tests/test_products_codec.py` | file-level gates: parse, caps (INV-18), required columns, dialect | +| `backend/tests/test_products_validate.py` | every §6.5.1 row-error rule, one fixture each | +| `backend/tests/test_products_diff.py` | classification, blank-vs-absent, option matching, fingerprint | +| `backend/tests/test_products_service.py` | draft lifecycle: validate/preview/discard, expiry, TEL-1 | +| `backend/tests/test_products_invariants.py` | INV-10 (never deletes), INV-14 (storefront isolation), apply transactionality, TEL-6 | +| `backend/tests/test_products_endpoints.py` | §6.4 API scenarios + auth/storefront gates | +| `e2e/tests/import-preview-confirm.spec.ts` | happy path: upload → preview → confirm → history | +| `e2e/tests/import-errors.spec.ts` | actionable row errors at preview and on the run report | +| `e2e/tests/import-file-rejected.spec.ts` | file-level rejection, picker stays live, no trace | +| `e2e/tests/import-cancel.spec.ts` | cancel at preview leaves no trace | diff --git a/frontend/package.json b/frontend/package.json index 4103db0..9a77ee0 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,7 +1,7 @@ { "name": "wiggleverse-ecomm-frontend", "private": true, - "version": "0.4.0", + "version": "0.5.0", "type": "module", "scripts": { "dev": "vite",