docs(products): operator guide (DOC-1) + domain notes (DOC-4); version 0.5.0
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,111 @@
|
||||
# Operating ecomm
|
||||
|
||||
The framework-repo operator guide (SD-0002 DOC-1), started at SLICE-5. It covers
|
||||
the app's operational surface — telemetry, runbooks, alert gestures, the E2E
|
||||
gate. Per-deployment mechanics (deploy, secrets, VM access) live in the
|
||||
deployment's flotilla docs and `deployment.toml`; environment bring-up is in
|
||||
[`BOOTSTRAP.md`](./BOOTSTRAP.md).
|
||||
|
||||
## Products import/export ops (SD-0002, SLICE-5)
|
||||
|
||||
The §6.4 surface: a merchant uploads a catalog CSV (`POST
|
||||
/api/products/imports`), the app validates it and stores an **import draft**
|
||||
with a full preview (adds / updates / unchanged / errors), the merchant
|
||||
confirms or cancels at the preview gate, and a confirm applies the previewed
|
||||
diff as one **import run** recorded in the history (`/api/products/imports/runs`).
|
||||
|
||||
Caps and behavior to know (all enforced in code, not config):
|
||||
|
||||
- **Caps (INV-18):** ≤ 5,000 data rows and ≤ 10 MB per file —
|
||||
`MAX_DATA_ROWS` / `MAX_FILE_BYTES` in `backend/app/domains/products/models.py`,
|
||||
enforced in `backend/app/domains/products/codec.py` (file-level rejection)
|
||||
plus a 413 `file_too_large` guard in the BFF (`backend/app/main.py`).
|
||||
- **Draft expiry:** ~1 hour (`expires_at = now() + interval '1 hour'`). Cleanup
|
||||
is a lazy sweep — expired drafts are deleted on the next upload and on any
|
||||
access to an expired draft; there is no background job to babysit.
|
||||
- **Upsert-only (INV-10):** an import adds and updates, never deletes. Catalog
|
||||
products/variants/images absent from the file are untouched.
|
||||
- **One-transaction apply (INV-11):** a confirm applies the whole previewed
|
||||
diff in a single DB transaction — it lands completely or not at all.
|
||||
|
||||
### Telemetry
|
||||
|
||||
Structured JSON events on the `ecomm.telemetry` logger
|
||||
(`backend/app/platform/telemetry.py`), one JSON object per line, emitted from
|
||||
`backend/app/domains/products/service.py`. The app's `ecomm.*` log handler
|
||||
writes to the process's stderr, which journald captures on the VM (and Cloud
|
||||
Logging where the agent ships it). Events carry counts and durations only —
|
||||
never file names, URLs, catalog content, or secret bytes.
|
||||
|
||||
| Event | Trigger | Payload fields |
|
||||
| --- | --- | --- |
|
||||
| TEL-1 `import_draft_created` | validation completes, draft stored | `storefront_id, dialect, row_count, adds, updates, unchanged, errors, unknown_columns_count, duration_ms` |
|
||||
| TEL-2 `import_run_completed` | apply transaction commits | `run_id, storefront_id, added, updated, errored, duration_ms` |
|
||||
| TEL-6 `import_apply_failed` | apply transaction aborts unexpectedly | `draft_id, storefront_id, error_class` |
|
||||
|
||||
### RB-2 — import apply failed
|
||||
|
||||
Triggered by ALR-2 (any TEL-6 event). The apply raised mid-transaction and
|
||||
rolled back.
|
||||
|
||||
1. **Locate the failure.** On the VM, filter the journal for the event and note
|
||||
the `draft_id`, `storefront_id`, and `error_class`:
|
||||
|
||||
```
|
||||
journalctl -u ecomm.service | grep import_apply_failed
|
||||
```
|
||||
|
||||
2. **Confirm the rollback held (INV-11).** The apply is one transaction, so a
|
||||
failure leaves the catalog exactly as it was: the storefront's runs history
|
||||
(`GET /api/products/imports/runs`) shows **no new run**, and the products
|
||||
summary (`GET /api/products/summary`) shows an unchanged `product_count`.
|
||||
3. **The merchant's draft is intact.** A failed apply does not consume the
|
||||
draft — the merchant can retry confirm, or re-upload if the draft has since
|
||||
expired (~1 h). Advise accordingly.
|
||||
4. **File a bug** on `wiggleverse/wiggleverse-ecomm` with the `error_class`
|
||||
and the surrounding log context (the traceback is in the app log next to
|
||||
the event).
|
||||
|
||||
### ALR-2 — the log-based alert (one gesture per environment)
|
||||
|
||||
Run once per environment, at this slice's PPE deploy. This is an ad hoc op on
|
||||
the existing GCP project (`wiggleverse-ecomm`), not a provisioning gesture.
|
||||
|
||||
```
|
||||
# Select the deployment's gcloud config for this one process (handbook §8.4).
|
||||
export CLOUDSDK_ACTIVE_CONFIG_NAME=wiggleverse-ecomm
|
||||
|
||||
# Log-based metric counting import-apply failures (TEL-6).
|
||||
gcloud logging metrics create ecomm_import_apply_failed \
|
||||
--description="ecomm TEL-6 import_apply_failed events (SD-0002 ALR-2)" \
|
||||
--log-filter='resource.type="gce_instance" AND jsonPayload.message:"import_apply_failed" OR textPayload:"import_apply_failed"'
|
||||
```
|
||||
|
||||
Then attach an alert policy to the metric — **operator email channel, threshold
|
||||
any event > 0 in 5 minutes, severity notify-only** (pre-v1: no paging). The
|
||||
policy is created in the Cloud Console or with
|
||||
`gcloud alpha monitoring policies create`; the exact command depends on the
|
||||
notification-channel id, so list channels first:
|
||||
|
||||
```
|
||||
# Find the operator email channel's id for the policy.
|
||||
gcloud beta monitoring channels list
|
||||
```
|
||||
|
||||
### E2E browser suite
|
||||
|
||||
- Lives at `e2e/` — Playwright, Chromium, four SLICE-5 scenarios
|
||||
(preview/confirm happy path, actionable errors, file rejection, cancel).
|
||||
- Run with `bash scripts/e2e.sh`. The harness boots a **fresh `ecomm_e2e`
|
||||
database** against the local compose Postgres and serves the built SPA from
|
||||
the backend on **:8765** (the deployed topology), so it needs the dev
|
||||
Postgres up (`scripts/dev.sh`).
|
||||
- **Not in `scripts/check.sh` / CI yet** — the Gitea runner has no browsers
|
||||
(the §10.6 machinery gap). Run it locally before merge, and against PPE per
|
||||
the §9 pipeline (the PPE browser run is still manual this slice).
|
||||
|
||||
## Cross-references
|
||||
|
||||
- SLO and alert definitions: SD-0002 §9–§10 (content repo,
|
||||
`wiggleverse-ecomm-content/specs/SD-0002-products-bulk-csv-import-export.md`).
|
||||
- Import/diff engine internals: [`products-domain.md`](./products-domain.md).
|
||||
@@ -0,0 +1,116 @@
|
||||
# products domain — developer notes
|
||||
|
||||
DOC-4 (SD-0002 §11): the import/export spine as built in SLICE-5, extended by
|
||||
SLICE-6–8. Operator-facing material is in [`OPERATIONS.md`](./OPERATIONS.md);
|
||||
the spec is SD-0002 in the content repo.
|
||||
|
||||
## Layout and pipeline
|
||||
|
||||
`backend/app/domains/products/` is layered like the rest of the app
|
||||
(main → domains → platform, enforced by import-linter):
|
||||
|
||||
- `models.py` — canonical row model + the column registry.
|
||||
- `codec.py` — bytes → `ParsedFile`; file-level gates only.
|
||||
- `validate.py` — rows → `CanonicalProduct` blocks + per-row errors.
|
||||
- `diff.py` — catalog × canonical products → apply plan + preview records.
|
||||
- `repo.py` — SQL only: catalog snapshot, draft/run CRUD, apply primitives.
|
||||
Never commits or rolls back.
|
||||
- `service.py` — use-case orchestration; owns every transaction boundary and
|
||||
emits the TEL events via `backend/app/platform/telemetry.py`.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph validate_path [import_validate]
|
||||
A[parse_csv] --> B[build_products] --> D[compute_diff] --> E[(import_draft)]
|
||||
C[load_catalog] --> D
|
||||
end
|
||||
subgraph confirm_path [confirm_draft]
|
||||
E --> F[re-derive: parse → build → diff] --> G{fingerprint match?}
|
||||
G -- yes --> H[one-transaction apply<br/>run + products + delete draft]
|
||||
G -- no --> I[PreviewStale<br/>draft kept]
|
||||
end
|
||||
```
|
||||
|
||||
Confirm re-derives everything from the draft's stored `file_bytes` against the
|
||||
live catalog, then checks the fingerprint — so what lands is exactly what the
|
||||
preview showed, or the confirm refuses (`preview_stale`). A confirm with no
|
||||
adds and no updates refuses with `nothing_to_apply`.
|
||||
|
||||
## Canonical model and blank-vs-absent
|
||||
|
||||
`models.py` is the one model every dialect maps to (INV-17). `KNOWN_COLUMNS`
|
||||
is the registry header detection, unknown-column warnings, and validation all
|
||||
read; `CLEAR_DEFAULTS` holds the reset values for clearable fields
|
||||
(`status`, `published`, `product_type`, `tags`).
|
||||
|
||||
The §6.5.1 cell semantics, as implemented:
|
||||
|
||||
- **Absent column** → the field never enters `fields{}` → untouched by diff
|
||||
and apply (never a change).
|
||||
- **Present-but-empty cell** → `fields[name] = None` (an explicit clear) →
|
||||
resolved at diff time to its `CLEAR_DEFAULTS` entry, or NULL where none
|
||||
exists.
|
||||
- **The position exception:** a cleared `Variant Position` has no
|
||||
`CLEAR_DEFAULTS` entry — `diff.resolved_variant_fields` resolves it to the
|
||||
variant's 1-based file order within its product, never to NULL.
|
||||
|
||||
Option names live both in `CanonicalProduct.option_names` (the values) and in
|
||||
`fields{}` (the file-presence marker the diff needs for the absent-vs-clear
|
||||
distinction).
|
||||
|
||||
## Error granularity
|
||||
|
||||
`validate.py` never raises on a row problem: every violation is recorded as a
|
||||
merchant-language `RowError` and **poisons its whole product block** — the
|
||||
product previews as `kind="error"` and is excluded from apply, while parsing
|
||||
continues so one pass yields a complete accounting (BUC-1a). On apply, error
|
||||
rows are recorded per line in `import_run_error`; `rows_errored` on the run is
|
||||
the **error-row count**, while the preview's errors tile counts error
|
||||
**products** — the two numbers legitimately differ.
|
||||
|
||||
File-level problems (`not_csv`, `missing_required_column`, `too_many_rows`,
|
||||
`file_too_large`) raise `FileRejected` in `codec.py` instead: no draft is
|
||||
created. The BFF adds an early 413 for oversized uploads (`main.py`).
|
||||
|
||||
## INV-11 mechanics
|
||||
|
||||
`compute_diff` makes **one walk** that produces two views of the same
|
||||
computation: the typed apply `plan` (resolved natives — `Decimal`, `bool`,
|
||||
lists) that `confirm_draft` executes, and the JSON-safe preview `records`
|
||||
stored as draft JSONB and served verbatim to the SPA. Because both derive from
|
||||
the same walk they cannot diverge. The `fingerprint` is
|
||||
`sha256(json.dumps(records, sort_keys=True, separators=(",", ":")))`; a
|
||||
mismatch at confirm means the catalog drifted since preview → `PreviewStale`
|
||||
(409 `preview_stale`, draft kept for re-validation). The whole apply — run row,
|
||||
product/variant/image writes, error rows, draft delete — is one transaction;
|
||||
any exception rolls it back and emits TEL-6.
|
||||
|
||||
## Named seams (what later slices replace)
|
||||
|
||||
- **`import_draft.file_bytes` → objectstore key (SLICE-7).** The upload
|
||||
currently lives as BYTEA on the draft row (`0002_products.sql`); SLICE-7
|
||||
moves the bytes to object storage and stores a key.
|
||||
- **`codec.detect_dialect` → Shopify (SLICE-8).** Today it always returns
|
||||
`"canonical"`; SLICE-8 recognizes Shopify's exact header set here (INV-17).
|
||||
- **Run-status complete shortcut → `fetching_images` (SLICE-7).**
|
||||
`confirm_draft` inserts the run with `status="complete"` directly; SLICE-7
|
||||
inserts it as `fetching_images` and hands off to the image-fetch task.
|
||||
Relatedly, image rows are created with the schema default
|
||||
`status='pending'` **today and stay pending** — placeholder behavior until
|
||||
SLICE-7's fetch phase (the run-detail payload already carries the stable
|
||||
`image_progress` / `image_outcomes` shape, zeroed).
|
||||
|
||||
## Test map
|
||||
|
||||
| File | Covers |
|
||||
| --- | --- |
|
||||
| `backend/tests/test_products_codec.py` | file-level gates: parse, caps (INV-18), required columns, dialect |
|
||||
| `backend/tests/test_products_validate.py` | every §6.5.1 row-error rule, one fixture each |
|
||||
| `backend/tests/test_products_diff.py` | classification, blank-vs-absent, option matching, fingerprint |
|
||||
| `backend/tests/test_products_service.py` | draft lifecycle: validate/preview/discard, expiry, TEL-1 |
|
||||
| `backend/tests/test_products_invariants.py` | INV-10 (never deletes), INV-14 (storefront isolation), apply transactionality, TEL-6 |
|
||||
| `backend/tests/test_products_endpoints.py` | §6.4 API scenarios + auth/storefront gates |
|
||||
| `e2e/tests/import-preview-confirm.spec.ts` | happy path: upload → preview → confirm → history |
|
||||
| `e2e/tests/import-errors.spec.ts` | actionable row errors at preview and on the run report |
|
||||
| `e2e/tests/import-file-rejected.spec.ts` | file-level rejection, picker stays live, no trace |
|
||||
| `e2e/tests/import-cancel.spec.ts` | cancel at preview leaves no trace |
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "wiggleverse-ecomm-frontend",
|
||||
"private": true,
|
||||
"version": "0.4.0",
|
||||
"version": "0.5.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite",
|
||||
|
||||
Reference in New Issue
Block a user