diff --git a/specs/SD-0002-products-bulk-csv-import-export.md b/specs/SD-0002-products-bulk-csv-import-export.md index 979239c..74ea2e1 100644 --- a/specs/SD-0002-products-bulk-csv-import-export.md +++ b/specs/SD-0002-products-bulk-csv-import-export.md @@ -732,7 +732,7 @@ values are `NUMERIC`; no currency column — the MVP is single-implicit-currency | --- | --- | --- | | `product` | `id`, `storefront_id` (FK, INV-14), `handle` (unique per storefront, INV-13), `title`, `description_html` (sanitized, INV-15), `vendor`, `product_type` (`standalone`; `kit_virtual` / `kit_assembled` reserved for #15), `google_product_category`, `tags` (`text[]`), `status` (`draft \| active \| archived`), `published` (bool), `option1_name`…`option3_name`, timestamps | Option *names* live on the product; a no-option product has all names NULL and exactly one variant. | | `variant` | `id`, `product_id` (FK), `position`, `option1_value`…`option3_value` (unique combo per product, INV-13), `sku`, `barcode`, `price`, `cost`, `weight` + `weight_unit`, `volume` + `volume_unit`, `tax_id_1`, `tax_id_2`, `inventory_tracker`, `inventory_qty`, `image_id` (FK → `product_image`, nullable), timestamps | The no-option product's single variant has NULL option values (uniqueness via `NULLS NOT DISTINCT`). SKU is indexed (component references, #15) but not unique — it is data, not identity. | -| `product_image` | `id`, `product_id` (FK), `position`, `source_url`, `alt_text`, `status` (`pending \| fetched \| rejected_low_res \| rejected_not_image \| failed`), `failure_reason`, storage keys (`original`, `thumb`, `card`, `detail`), `import_run_id` (FK — the run that introduced/last fetched it), `fetched_at` | Identity within a product is `source_url`: re-importing the same URL is "unchanged"; a new URL at a position is an update. INV-16 governs serving. | +| `product_image` | `id`, `product_id` (FK), `position`, `source_url`, `alt_text`, `status` (`pending \| fetched \| rejected_low_res \| rejected_not_image \| failed`), `failure_reason`, storage keys (`original`, `thumb`, `card`, `detail`), `import_run_id` (FK — the run that introduced/last fetched it), `fetched_at` | Identity within a product is `source_url`: re-importing the same URL is "unchanged" — never re-fetched. A **platform-hosted URL** (an export's `Image Src`) resolves to the existing image record by storage key — never fetched (same storefront required, else row error; INV-12/14). A fetch happens only for a genuinely new URL, or as retry of a `failed`/`rejected` one. INV-16 governs serving. | | `import_draft` | `id`, `storefront_id`, `account_id`, `file_name`, `dialect`, file blob (objectstore key), diff summary + fingerprint, `unknown_columns`, `expires_at` (~1 h), `created_at` | The preview's server side. Canceled or expired drafts are deleted outright — no trace (PUC-3a). Never appears in history. | | `import_run` | `id`, `storefront_id`, `account_id`, `file_name`, `dialect`, counts (`products_added`, `products_updated`, `rows_errored`), `status` (`applying \| fetching_images \| complete \| complete_with_problems`), `created_at`, `completed_at` | Created only at confirm (PUC-4). The durable record (PUC-8). | | `import_run_error` | `id`, `run_id` (FK), `line_number`, `column_name`, `message` | One row per rejected CSV row (PUC-5); merchant-language message. | @@ -763,9 +763,286 @@ a spec change. | `GET /api/products/sample.csv` | — | `200 text/csv` — the documented sample (PUC-11) | — | | `GET /api/products/summary` | — | `200 {product_count, image_problem_count, latest_run_id}` — drives the Products page header + notice band (§5.2) | — | -*(§§6.5–6.10 — per-PUC design incl. the canonical column spec and Shopify -mapping, NFRs, decisions, testing, failure modes, migration — next in the -loop.)* +### 6.5 Per–Product-Use-Case design + +#### 6.5.1 The canonical CSV format (the contract behind every PUC) + +UTF-8 (BOM tolerated) · comma-delimited · RFC 4180 quoting · header row +required · caps per INV-18. + +**Row grammar (Shopify-style):** consecutive rows sharing a `Handle` form one +product. The product's first row carries the product-level fields (`Title` +required there); each row may carry a variant (its `Option n Value`s + +`Variant *` fields), an image (`Image Src` + `Image Position`/`Alt Text`), +or both. Image-only rows (just `Handle` + `Image *`) are valid — that's how +a product carries more images than variants. + +| Column | Level | Required | Notes | +| --- | --- | --- | --- | +| `Handle` | product | every row | identity (INV-13): lowercase letters, numbers, dashes | +| `Title` | product | first row of a product | | +| `Description` | product | — | HTML allowed; sanitized (INV-15) | +| `Vendor` | product | — | free text | +| `Type` | product | — | `standalone` (default); `kit_virtual` / `kit_assembled` reserved — non-`standalone` is a row error until #15 | +| `Google Product Category` | product | — | taxonomy string | +| `Tags` | product | — | comma-separated within the cell | +| `Status` | product | — | `draft \| active \| archived`; default `active` on add | +| `Published` | product | — | `TRUE \| FALSE`; default `TRUE` on add | +| `Option1–3 Name` | product | with values | e.g. "Size"; a value without its name is a row error | +| `Option1–3 Value` | variant | per variant | the variant's identity combo (INV-13) | +| `Variant SKU`, `Variant Barcode` | variant | — | SKU indexed, not identity | +| `Variant Price`, `Variant Cost` | variant | — | decimal; no currency (Q-1) | +| `Variant Weight` + `Unit`, `Variant Volume` + `Unit` | variant | — | decimal + unit string | +| `Variant Tax ID 1–2` | variant | — | opaque references | +| `Variant Inventory Tracker`, `Variant Inventory Qty` | variant | — | qty integer ≥ 0 | +| `Variant Position` | variant | — | display order; defaults to file order | +| `Image Src`, `Image Position`, `Image Alt Text` | image | — | URL (or platform-hosted URL on re-import — §6.3) | +| `Variant Image` | variant | — | URL for this variant's specific image | +| `Component 1–10 SKU` / `Quantity` | variant | — | **reserved for #15** — non-empty is a row error ("kits arrive in a coming release") | + +**Update semantics (blank vs absent):** a column *absent from the file* is +untouched by the import; a cell *present but empty* clears that field (or +resets it to its default). The preview's diff shows clears explicitly — this +is previewable, deliberate behavior, not a footgun (INV-11). + +**The Shopify dialect** is recognized by its exact header set and mapped at +the boundary (INV-17): `Body (HTML)` → `Description`, `Product Category` → +`Google Product Category`, `Cost per item` → `Variant Cost`, +`Variant Grams`/`Variant Weight Unit` → weight fields; `Handle`, `Title`, +`Vendor`, `Tags`, `Status`, `Published`, options, SKU/barcode/price, +inventory and image columns map directly. Shopify columns with no canonical +home — `Type` (Shopify's free-text type), `Compare At Price`, SEO fields, +Google Shopping fields, fulfillment/shipping/tax-code flags, `Gift Card` — +are **warned as not-imported** in the preview (PUC-6); the implementation +plan pins the exhaustive mapping table as a fixture. + +#### 6.5.2 Import: upload → validate → preview (PUC-2, PUC-3, PUC-3a, PUC-5a, PUC-6) + +```mermaid +sequenceDiagram + actor M as Merchant + participant S as SPA + participant A as BFF + participant P as products domain + participant O as objectstore + participant D as Postgres + M->>S: select CSV file + S->>A: POST /api/products/imports (multipart) + A->>P: import_validate(storefront, file) + P->>P: parse · detect dialect · map to canonical (INV-17) + alt file-level problem (PUC-5a) + P-->>A: not_csv / missing_required_column / unknown_dialect / too_many_rows + A-->>S: 400 — no draft, no trace + else valid file + P->>P: validate rows · diff against catalog (read-only, INV-11) + P->>O: store file blob (private) + P->>D: insert import_draft (summary, fingerprint, expires_at) + A-->>S: 201 draft {dialect, counts, unknown_columns} + S-->>M: preview (§5.4); records paged via GET …/records + end + opt cancel (PUC-3a) + M->>S: Cancel + S->>A: DELETE …/drafts/{id} + A->>P: discard(draft) — row + blob deleted, no trace + end +``` + +The diff engine compares canonical rows to the storefront's catalog +per-product: field-by-field over product, variants (matched by option combo, +INV-13), and images (matched by `source_url`, with platform-hosted URLs +resolved to existing records — §6.3 — so an exported file diffs clean, +INV-12). Every product classifies as add / update / unchanged / error; +errors carry line + column + merchant-language message (PUC-5). + +#### 6.5.3 Import: confirm → apply (PUC-4, PUC-5, PUC-10) + +```mermaid +sequenceDiagram + actor M as Merchant + participant S as SPA + participant A as BFF + participant P as products domain + participant D as Postgres + participant T as image-fetch task + M->>S: "Import N products" + S->>A: POST …/drafts/{id}/confirm + A->>P: import_apply(draft) + P->>P: recompute diff · compare fingerprint (INV-11) + alt fingerprint differs + P-->>A: 409 preview_stale → re-upload + else nothing to apply (PUC-10) + P-->>A: 409 nothing_to_apply + else applies + P->>D: ONE transaction: upsert products/variants/images (INV-10/13),
insert import_run + errors, delete draft + P->>T: start(run) — images with status=pending + A-->>S: 201 {run_id} + S-->>M: run detail (§5.5), polling + end +``` + +Per-image work in the transaction: new `source_url` → `pending` row; +unchanged URL → untouched; platform-hosted URL → resolved, untouched; +prior `failed`/`rejected` URL re-supplied → reset to `pending` (the retry +path, PUC-7). + +#### 6.5.4 The image-fetch phase (PUC-7) + +```mermaid +sequenceDiagram + participant T as image-fetch task + participant H as merchant image host + participant I as platform/images + participant O as objectstore + participant D as Postgres + loop pending images of the run (bounded concurrency ~4) + T->>H: GET source_url (INV-18 bounds: 20 MB · 30 s · SSRF guard §6.6) + alt unreachable / oversize / wrong type + T->>D: status=failed + reason + else bytes received + T->>I: decode · resolution bar · renditions (thread pool) + alt not an image / below the bar + T->>D: status=rejected_* + reason + else ok + T->>O: put original + thumb/card/detail + T->>D: status=fetched + storage keys + end + end + T->>D: bump run image_progress + end + T->>D: run status → complete / complete_with_problems +``` + +Idempotent per image (keyed by image id; statuses only move +`pending → terminal`), so the task can die and resume at any point (§6.9). +The SPA polls `GET …/runs/{id}` for progress and outcomes. + +#### 6.5.5 Export (PUC-9) · the simple PUCs + +Export streams the catalog (optionally status-filtered) through the canonical +serializer — the same codec the importer parses (one codec, two directions; +INV-12 is a property of that codec plus hosted-URL resolution). `Image Src` +carries the hosted `detail`-rendition URL (or the original `source_url` for +never-fetched/`failed` images, so nothing is lost). PUC-1 (Products page) is +`GET /api/products/summary`; PUC-8 (history) is the runs list/detail +endpoints; PUC-11 (sample CSV) is a static documented asset served at +`GET /api/products/sample.csv` and linked from the column reference (DOC-2, +§11). + +### 6.6 Non-functional requirements & cross-cutting concerns + +- **Security & privacy:** all endpoints behind SD-0001's session + + storefront-membership gates; every query storefront-scoped (INV-14). + **SSRF guard on image fetch:** http/https only, public IPs only (resolve + and reject private/link-local/metadata ranges), no redirects across the + guard, bounded size/time (INV-18). HTML sanitization at the boundary + (INV-15). Uploaded file blobs are private objects, deleted with their + draft; catalog data is merchant business data (no shopper PII in this + feature). Run records and errors are retained indefinitely (they are the + merchant's audit trail); drafts expire in ~1 h. +- **Performance & scale:** validate + diff ≤ 5,000 rows in < 10 s + (one catalog read, in-memory diff); apply in one transaction < 5 s; + export streams (no whole-catalog buffering); image phase: ~4 concurrent + fetches, Pillow in a worker thread pool — a 5,000-image run completes in + tens of minutes *in the background* without touching request latency. +- **Availability & resilience:** the import transaction is short; the image + task is resumable across restarts/deploys (§6.9); a dead merchant image + host degrades to per-image `failed`, never a hung import. +- **Observability hooks:** `import_run.id` is the correlation id — the + fetch task and all run-scoped log lines carry it (substance in §9). +- **Accessibility:** preview/summary tiles and tables keyboard-navigable; + progress updates `aria-live`; diff colors paired with +/− glyphs (not + color-only). + +### 6.7 Key decisions & alternatives considered + +| Decision | Chosen | Alternatives considered | Why chosen | +| --- | --- | --- | --- | +| Matching identity | `Handle` for products, option-combo for variants (INV-13) | SKU as identity (the corpus's 11-05-0035) | Real catalogs have missing/duplicated SKUs; a SKU typo must be *correctable by import*, which identity can't be. Divergence from corpus recorded in §4. | +| Image handling | Fetch-and-host with renditions; reject low-res | Hotlink merchant URLs; validate-only + hotlink; accept low-res with warning | Hotlinking leaks shopper traffic to third parties, rots, and breaks INV-16; rejected-low-res keeps every hosted image renderable at all surfaces (operator decision). | +| Image timing | Post-commit background phase | Fetch during import (stalls minutes); fetch at preview (wasted on cancel) | Catalog lands in seconds; images are the async seam's first tenant. | +| Background model | In-process asyncio task, resumable | Job queue (Cloud Tasks/celery); cron sweep | No queue infra for an MVP with one fire-and-forget job; the task body is queue-portable when scale demands (§6.9 names the seam). | +| Preview state | Server-side `import_draft` + fingerprint | Client holds the diff and posts it back; re-validate blindly at confirm | The server applying exactly what it previewed is INV-11; a client-posted diff is unverifiable; fingerprint check catches catalog drift honestly (`409 preview_stale`). | +| Format | Canonical = Shopify-flavored superset | Invent a clean format; exact Shopify schema | Familiarity + migration head start without inheriting Shopify's gaps (no kit seam, no volume/tax fields). | +| Shopify free-text `Type` | Warned, not imported | Map into `Tags`; map into canonical `Type` | Canonical `Type` is structural (`standalone`/kits, #15); silently folding a category string into it (or tags) invents data the merchant didn't write. | +| Blank vs absent | Absent column untouched; empty cell clears | Empty cell ignored (Shopify-ish); empty cell always clears including absent | Merchants must be able to clear a field via import; the preview shows every clear (INV-11), making the sharper semantic safe. | +| Progress delivery | SPA polls run detail | WebSockets/SSE | One poll endpoint, no connection infra; seconds-stale progress is fine for this UX. | +| Notifications | In-app only | Email summaries via SmtpMailer | Operator decision; revisit if imports go async/long. | +| Object storage | New `platform/objectstore` port (local/GCS) | Store images as DB blobs; reuse a public bucket directly in domain code | Follows the mailer port pattern (SD-0001); DB blobs bloat backups and can't serve renditions efficiently. | + +### 6.8 Testing strategy + +Two-tier per handbook §9: unit/integration locally and in CI; E2E browser +tests (Playwright) against localhost and PPE as the pipeline gates. + +- **Unit — codec & dialects:** parse/serialize round-trip property tests + (INV-12: `diff(catalog, import(export(catalog))) = ∅` over generated + catalogs); a real Shopify export fixture mapping correctly; unknown-column + warning extraction; blank-vs-absent semantics; every §6.5.1 row-error rule + has a fixture. +- **Unit — diff engine:** add/update/unchanged/error classification; + option-combo matching incl. the no-option single variant; hosted-URL + resolution (no re-fetch); clear-vs-untouched fields; fingerprint stability. +- **Unit — images:** `platform/images` against fixture bytes (good jpeg/png/ + webp, tiny image, not-an-image, oversize); rendition dimensions. +- **Unit — sanitizer:** hostile HTML fixtures (scripts, handlers, embeds). +- **Integration:** apply transactionality (a mid-apply failure leaves zero + catalog change); two-storefront isolation (INV-14); fetch-task resume + after a simulated kill; SSRF guard against private-range URLs. +- **E2E browser scenarios (named):** + `e2e_import_preview_confirm` (canonical file → preview counts → drill-in + diff → confirm → run detail → summary count updated); + `e2e_import_shopify_dialect` (unmodified Shopify fixture, mapped banner + + not-imported warnings); `e2e_import_errors_actionable` (bad rows: error + table lists line/column/reason; valid rows import); + `e2e_import_file_rejected` (missing `Title` column — whole-file message, + no run recorded); `e2e_import_cancel_no_trace`; + `e2e_roundtrip_noop` (export → re-import → all-unchanged, import action + disabled); `e2e_export_download` (status filter respected); + `e2e_image_outcomes` (fixture image host serving one good / one 404 / one + tiny image → statuses, placeholder, Products-page notice). The E2E suite + runs against a local fixture image host (no real egress in tests). +- **Bootstrap:** SD-0001's INV-1 test extends — fresh DB → sign-up → + storefront → import sample CSV → export matches. + +### 6.9 Failure modes, rollback & flags + +- **Failure mode:** crash mid-apply → **behavior:** transaction rolls back — + zero catalog change; draft intact; SPA shows honest error → + **rollback:** none needed → **alerted by:** ALR-2 (§9.3). +- **Failure mode:** process restart (deploy) mid image-fetch → + **behavior:** on startup, a recovery scan finds runs in `fetching_images` + with `pending` images and resumes the task; per-image idempotency (§6.5.4) + makes re-entry safe → **alerted by:** TEL/log line on recovery; ALR-3 if a + run stays incomplete past threshold. +- **Failure mode:** merchant image host down/slow → **behavior:** per-image + timeout → `failed` with reason; run completes `complete_with_problems`; + retry is a re-import (PUC-7) → no alert (expected operation). +- **Failure mode:** object storage unavailable → **behavior:** affected + images `failed` honestly; uploads (drafts) refuse with a clear error → + **alerted by:** ALR-3. +- **Failure mode:** draft expiry / catalog drift between preview and confirm + → **behavior:** `410 draft_expired` / `409 preview_stale`; UI prompts + re-upload — never applies stale diffs (INV-11). +- **Feature flag / kill switch:** none — the Products section ships whole; + rollback is redeploying the prior version (pipeline §7.3). Migrations are + additive new tables, so the prior version runs unaffected against the + migrated schema (SD-0001 INV-7 forward-only). +- **The async seam, named:** if catalogs outgrow the in-process model + (imports > 5k rows, image sets taking hours), the fetch-task body moves + onto a real queue (Cloud Tasks or a worker process) with identical + inputs/outputs; `import_run`/`product_image` statuses already model the + asynchrony, so the UX and API do not change shape. + +### 6.10 Migration & compatibility + +Greenfield within a live app: no existing catalog data to migrate. New +additive tables (§6.3) via forward-only migrations (SD-0001 INV-7); the +admin nav gains "Products"; no existing endpoint or contract changes. +SD-0001's INV-1 (empty is a working state) holds — an empty catalog is the +designed Products-page empty state. **Backwards compatibility:** none owed +yet (no external API consumers). **Cutover:** ships whole behind the +ordinary release pipeline; no dual-write, no point of no return short of +merchants having imported real data (which INV-10/12 protect). ## 7. Delivery Plan