diff --git a/specs/SD-0002-products-bulk-csv-import-export.md b/specs/SD-0002-products-bulk-csv-import-export.md index 87a7ca5..979239c 100644 --- a/specs/SD-0002-products-bulk-csv-import-export.md +++ b/specs/SD-0002-products-bulk-csv-import-export.md @@ -597,7 +597,175 @@ on demand and never committed. All surfaces live inside SD-0001's admin shell ## 6. Technical Design -*(not yet authored)* +Builds on SD-0001's shipped architecture (FastAPI backend, layered +`main → domains → platform` with import-linter enforcement; React/Vite admin +SPA; PostgreSQL everywhere). Invariant numbering continues from SD-0001 +(INV-1…9) so citations stay unique app-wide. + +### 6.1 Invariants + +- **INV-10 — Import never deletes.** No code path of the import/export + mechanism removes a product, variant, or image record from the catalog. + The domain API exposes no bulk-delete; "upsert" is the only mutation. + *Enforced:* domain surface (no delete function exists) + tests asserting + catalog cardinality never decreases across any import. +- **INV-11 — Nothing applies without confirmation.** Validation and preview + write nothing to catalog tables; only an explicit confirm mutates, and it + applies exactly the diff the preview showed — or refuses as stale + (BUC-2's faithfulness, mechanically). *Enforced:* the preview path holds + no write connection to catalog tables (tests assert zero writes); confirm + recomputes the diff and compares its fingerprint to the previewed one + (§6.5, PUC-3/4). +- **INV-12 — Export → import round-trips to a no-op.** Exporting a catalog + and re-importing the file unmodified produces an empty diff (PUC-10, + issue #13's acceptance bar). The canonical codec is lossless for every + field the catalog stores. *Enforced:* property-style test — for arbitrary + catalogs: `diff(catalog, import(export(catalog))) = ∅`. +- **INV-13 — `Handle` is product identity; the option-value combination is + variant identity.** Per storefront, one product per handle; per product, + one variant per (Option1, Option2, Option3) value combination. SKU is + correctable *data*, never identity. *Enforced:* unique indexes + (`(storefront_id, handle)`; `(product_id, option1_value, option2_value, + option3_value)`). +- **INV-14 — Every catalog row carries its storefront.** All product-domain + tables reference `storefront_id` directly or via their product; every + query is storefront-scoped (extends SD-0001 INV-5 to its first real + tenant-data domain). *Enforced:* FK constraints + service-layer scoping + + tests with two storefronts asserting zero bleed. +- **INV-15 — Imported HTML is sanitized at the boundary.** `Description` + HTML is sanitized against an allowlist (formatting tags; no scripts, + event handlers, or embeds) before storage. The catalog never stores HTML + we would not render. *Enforced:* sanitizer in the import path + tests + with hostile fixtures. +- **INV-16 — Surfaces serve only hosted images.** No storefront or admin + surface ever hotlinks a merchant-supplied URL; an image is + fetched-and-hosted (with renditions) or it is a placeholder with a + tracked status. The source URL is retained as data (provenance, retry). + *Enforced:* render paths take storage keys, not URLs; tests. +- **INV-17 — Dialects normalize at the boundary.** Every accepted file is + mapped to the canonical row model *before* validation and diffing; the + products domain knows exactly one model. A new dialect (XLSX, another + platform) is a new edge adapter, never a second code path through the + domain. *Enforced:* layering — dialect adapters live at the API/codec + edge; domain functions accept only canonical rows. +- **INV-18 — Import work is bounded and fails honestly.** Hard caps, + rejected whole-file with a named reason (PUC-5a): ≤ 5,000 data rows, + ≤ 10 MB file. Image fetching is bounded per image (≤ 20 MB, 30 s + timeout, content-type must be an image format we process). Over a cap is + an honest refusal, never a partial silent attempt (SD-0001 INV-9). + *Enforced:* checks at the upload/fetch boundaries + tests at the caps. + +### 6.2 High-level architecture + +```mermaid +flowchart LR + subgraph Browser + SPA[Admin SPA
Products section §5] + end + subgraph "ecomm backend (FastAPI, one process)" + BFF[REST BFF
main.py] + subgraph domains + ACC[accounts] + SF[storefronts] + PROD[products ◄ new
catalog · import/export
csv codec · dialects · diff] + end + subgraph platform + DB[db
Postgres + migrations] + OBJ[objectstore ◄ new
port: local / GCS] + IMG[images ◄ new
validate + renditions] + DEPS[deps] + end + BG[image-fetch task
in-process, post-commit ◄ new] + end + STORE[(PostgreSQL)] + GCS[(Object storage
GCS · dev: local disk)] + HOSTS[Merchant image hosts
egress fetch] + + SPA -->|JSON + multipart /api/*| BFF + BFF --> PROD + BFF --> ACC + BFF --> SF + PROD --> DB + PROD --> OBJ + PROD --> IMG + DB --> STORE + OBJ --> GCS + BG --> HOSTS + BG --> OBJ + BG --> DB +``` + +- **`products` domain (new)** — owns the catalog (products, variants, + images) and the import/export use cases: the canonical row model, the CSV + codec (parse/serialize), dialect detection + the Shopify mapping + (INV-17), validation, the diff engine (catalog × canonical rows → + add/update/unchanged/error), upsert application (INV-10/11/13), import + drafts and runs. Must never mint identity or storefronts; everything it + touches is storefront-scoped (INV-14). +- **`platform/objectstore` (new)** — a small port (`put / get / delete / + public_url`) with two adapters per the mailer pattern: local-disk + (dev/tests) and GCS (PPE/prod; bucket + credentials from deployment + config, SD-0001 INV-8). Owns no semantics. +- **`platform/images` (new)** — pure image processing: verify the bytes are + a decodable image (jpeg/png/webp), check the resolution bar, emit + renditions (thumb ~160px / card ~480px / detail ≤ 1600px — downscale + only). Model-free, deterministic, no I/O of its own. +- **Image-fetch task (new)** — the post-commit phase (PUC-7): an in-process + asyncio task per run that walks the run's pending images, fetches with + INV-18's bounds, runs `platform/images`, stores renditions via + `objectstore`, and updates per-image status + run progress. **This is the + async seam:** it is the only fire-and-forget work in the app; when + catalogs outgrow the in-process model, this task body moves onto a real + queue without changing its inputs/outputs (§6.9 covers crash recovery). +- **REST BFF** — gains the `/api/products/*` endpoints (§6.4); still owns no + business logic (SD-0001 INV-6). +- **Admin SPA** — gains the Products section (§5); renders what the BFF + returns; live progress is polling (no websockets in the MVP — deliberate). + +### 6.3 Data model & ownership + +All owned by the `products` domain; system of record PostgreSQL. Monetary +values are `NUMERIC`; no currency column — the MVP is single-implicit-currency +(open question Q-1, §13). + +| Entity | Key fields | Notes | +| --- | --- | --- | +| `product` | `id`, `storefront_id` (FK, INV-14), `handle` (unique per storefront, INV-13), `title`, `description_html` (sanitized, INV-15), `vendor`, `product_type` (`standalone`; `kit_virtual` / `kit_assembled` reserved for #15), `google_product_category`, `tags` (`text[]`), `status` (`draft \| active \| archived`), `published` (bool), `option1_name`…`option3_name`, timestamps | Option *names* live on the product; a no-option product has all names NULL and exactly one variant. | +| `variant` | `id`, `product_id` (FK), `position`, `option1_value`…`option3_value` (unique combo per product, INV-13), `sku`, `barcode`, `price`, `cost`, `weight` + `weight_unit`, `volume` + `volume_unit`, `tax_id_1`, `tax_id_2`, `inventory_tracker`, `inventory_qty`, `image_id` (FK → `product_image`, nullable), timestamps | The no-option product's single variant has NULL option values (uniqueness via `NULLS NOT DISTINCT`). SKU is indexed (component references, #15) but not unique — it is data, not identity. | +| `product_image` | `id`, `product_id` (FK), `position`, `source_url`, `alt_text`, `status` (`pending \| fetched \| rejected_low_res \| rejected_not_image \| failed`), `failure_reason`, storage keys (`original`, `thumb`, `card`, `detail`), `import_run_id` (FK — the run that introduced/last fetched it), `fetched_at` | Identity within a product is `source_url`: re-importing the same URL is "unchanged"; a new URL at a position is an update. INV-16 governs serving. | +| `import_draft` | `id`, `storefront_id`, `account_id`, `file_name`, `dialect`, file blob (objectstore key), diff summary + fingerprint, `unknown_columns`, `expires_at` (~1 h), `created_at` | The preview's server side. Canceled or expired drafts are deleted outright — no trace (PUC-3a). Never appears in history. | +| `import_run` | `id`, `storefront_id`, `account_id`, `file_name`, `dialect`, counts (`products_added`, `products_updated`, `rows_errored`), `status` (`applying \| fetching_images \| complete \| complete_with_problems`), `created_at`, `completed_at` | Created only at confirm (PUC-4). The durable record (PUC-8). | +| `import_run_error` | `id`, `run_id` (FK), `line_number`, `column_name`, `message` | One row per rejected CSV row (PUC-5); merchant-language message. | + +**Deliberately absent:** any `component` / kit table — the CSV format +reserves the columns (§6.5 dialect spec), but the entities, validation, and +meaning are #15's design. In this MVP a non-`standalone` `Type` or a +non-empty `Component n` column is a row error ("kits arrive in a coming +release"). + +### 6.4 Interfaces & contracts + +All under `/api/products/*`; session cookie + storefront membership required +(SD-0001's gates); errors share SD-0001's shape +`{"error": {"code", "message"}}`. Adding fields is free; renaming/removing is +a spec change. + +| Endpoint | In | Out | Errors | +| --- | --- | --- | --- | +| `POST /api/products/imports` | multipart CSV file | `201 {draft}` — dialect, summary counts, `unknown_columns`, `expires_at` (draft persisted server-side, INV-11) | `400 not_csv` · `400 missing_required_column` · `400 unknown_dialect` · `400 too_many_rows` · `413 file_too_large` (all PUC-5a — no draft created) | +| `GET /api/products/imports/drafts/{id}` | — | `200 {draft}` summary (re-fetch for the preview shell) | `404` · `410 draft_expired` | +| `GET /api/products/imports/drafts/{id}/records` | `?kind=add\|update\|unchanged\|error`, paging | `200 {records: [{handle, title, kind, variant_count, detail}]}` — `detail` carries field-level before → after diffs (PUC-3) or the error rows | `404` · `410 draft_expired` | +| `POST /api/products/imports/drafts/{id}/confirm` | — | `201 {run_id}` — diff re-computed and fingerprint-checked, applied in one transaction (INV-11), image task started, draft deleted | `404` · `410 draft_expired` · `409 preview_stale` (catalog changed since validation — re-upload) · `409 nothing_to_apply` (PUC-10) | +| `DELETE /api/products/imports/drafts/{id}` | — | `204` — draft and its blob deleted, no trace (PUC-3a) | `404` (idempotent enough) | +| `GET /api/products/imports/runs` | paging | `200 {runs: [...]}` — history, newest first (PUC-8) | — | +| `GET /api/products/imports/runs/{id}` | — | `200 {run}` — counts, status, error rows, image outcomes + `image_progress {done, total}`; the SPA polls this while `status != complete*` (PUC-4/7) | `404` | +| `GET /api/products/export` | `?status=all\|active\|draft\|archived` | `200 text/csv` (streamed) — canonical format, hosted image URLs (PUC-9, INV-12) | `409 empty_catalog` | +| `GET /api/products/sample.csv` | — | `200 text/csv` — the documented sample (PUC-11) | — | +| `GET /api/products/summary` | — | `200 {product_count, image_problem_count, latest_run_id}` — drives the Products page header + notice band (§5.2) | — | + +*(§§6.5–6.10 — per-PUC design incl. the canonical column spec and Shopify +mapping, NFRs, decisions, testing, failure modes, migration — next in the +loop.)* ## 7. Delivery Plan