spec(SD-0002 §6.5–6.10): column spec, dialect mapping, per-PUC sequences, hosted-URL no-refetch, NFRs incl. SSRF guard, decisions, testing, failure modes
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -732,7 +732,7 @@ values are `NUMERIC`; no currency column — the MVP is single-implicit-currency
|
||||
| --- | --- | --- |
|
||||
| `product` | `id`, `storefront_id` (FK, INV-14), `handle` (unique per storefront, INV-13), `title`, `description_html` (sanitized, INV-15), `vendor`, `product_type` (`standalone`; `kit_virtual` / `kit_assembled` reserved for #15), `google_product_category`, `tags` (`text[]`), `status` (`draft \| active \| archived`), `published` (bool), `option1_name`…`option3_name`, timestamps | Option *names* live on the product; a no-option product has all names NULL and exactly one variant. |
|
||||
| `variant` | `id`, `product_id` (FK), `position`, `option1_value`…`option3_value` (unique combo per product, INV-13), `sku`, `barcode`, `price`, `cost`, `weight` + `weight_unit`, `volume` + `volume_unit`, `tax_id_1`, `tax_id_2`, `inventory_tracker`, `inventory_qty`, `image_id` (FK → `product_image`, nullable), timestamps | The no-option product's single variant has NULL option values (uniqueness via `NULLS NOT DISTINCT`). SKU is indexed (component references, #15) but not unique — it is data, not identity. |
|
||||
| `product_image` | `id`, `product_id` (FK), `position`, `source_url`, `alt_text`, `status` (`pending \| fetched \| rejected_low_res \| rejected_not_image \| failed`), `failure_reason`, storage keys (`original`, `thumb`, `card`, `detail`), `import_run_id` (FK — the run that introduced/last fetched it), `fetched_at` | Identity within a product is `source_url`: re-importing the same URL is "unchanged"; a new URL at a position is an update. INV-16 governs serving. |
|
||||
| `product_image` | `id`, `product_id` (FK), `position`, `source_url`, `alt_text`, `status` (`pending \| fetched \| rejected_low_res \| rejected_not_image \| failed`), `failure_reason`, storage keys (`original`, `thumb`, `card`, `detail`), `import_run_id` (FK — the run that introduced/last fetched it), `fetched_at` | Identity within a product is `source_url`: re-importing the same URL is "unchanged" — never re-fetched. A **platform-hosted URL** (an export's `Image Src`) resolves to the existing image record by storage key — never fetched (same storefront required, else row error; INV-12/14). A fetch happens only for a genuinely new URL, or as retry of a `failed`/`rejected` one. INV-16 governs serving. |
|
||||
| `import_draft` | `id`, `storefront_id`, `account_id`, `file_name`, `dialect`, file blob (objectstore key), diff summary + fingerprint, `unknown_columns`, `expires_at` (~1 h), `created_at` | The preview's server side. Canceled or expired drafts are deleted outright — no trace (PUC-3a). Never appears in history. |
|
||||
| `import_run` | `id`, `storefront_id`, `account_id`, `file_name`, `dialect`, counts (`products_added`, `products_updated`, `rows_errored`), `status` (`applying \| fetching_images \| complete \| complete_with_problems`), `created_at`, `completed_at` | Created only at confirm (PUC-4). The durable record (PUC-8). |
|
||||
| `import_run_error` | `id`, `run_id` (FK), `line_number`, `column_name`, `message` | One row per rejected CSV row (PUC-5); merchant-language message. |
|
||||
@@ -763,9 +763,286 @@ a spec change.
|
||||
| `GET /api/products/sample.csv` | — | `200 text/csv` — the documented sample (PUC-11) | — |
|
||||
| `GET /api/products/summary` | — | `200 {product_count, image_problem_count, latest_run_id}` — drives the Products page header + notice band (§5.2) | — |
|
||||
|
||||
*(§§6.5–6.10 — per-PUC design incl. the canonical column spec and Shopify
|
||||
mapping, NFRs, decisions, testing, failure modes, migration — next in the
|
||||
loop.)*
|
||||
### 6.5 Per–Product-Use-Case design
|
||||
|
||||
#### 6.5.1 The canonical CSV format (the contract behind every PUC)
|
||||
|
||||
UTF-8 (BOM tolerated) · comma-delimited · RFC 4180 quoting · header row
|
||||
required · caps per INV-18.
|
||||
|
||||
**Row grammar (Shopify-style):** consecutive rows sharing a `Handle` form one
|
||||
product. The product's first row carries the product-level fields (`Title`
|
||||
required there); each row may carry a variant (its `Option n Value`s +
|
||||
`Variant *` fields), an image (`Image Src` + `Image Position`/`Alt Text`),
|
||||
or both. Image-only rows (just `Handle` + `Image *`) are valid — that's how
|
||||
a product carries more images than variants.
|
||||
|
||||
| Column | Level | Required | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| `Handle` | product | every row | identity (INV-13): lowercase letters, numbers, dashes |
|
||||
| `Title` | product | first row of a product | |
|
||||
| `Description` | product | — | HTML allowed; sanitized (INV-15) |
|
||||
| `Vendor` | product | — | free text |
|
||||
| `Type` | product | — | `standalone` (default); `kit_virtual` / `kit_assembled` reserved — non-`standalone` is a row error until #15 |
|
||||
| `Google Product Category` | product | — | taxonomy string |
|
||||
| `Tags` | product | — | comma-separated within the cell |
|
||||
| `Status` | product | — | `draft \| active \| archived`; default `active` on add |
|
||||
| `Published` | product | — | `TRUE \| FALSE`; default `TRUE` on add |
|
||||
| `Option1–3 Name` | product | with values | e.g. "Size"; a value without its name is a row error |
|
||||
| `Option1–3 Value` | variant | per variant | the variant's identity combo (INV-13) |
|
||||
| `Variant SKU`, `Variant Barcode` | variant | — | SKU indexed, not identity |
|
||||
| `Variant Price`, `Variant Cost` | variant | — | decimal; no currency (Q-1) |
|
||||
| `Variant Weight` + `Unit`, `Variant Volume` + `Unit` | variant | — | decimal + unit string |
|
||||
| `Variant Tax ID 1–2` | variant | — | opaque references |
|
||||
| `Variant Inventory Tracker`, `Variant Inventory Qty` | variant | — | qty integer ≥ 0 |
|
||||
| `Variant Position` | variant | — | display order; defaults to file order |
|
||||
| `Image Src`, `Image Position`, `Image Alt Text` | image | — | URL (or platform-hosted URL on re-import — §6.3) |
|
||||
| `Variant Image` | variant | — | URL for this variant's specific image |
|
||||
| `Component 1–10 SKU` / `Quantity` | variant | — | **reserved for #15** — non-empty is a row error ("kits arrive in a coming release") |
|
||||
|
||||
**Update semantics (blank vs absent):** a column *absent from the file* is
|
||||
untouched by the import; a cell *present but empty* clears that field (or
|
||||
resets it to its default). The preview's diff shows clears explicitly — this
|
||||
is previewable, deliberate behavior, not a footgun (INV-11).
|
||||
|
||||
**The Shopify dialect** is recognized by its exact header set and mapped at
|
||||
the boundary (INV-17): `Body (HTML)` → `Description`, `Product Category` →
|
||||
`Google Product Category`, `Cost per item` → `Variant Cost`,
|
||||
`Variant Grams`/`Variant Weight Unit` → weight fields; `Handle`, `Title`,
|
||||
`Vendor`, `Tags`, `Status`, `Published`, options, SKU/barcode/price,
|
||||
inventory and image columns map directly. Shopify columns with no canonical
|
||||
home — `Type` (Shopify's free-text type), `Compare At Price`, SEO fields,
|
||||
Google Shopping fields, fulfillment/shipping/tax-code flags, `Gift Card` —
|
||||
are **warned as not-imported** in the preview (PUC-6); the implementation
|
||||
plan pins the exhaustive mapping table as a fixture.
|
||||
|
||||
#### 6.5.2 Import: upload → validate → preview (PUC-2, PUC-3, PUC-3a, PUC-5a, PUC-6)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor M as Merchant
|
||||
participant S as SPA
|
||||
participant A as BFF
|
||||
participant P as products domain
|
||||
participant O as objectstore
|
||||
participant D as Postgres
|
||||
M->>S: select CSV file
|
||||
S->>A: POST /api/products/imports (multipart)
|
||||
A->>P: import_validate(storefront, file)
|
||||
P->>P: parse · detect dialect · map to canonical (INV-17)
|
||||
alt file-level problem (PUC-5a)
|
||||
P-->>A: not_csv / missing_required_column / unknown_dialect / too_many_rows
|
||||
A-->>S: 400 — no draft, no trace
|
||||
else valid file
|
||||
P->>P: validate rows · diff against catalog (read-only, INV-11)
|
||||
P->>O: store file blob (private)
|
||||
P->>D: insert import_draft (summary, fingerprint, expires_at)
|
||||
A-->>S: 201 draft {dialect, counts, unknown_columns}
|
||||
S-->>M: preview (§5.4); records paged via GET …/records
|
||||
end
|
||||
opt cancel (PUC-3a)
|
||||
M->>S: Cancel
|
||||
S->>A: DELETE …/drafts/{id}
|
||||
A->>P: discard(draft) — row + blob deleted, no trace
|
||||
end
|
||||
```
|
||||
|
||||
The diff engine compares canonical rows to the storefront's catalog
|
||||
per-product: field-by-field over product, variants (matched by option combo,
|
||||
INV-13), and images (matched by `source_url`, with platform-hosted URLs
|
||||
resolved to existing records — §6.3 — so an exported file diffs clean,
|
||||
INV-12). Every product classifies as add / update / unchanged / error;
|
||||
errors carry line + column + merchant-language message (PUC-5).
|
||||
|
||||
#### 6.5.3 Import: confirm → apply (PUC-4, PUC-5, PUC-10)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
actor M as Merchant
|
||||
participant S as SPA
|
||||
participant A as BFF
|
||||
participant P as products domain
|
||||
participant D as Postgres
|
||||
participant T as image-fetch task
|
||||
M->>S: "Import N products"
|
||||
S->>A: POST …/drafts/{id}/confirm
|
||||
A->>P: import_apply(draft)
|
||||
P->>P: recompute diff · compare fingerprint (INV-11)
|
||||
alt fingerprint differs
|
||||
P-->>A: 409 preview_stale → re-upload
|
||||
else nothing to apply (PUC-10)
|
||||
P-->>A: 409 nothing_to_apply
|
||||
else applies
|
||||
P->>D: ONE transaction: upsert products/variants/images (INV-10/13),<br/>insert import_run + errors, delete draft
|
||||
P->>T: start(run) — images with status=pending
|
||||
A-->>S: 201 {run_id}
|
||||
S-->>M: run detail (§5.5), polling
|
||||
end
|
||||
```
|
||||
|
||||
Per-image work in the transaction: new `source_url` → `pending` row;
|
||||
unchanged URL → untouched; platform-hosted URL → resolved, untouched;
|
||||
prior `failed`/`rejected` URL re-supplied → reset to `pending` (the retry
|
||||
path, PUC-7).
|
||||
|
||||
#### 6.5.4 The image-fetch phase (PUC-7)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant T as image-fetch task
|
||||
participant H as merchant image host
|
||||
participant I as platform/images
|
||||
participant O as objectstore
|
||||
participant D as Postgres
|
||||
loop pending images of the run (bounded concurrency ~4)
|
||||
T->>H: GET source_url (INV-18 bounds: 20 MB · 30 s · SSRF guard §6.6)
|
||||
alt unreachable / oversize / wrong type
|
||||
T->>D: status=failed + reason
|
||||
else bytes received
|
||||
T->>I: decode · resolution bar · renditions (thread pool)
|
||||
alt not an image / below the bar
|
||||
T->>D: status=rejected_* + reason
|
||||
else ok
|
||||
T->>O: put original + thumb/card/detail
|
||||
T->>D: status=fetched + storage keys
|
||||
end
|
||||
end
|
||||
T->>D: bump run image_progress
|
||||
end
|
||||
T->>D: run status → complete / complete_with_problems
|
||||
```
|
||||
|
||||
Idempotent per image (keyed by image id; statuses only move
|
||||
`pending → terminal`), so the task can die and resume at any point (§6.9).
|
||||
The SPA polls `GET …/runs/{id}` for progress and outcomes.
|
||||
|
||||
#### 6.5.5 Export (PUC-9) · the simple PUCs
|
||||
|
||||
Export streams the catalog (optionally status-filtered) through the canonical
|
||||
serializer — the same codec the importer parses (one codec, two directions;
|
||||
INV-12 is a property of that codec plus hosted-URL resolution). `Image Src`
|
||||
carries the hosted `detail`-rendition URL (or the original `source_url` for
|
||||
never-fetched/`failed` images, so nothing is lost). PUC-1 (Products page) is
|
||||
`GET /api/products/summary`; PUC-8 (history) is the runs list/detail
|
||||
endpoints; PUC-11 (sample CSV) is a static documented asset served at
|
||||
`GET /api/products/sample.csv` and linked from the column reference (DOC-2,
|
||||
§11).
|
||||
|
||||
### 6.6 Non-functional requirements & cross-cutting concerns
|
||||
|
||||
- **Security & privacy:** all endpoints behind SD-0001's session +
|
||||
storefront-membership gates; every query storefront-scoped (INV-14).
|
||||
**SSRF guard on image fetch:** http/https only, public IPs only (resolve
|
||||
and reject private/link-local/metadata ranges), no redirects across the
|
||||
guard, bounded size/time (INV-18). HTML sanitization at the boundary
|
||||
(INV-15). Uploaded file blobs are private objects, deleted with their
|
||||
draft; catalog data is merchant business data (no shopper PII in this
|
||||
feature). Run records and errors are retained indefinitely (they are the
|
||||
merchant's audit trail); drafts expire in ~1 h.
|
||||
- **Performance & scale:** validate + diff ≤ 5,000 rows in < 10 s
|
||||
(one catalog read, in-memory diff); apply in one transaction < 5 s;
|
||||
export streams (no whole-catalog buffering); image phase: ~4 concurrent
|
||||
fetches, Pillow in a worker thread pool — a 5,000-image run completes in
|
||||
tens of minutes *in the background* without touching request latency.
|
||||
- **Availability & resilience:** the import transaction is short; the image
|
||||
task is resumable across restarts/deploys (§6.9); a dead merchant image
|
||||
host degrades to per-image `failed`, never a hung import.
|
||||
- **Observability hooks:** `import_run.id` is the correlation id — the
|
||||
fetch task and all run-scoped log lines carry it (substance in §9).
|
||||
- **Accessibility:** preview/summary tiles and tables keyboard-navigable;
|
||||
progress updates `aria-live`; diff colors paired with +/− glyphs (not
|
||||
color-only).
|
||||
|
||||
### 6.7 Key decisions & alternatives considered
|
||||
|
||||
| Decision | Chosen | Alternatives considered | Why chosen |
|
||||
| --- | --- | --- | --- |
|
||||
| Matching identity | `Handle` for products, option-combo for variants (INV-13) | SKU as identity (the corpus's 11-05-0035) | Real catalogs have missing/duplicated SKUs; a SKU typo must be *correctable by import*, which identity can't be. Divergence from corpus recorded in §4. |
|
||||
| Image handling | Fetch-and-host with renditions; reject low-res | Hotlink merchant URLs; validate-only + hotlink; accept low-res with warning | Hotlinking leaks shopper traffic to third parties, rots, and breaks INV-16; rejected-low-res keeps every hosted image renderable at all surfaces (operator decision). |
|
||||
| Image timing | Post-commit background phase | Fetch during import (stalls minutes); fetch at preview (wasted on cancel) | Catalog lands in seconds; images are the async seam's first tenant. |
|
||||
| Background model | In-process asyncio task, resumable | Job queue (Cloud Tasks/celery); cron sweep | No queue infra for an MVP with one fire-and-forget job; the task body is queue-portable when scale demands (§6.9 names the seam). |
|
||||
| Preview state | Server-side `import_draft` + fingerprint | Client holds the diff and posts it back; re-validate blindly at confirm | The server applying exactly what it previewed is INV-11; a client-posted diff is unverifiable; fingerprint check catches catalog drift honestly (`409 preview_stale`). |
|
||||
| Format | Canonical = Shopify-flavored superset | Invent a clean format; exact Shopify schema | Familiarity + migration head start without inheriting Shopify's gaps (no kit seam, no volume/tax fields). |
|
||||
| Shopify free-text `Type` | Warned, not imported | Map into `Tags`; map into canonical `Type` | Canonical `Type` is structural (`standalone`/kits, #15); silently folding a category string into it (or tags) invents data the merchant didn't write. |
|
||||
| Blank vs absent | Absent column untouched; empty cell clears | Empty cell ignored (Shopify-ish); empty cell always clears including absent | Merchants must be able to clear a field via import; the preview shows every clear (INV-11), making the sharper semantic safe. |
|
||||
| Progress delivery | SPA polls run detail | WebSockets/SSE | One poll endpoint, no connection infra; seconds-stale progress is fine for this UX. |
|
||||
| Notifications | In-app only | Email summaries via SmtpMailer | Operator decision; revisit if imports go async/long. |
|
||||
| Object storage | New `platform/objectstore` port (local/GCS) | Store images as DB blobs; reuse a public bucket directly in domain code | Follows the mailer port pattern (SD-0001); DB blobs bloat backups and can't serve renditions efficiently. |
|
||||
|
||||
### 6.8 Testing strategy
|
||||
|
||||
Two-tier per handbook §9: unit/integration locally and in CI; E2E browser
|
||||
tests (Playwright) against localhost and PPE as the pipeline gates.
|
||||
|
||||
- **Unit — codec & dialects:** parse/serialize round-trip property tests
|
||||
(INV-12: `diff(catalog, import(export(catalog))) = ∅` over generated
|
||||
catalogs); a real Shopify export fixture mapping correctly; unknown-column
|
||||
warning extraction; blank-vs-absent semantics; every §6.5.1 row-error rule
|
||||
has a fixture.
|
||||
- **Unit — diff engine:** add/update/unchanged/error classification;
|
||||
option-combo matching incl. the no-option single variant; hosted-URL
|
||||
resolution (no re-fetch); clear-vs-untouched fields; fingerprint stability.
|
||||
- **Unit — images:** `platform/images` against fixture bytes (good jpeg/png/
|
||||
webp, tiny image, not-an-image, oversize); rendition dimensions.
|
||||
- **Unit — sanitizer:** hostile HTML fixtures (scripts, handlers, embeds).
|
||||
- **Integration:** apply transactionality (a mid-apply failure leaves zero
|
||||
catalog change); two-storefront isolation (INV-14); fetch-task resume
|
||||
after a simulated kill; SSRF guard against private-range URLs.
|
||||
- **E2E browser scenarios (named):**
|
||||
`e2e_import_preview_confirm` (canonical file → preview counts → drill-in
|
||||
diff → confirm → run detail → summary count updated);
|
||||
`e2e_import_shopify_dialect` (unmodified Shopify fixture, mapped banner +
|
||||
not-imported warnings); `e2e_import_errors_actionable` (bad rows: error
|
||||
table lists line/column/reason; valid rows import);
|
||||
`e2e_import_file_rejected` (missing `Title` column — whole-file message,
|
||||
no run recorded); `e2e_import_cancel_no_trace`;
|
||||
`e2e_roundtrip_noop` (export → re-import → all-unchanged, import action
|
||||
disabled); `e2e_export_download` (status filter respected);
|
||||
`e2e_image_outcomes` (fixture image host serving one good / one 404 / one
|
||||
tiny image → statuses, placeholder, Products-page notice). The E2E suite
|
||||
runs against a local fixture image host (no real egress in tests).
|
||||
- **Bootstrap:** SD-0001's INV-1 test extends — fresh DB → sign-up →
|
||||
storefront → import sample CSV → export matches.
|
||||
|
||||
### 6.9 Failure modes, rollback & flags
|
||||
|
||||
- **Failure mode:** crash mid-apply → **behavior:** transaction rolls back —
|
||||
zero catalog change; draft intact; SPA shows honest error →
|
||||
**rollback:** none needed → **alerted by:** ALR-2 (§9.3).
|
||||
- **Failure mode:** process restart (deploy) mid image-fetch →
|
||||
**behavior:** on startup, a recovery scan finds runs in `fetching_images`
|
||||
with `pending` images and resumes the task; per-image idempotency (§6.5.4)
|
||||
makes re-entry safe → **alerted by:** TEL/log line on recovery; ALR-3 if a
|
||||
run stays incomplete past threshold.
|
||||
- **Failure mode:** merchant image host down/slow → **behavior:** per-image
|
||||
timeout → `failed` with reason; run completes `complete_with_problems`;
|
||||
retry is a re-import (PUC-7) → no alert (expected operation).
|
||||
- **Failure mode:** object storage unavailable → **behavior:** affected
|
||||
images `failed` honestly; uploads (drafts) refuse with a clear error →
|
||||
**alerted by:** ALR-3.
|
||||
- **Failure mode:** draft expiry / catalog drift between preview and confirm
|
||||
→ **behavior:** `410 draft_expired` / `409 preview_stale`; UI prompts
|
||||
re-upload — never applies stale diffs (INV-11).
|
||||
- **Feature flag / kill switch:** none — the Products section ships whole;
|
||||
rollback is redeploying the prior version (pipeline §7.3). Migrations are
|
||||
additive new tables, so the prior version runs unaffected against the
|
||||
migrated schema (SD-0001 INV-7 forward-only).
|
||||
- **The async seam, named:** if catalogs outgrow the in-process model
|
||||
(imports > 5k rows, image sets taking hours), the fetch-task body moves
|
||||
onto a real queue (Cloud Tasks or a worker process) with identical
|
||||
inputs/outputs; `import_run`/`product_image` statuses already model the
|
||||
asynchrony, so the UX and API do not change shape.
|
||||
|
||||
### 6.10 Migration & compatibility
|
||||
|
||||
Greenfield within a live app: no existing catalog data to migrate. New
|
||||
additive tables (§6.3) via forward-only migrations (SD-0001 INV-7); the
|
||||
admin nav gains "Products"; no existing endpoint or contract changes.
|
||||
SD-0001's INV-1 (empty is a working state) holds — an empty catalog is the
|
||||
designed Products-page empty state. **Backwards compatibility:** none owed
|
||||
yet (no external API consumers). **Cutover:** ships whole behind the
|
||||
ordinary release pipeline; no dual-write, no point of no return short of
|
||||
merchants having imported real data (which INV-10/12 protect).
|
||||
|
||||
## 7. Delivery Plan
|
||||
|
||||
|
||||
Reference in New Issue
Block a user